The `-Wno-missing-template-arg-list-after-template-kw` flag is only needed for the grpc library. Use `+=` for the default build flags to make it easier to extend in the future.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
Amendments activated for more than 2 years can be retired. This change retires the fix1515 amendment.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
As XRPL network demand grows and ledger sizes increase, the default 4K NuDB block size becomes a performance bottleneck, especially on high-performance storage systems. Modern SSDs and enterprise storage often perform better with larger block sizes, but rippled previously had no way to configure this parameter. This change therefore implements configurable NuDB block size support, allowing operators to optimize storage performance based on their hardware configuration. The feature adds a new `nudb_block_size` configuration parameter that enables block sizes from 4K to 32K bytes, with comprehensive validation and backward compatibility.
Specific changes are:
- Implements `parseBlockSize()` function with validation.
- Adds `nudb_block_size` configuration parameter.
- Supports block sizes from 4K to 32K (power of 2).
- Adds comprehensive logging and error handling.
- Maintains backward compatibility with 4K default.
- Adds unit tests for block size validation.
- Updates configuration documentation with performance guidance.
- Marks feature as experimental.
- Applies code formatting fixes.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
Similarly to other transaction typed that can create a trust line or MPToken for the transaction submitter (e.g. CashCheck #5285, EscrowFinish #5185 ), VaultWithdraw should enforce reserve before creating a new object. Additionally, the lsfRequireDestTag account flag should be enforced for the transaction submitter.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
The default job timeout is 5 hours, while build times are anywhere between 4-20 mins and test times between 2-10. As a runner occasionally gets stuck, we should fail much quicker.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This PR sets the fail-fast strategy option to false (it defaults to true), unless it is run by a merge group.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This change sanitizes inputs by setting them as environment variables, and adjusts the number of CPUs used for building. Namely, GitHub inputs should be sanitized, per recommendation by Semgrep, as using them directly poses a security risk. A recent change further overrode the global configuration by having builds use all cores, but as we have noticed an increased number of job cancelation this change updates it to use all cores less one.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
There are some tests that write out JSONs as a string instead of using the Json::Value library, which are cleaned up by this change.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
A regression was introduced in #5669 which would cause rippled to potentially dereference a disengaged std::optional when connecting to a peer. This would cause UB in release build and crash in debug.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This change replaces boost::lexical_cast<std::string> with to_string in some of the tests to make them more readable.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This change replaces instances of JSON LastLedgerSequence with last_ledger_seq, which makes the tests a bit simpler and easier to read.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
Windows is extremely chatty and generates tons of logs when building, making it practically impossible to use the build logs to debug issues. This change sets the verbosity to 'quiet' on Windows.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This PR changes fee().accountReserve(0) to fee().reserve, as the current network reserve amount should be used instead of the account reserve.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This change updates the Docker image hashes of the tools-rippled images to fix a missing dependency.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This change adds a paychan namespace to the TestHelpers and implementation files, improving organization and clarity. Additionally, it updates the AMM test to use the new `paychan::create` function for payment channel creation.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This change excludes from Codecov unreachable/difficult-to-test transaction code (such as `tecINTERNAL`) and old code (from amendments that have been enabled for a long time that are only around for ledger replay reasons). This removes about 200 lines of misses and increases the Codecov coverage by 0.3% (79.2% to 79.5%).
This change adds a wildcard to the release branch in the CI pipeline spec. Namely, after adopting an improved release process, with release branches that now look like release-X.Y, the trigger pipeline was no longer running as it only searched for an exact match to release.
This change downgrades OpenSSL 3.6.0 to 3.5.4. To avoid potential zero-day issues in a new major version of OpenSSL, 3.6.0, it is safer to stick with 3.5.4. While 3.6.0 has some nice new features, such as improved SHA512 hashing, it also introduces new features that could contain bugs. In contrast, 3.5.4 has seen quite a few bug fixes over 3.5.0 and has been used in the wild for a while now.
This change fixes a release build error with GCC 15.2.
The `fields` variable is only used in `XRPL_ASSERT`, which evaluates to nothing in a Release build, leaving the variable unused. This change silences the build warning.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
This change uses the new RHEL 9 and 10 images to build and test the binary, and adds support for having different Docker image SHAs per distro-compiler combination.
Instead of supporting RHEL each minor version, we are simplifying our pipelines by only supporting RHEL major versions. Our CI Docker images have already been updated accordingly, and we recently added support for RHEL 10 as well. Up until now, the CI Docker images had all been rebuilt at the same time, but that is not necessarily true as the most recent push to the CI repo has shown where the RHEL images now have a different SHA than the Debian and Ubuntu ones.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
* Restructures Transactor signature checking code to be able to handle a `sigObject`, which may be the full transaction, or may be an object field containing a separate signature. Either way, the `sigObject` can be a single- or multi-sign signature.
* Restructures `Transactor::preflight` to create several functions that will remove the need for error-prone boilerplate code in derived classes' implementations of `preflight`.
This change adds `STInt32` as a new `SType` under the `STInteger` umbrella, with `SType` value `12`. This is the first and only `STInteger` type that supports negative values.
This change adds more comprehensive tests for the `FeeVote` module, which previously only checked the basics, and not the more comprehensive flows in that class.
This change makes the regex in `HttpClient.cpp` that matches the content-length http header case insensitive to improve compatibility, as http headers are case insensitive.
This change raises logging severity from `INFO` to `WARN` when handling UNL manifest signed with an unexpected / invalid key. It also changes the internal error code for an invalid format of UNL manifest to `invalid` (from `untrusted`).
This is a follow up to problems experienced by an UNL node due to old manifest key configured in `validators.txt`, which would be easier to diagnose with improved logging.
It also replaces a log line with `UNREACHABLE` for an impossible situation when we match UNL manifest key against a configured key which has an invalid type (we cannot configure such a key because of checks when loading configured keys).
This adds a comment to avoid using `std::counting_semaphore` until the minimum compiler versions of GCC and Clang have been updated to no longer contain the bug that is present in older compilers.
This change reverts #5617, because it will require extensive testing that will take up more time than we have before the next scheduled release.
Reverting this change does not mean we are abandoning it. We aim to pick it back up once there's a sufficient time window to allow for testing on multiple distros running a mixture of OpenSSL 1.x and 3.x.
This change is to improve code coverage (and to simplify #5720 and #5725); there is otherwise no change in functionality. The change adds basic tests for `STInteger` and `STParsedJSON`, so it becomes easier to test smaller changes to the types, as well as removes `STParsedJSONArray`, since it is not used anywhere (including in Clio).
- Added a new Invariant: `ValidPseudoAccounts` which checks that all pseudo-accounts behave consistently through creation and updates, and that no "real" accounts look like pseudo-accounts (which means they don't have a 0 sequence).
- `to_short_string(base_uint)`. Like `to_string`, but only returns the first 8 characters. (Similar to how a git commit ID can be abbreviated.) Used as a wrapped sink to prefix most transaction-related messages. More can be added later.
- `XRPL_ASSERT_PARTS`. Convenience wrapper for `XRPL_ASSERT`, which takes the `function` and `description` as separate parameters.
- `SField::sMD_PseudoAccount`. Metadata option for `SField` definitions to indicate that the field, if set in an `AccountRoot` indicates that account is a pseudo-account. Removes the need for hard-coded field lists all over the place. Added the flag to `AMMID` and `VaultID`.
- Added functionality to `SField` ctor to detect both code and name collisions using asserts. And require all SFields to have a name
- Convenience type aliases `STLedgerEntry::const_pointer` and `STLedgerEntry::const_ref`. (`SLE` is an alias to `STLedgerEntry`.)
- Generalized `feeunit.h` (`TaggedFee`) into `unit.h` (`ValueUnit`) and added new "BIPS"-related tags for future use. Also refactored the type restrictions to use Concepts.
- Restructured `transactions.macro` to do two big things
1. Include the `#include` directives for transactor header files directly in the macro file. Removes the need to update `applySteps.cpp` and the resulting conflicts.
2. Added a `privileges` parameter to the `TRANSACTION` macro, which specifies some of the operations a transaction is allowed to do. These `privileges` are enforced by invariant checks. Again, removed the need to update scattered lists of transaction types in various checks.
- Unit tests:
1. Moved more helper functions into `TestHelpers.h` and `.cpp`.
2. Cleaned up the namespaces to prevent / mitigate random collisions and ambiguous symbols, particularly in unity builds.
3. Generalized `Env::balance` to add support for `MPTIssue` and `Asset`.
4. Added a set of helper classes to simplify `Env` transaction parameter classes: `JTxField`, `JTxFieldWrapper`, and a bunch of classes derived or aliased from it. For an example of how awesome it is, check the changes `src/test/jtx/escrow.h` for how much simpler the definitions are for `finish_time`, `cancel_time`, `condition`, and `fulfillment`.
5. Generalized several of the amount-related helper classes to understand `Asset`s.
6. `env.balance` for an MPT issuer will return a negative number (or 0) for consistency with IOUs.
This change re-enables building and testing all configurations, but only for the daily scheduled run. Previously all configurations were run for each merge into the develop branch, but that overwhelmed both the GitHub runners and the Conan remote, and thus they were limited to just a subset of configurations. Now that the number of jobs is limited via `max-parallel: 10`, we should be able to safely enable building all configurations again. However, building them all once a day instead of for each PR merge should be sufficient.
- Ensures the commits don't get orphaned, even though the relevant code
changes are already included.
* tag '2.5.1':
Set version to 2.5.1
Fix: Don't flag consensus as stalled prematurely (#5658)
GitHub runners have a limit on how many concurrent jobs they can actually process (even though they will try to run them all at the same time), and similarly the Conan remote cannot handle hundreds of concurrent requests. Previously, the Conan dependency uploading was already limited to max 10 jobs running in parallel, and this change makes the same change to the build+test workflow.
This change adds a fix amendment (`fixIncludeKeyletFields`) that adds:
* `sfSequence` to `Escrow` and `PayChannel`
* `sfOwner` to `SignerList`
* `sfOracleDocumentID` to `Oracle`
This ensures that all ledger entries hold all the information needed to determine their keylet.
The XRPL establishes connections in three stages: first a TCP connection, then a TLS/SSL handshake to secure the connection, and finally an upgrade to the bespoke XRP Ledger peer-to-peer protocol. During connection termination, xrpld directly closes the TCP connection, bypassing the TLS/SSL shutdown handshake. This makes peer disconnection diagnostics more difficult - abrupt TCP termination appears as if the peer crashed rather than disconnected gracefully.
This change refactors the connection lifecycle with the following changes:
- Enhanced outgoing connection logic with granular timeouts for each connection stage (TCP, TLS, XRPL handshake) to improve diagnostic capabilities
- Updated both PeerImp and ConnectAttempt to use proper asynchronous TLS shutdown procedures for graceful connection termination
* extends the functionality of the MPTokenIssuanceSet transaction, allowing the issuer to update fields or flags that were explicitly marked as mutable during creation.
Clio should only be notified when releases are about to be made, instead of for all PR, so this change only notifies Clio when a PR targets the release or master branch.
This change wraps all GitHub conditionals in `${{ .. }}`, both for consistency and to reduce unexpected failures, because it was previously noticed that not all conditionals work without those curly braces.
- Amendment: fixDelegateV1_1
- In DelegateSet, disallow invalid PermissionValues like 0, and transaction values when the transaction's amendment is not enabled. Acts as if the transaction doesn't exist, which is the same thing older versions without the amendment will do.
- Payment burn/mint should disallow DEX currency exchange.
- Support MPT for Payment burn/mint.
- Don't run upload-conan-deps in PRs, unless the PR changes the workflow file.
- Change cron schedule for uploading Conan dependencies to run after work hours for most dev.
- This should prevent Artifactory from being overloaded by too many requests at a time.
- Uses "max-parallel" to limit the build job to 10 simultaneous instances.
- Only run the minimal matrix on PRs.
For the purposes of being able to merge a PR, Github Actions jobs count as passed if they ran and passed, or were skipped.
With this change, if any of the jobs that "passed" depends on fail or are cancelled, then "passed" will fail. If they all succeed or are skipped, then "passed" is skipped, which does not prevent a merge.
This saves spinning up a runner in the usual case where things work, and will simplify our branch protection rules, so that only "passed" will need to be checked.
* Add and Scale to VaultCreate
* Add round-trip calculation to VaultDeposit VaultWithdraw and VaultClawback
* Implement Number::truncate() for VaultClawback
* Add rounding to DepositWithdraw
* Disallow zero shares withdraw or deposit with tecPRECISION_LOSS
* Return tecPATH_DRY on overflow when converting shares/assets
* Remove empty shares MPToken in clawback or withdraw (except for vault owner)
* Implicitly create shares MPToken for vault owner in VaultCreate
* Review feedback: defensive checks in shares/assets calculations
---------
Co-authored-by: Ed Hennis <ed@ripple.com>
Fix stalled consensus detection to prevent false positives in situations where there are no disputed transactions.
Stalled consensus detection was added to 2.5.0 in response to a network consensus halt that caused a round to run for over an hour. However, it has a flaw that makes it very easy to have false positives. Those false positives are usually mitigated by other checks that prevent them from having an effect, but there have been several instances of validators "running ahead" because there are circumstances where the other checks are "successful", allowing the stall state to be checked.
* chore: Use conan lockfile
* Add windows-specific dependencies as well
* Add more info about lockfiles
* Update lockfile to latest version
* Update BUILD.md with conan install note
This is a major refactor of LedgerEntry.cpp. It adds a number of helper functions to make the code easier to maintain.
It also splits up the ledger and ledger_entry tests into different files, and cleans up the ledger_entry tests to make them easier to write and maintain.
This refactor also caught a few bugs in some of the other RPC processing, so those are fixed along the way.
This is a follow-up to PR #5664 that further improves the specificity of logging for refused peer connections. The previous changes did not account for several key scenarios, leading to potentially misleading log messages.
It addresses the following
- Inbound Disabled: Connections are now explicitly logged as rejected when the server is not configured to accept inbound peers. Previously, this was logged as the server being "full," which was technically correct but lacked diagnostic clarity.
- Duplicate Connections: The logging now distinguishes between two types of duplicate connection refusals:
- When a peer with the same node public key is already connected (duplicate connection).
- When a connection is rejected because the limit for connections from a single IP address has been reached.
These changes provide more accurate and actionable diagnostic information when analyzing peer connection behavior.
Test jobs will run if
* Either the PR is non-draft or has the "DraftRunCI" label set *AND*
* One of the following:
* Certain files were changed *OR*
* The PR is non-draft and has the "Ready to merge" flag *OR*
* The workflow is being run from the merge queue.
Additionally, a meta "passed" job was added that is dependent on all the other test jobs, so the required jobs list under branch protection rules only needs to specify "passed" to ensure that *either* all the test jobs pass *or* all the test jobs are skipped because they don't need to be run.
This allows PRs that don't affect the build or binary to be merged without overriding.
This updates Boost to 1.88, which is needed because Clio wants to move to 1.88 as that fixes several ASAN false positives around coroutine usage. In order for Clio to move to newer boost, libXRPL needs to move too. Hence the changes in this PR. A lot has changed between 1.83 and 1.88 so there are lots of changes in the diff, especially in regards to Boost.Asio and coroutines in particular.
This change removes `labeled` and `unlabeled` as pipeline trigger actions, and instead adds `reopened` and `ready_for_review`. The logic whether to run the pipeline jobs is then simplified, although to get a draft PR with the `DraftCIRun` label to run it can be necessary to close and reopen a PR.
The change updates how clang-format is called in CI and locally, and adds prettier to the pre-commit hook. Proto files are now also formatted, while external files are excluded.
This change will skip running the notify-clio job when a PR is created from a fork, and reorders the strategy matrix configuration fields so GitHub will more clearly show which configuration is running.
This change reverts the formatting applied to external files and adds formatting of proto files.
As clang-format will complain if a proto file is modified or moved, since the .clang-format file does not explicitly contain a section for proto files, the change has been included in this PR as well.
This change updates OpenSSL from 1.1.1w to 3.5.2. The code works as-is, but many functions have been marked as deprecated and thus will need to be rewritten. For now we explicitly add the `-DOPENSSL_SUPPRESS_DEPRECATED` to give us time to do so, while providing us with the benefits of the updated version.
This change modifies the `build_only` check used to determine whether to run tests. For easier debugging in the future it also prints out the contents of the strategy matrix.
Reduce log noise by changing two log statements from error/warn level to debug level. These logs occur during normal operation when AMM offers are not available or when IOU authorization checks fail, which are expected scenarios that don't require an elevated log level.
Currently, all peer connection rejections are logged with the reason "slots full". This is inaccurate, as the PeerFinder can also reject connections if they are a duplicate. This change updates the logging logic to correctly report the specific reason (full or duplicate) for a rejected peer connection, providing more accurate diagnostic information.
This change fixes the suite names all around the test files, to make them match to the folder name in which this test files are located. Also, the RCL test files are relocated to the consensus folder, because they are testing consensus functionality.
This change replaces the configuration variable with the hardcoded `https://conan.ripplex.io`, making it possible for PRs from forks to use our Conan remote containing workarounds.
We're currently calling `XXH3_createState` and `XXH3_freeState` when hashing an object. However, it may be slow because they call `malloc` and `free`, which may affect the performance. This change avoids the use of the streaming API as much as possible by using an internal buffer.
Fix stalled consensus detection to prevent false positives in situations where there are no disputed transactions.
Stalled consensus detection was added to 2.5.0 in response to a network consensus halt that caused a round to run for over an hour. However, it has a flaw that makes it very easy to have false positives. Those false positives are usually mitigated by other checks that prevent them from having an effect, but there have been several instances of validators "running ahead" because there are circumstances where the other checks are "successful", allowing the stall state to be checked.
This change updates some incorrect Conan commands for Conan 2. As some flags do not exist in Conan 2, such as --settings build_type=[configuration], the commands have been adjusted accordingly. This change further uses the org-level variables and secrets rather than the repo-level ones.
This change introduces two key optimizations:
* Mutex scope reduction: Limits the lock to individual partitions within `TaggedCache`, reducing contention.
* Decoupling: Removes the tight coupling between `LedgerHistory` and `TaggedCache`, improving modularity and testability.
Lock contention analysis based on eBPF showed significant improvements as a result of this change.
This change uploads built Conan dependencies to the Conan remote upon merge into the develop branch.
At the moment, whenever Conan dependencies change, we need to remember to manually push them to our Conan remote, so they are cached for future reuse. If we forget to do so, these changed dependencies need to be rebuilt over and over again, which can take a long time.
This change:
* Removes the patched Conan recipes from the `external/` directory.
* Adds instructions for contributors how to obtain our patched recipes.
* Updates the Conan remote name and remote URL (the underlying package repository isn't changed).
* If the remote already exists, updates the URL instead of removing and re-adding.
* This is not done for the libXRPL job as it still uses Conan 1. This job will be switched to Conan 2 soon.
* Removes duplicate Conan remote CI pipeline steps.
* Overwrites the existing global.conf on MacOS and Windows machines, as those do not run CI pipelines in isolation but all share the same Conan installation; appending the same config over and over bloats the file.
This change updates BUILD.md for Conan 2, add fixes/workarounds for Apple Clang 17, Clang 20 and CMake 4. This also removes (from BUILD.md only) workarounds for compiler versions which we no longer support e.g. Clang 15 and adds compilation flag -Wno-deprecated-declarations to enable building with Clang 20 on Linux.
This change fixes an issue where the order of `PriceDataSeries` was out of sync between when `PriceOracle` was created and when it was updated. Although they are registered in the canonical order when updated, they are created using the order specified in the transaction; this change ensures that they are also registered in the canonical order when created.
This change decouples `ledger` from `xrpld/app`, and therefore fully clears the path to the modularisation of the ledger component. Before this change, `View.cpp` relied on `MPTokenAuthorize::authorize; this change moves `MPTokenAuthorize::authorize` to `View.cpp` to invert the dependency, making ledger a standalone module.
The Payment transaction metadata is missing the `DeliveredAmount` field that displays the actual amount delivered to the destination excluding transfer fees. This amendment fixes this problem.
#5224 added (among other things) a `VaultWithdraw` transaction that allows setting the recipient of the withdrawn funds in the `Destination` transaction field. This technically turns this transaction into a payment, and in some respect the implementation does follow payment rules, e.g. enforcement of `lsfRequireDestTag` or `lsfDepositAuth`, or that MPT transfer has destination `MPToken`. However for IOUs, it missed verification that the destination account has a trust line to the asset issuer. Since the default behavior of `accountSendIOU` is to create this trust line (if missing), this is what `VaultWithdraw` currently does. This is incorrect, since the `Destination` might not be interested in holding the asset in question; this basically enables spammy transfers. This change, therefore, removes automatic creation of a trust line to the `Destination` account in `VaultWithdraw`.
This change updates RocksDB to its latest version. RocksDB is backward-compatible, so even though this is a major version bump, databases created with previous versions will continue to function.
The external RocksDB folder is removed, as the latest version available via Conan Center no longer needs custom patches.
Before `XRPLF/ci` images, we did not have a `dependencies:` job for clang-16, so `instrumentation:` had to build its own dependencies. Now we have clang-16 Conan dependencies built in a separate job that can be used.
This change includes `network_id` data in the validations and ledger subscription stream responses, as well as unit tests to validate the response fields. Fixes#4783
This change adds support for `DomainID` to existing transactions `MPTokenIssuanceCreate` and `MPTokenIssuanceSet`.
In #5224 `DomainID` was added as an access control mechanism for `SingleAssetVault`. The actual implementation of this feature lies in `MPToken` and `MPTokenIssuance`, hence it makes sense to enable the use of `DomainID` also in `MPTokenIssuanceCreate` and `MPTokenIssuanceSet`, following same rules as in Vault:
* `MPTokenIssuanceCreate` and `MPTokenIssuanceSet` can only set `DomainID` if flag `MPTRequireAuth` is set.
* `MPTokenIssuanceCreate` requires that `DomainID` be a non-zero, uint256 number.
* `MPTokenIssuanceSet` allows `DomainID` to be zero (or empty) in which case it will remove `DomainID` from the `MPTokenIssuance` object.
The change is amendment-gated by `SingleAssetVault`. This is a non-breaking change because `SingleAssetVault` amendment is `Supported::no`, i.e. at this moment considered a work in progress, which cannot be enabled on the network.
For jobs running in containers, $GITHUB_WORKSPACE and ${{ github.workspace }} might not be the same directory. The actions/checkout step is supposed to checkout into `$GITHUB_WORKSPACE` and then add it to safe.directory (see instructions at https://github.com/actions/checkout), but that's apparently not happening for some container images. We can't be sure what is actually happening, so we preemptively add both directories to `safe.directory`. See also the GitHub issue opened in 2022 that still has not been resolved https://github.com/actions/runner/issues/2058.
The current implementation of rngfill is prone to false warnings from GCC about array bounds violations. Looking at the code, the implementation naively manipulates both the bytes count and the buffer pointer directly to ensure the trailing memcpy doesn't overrun the buffer. As expressed, there is a data dependency on both fields between loop iterations.
Now, ideally, an optimizing compiler would realize that these dependencies were unnecessary and end up restructuring its intermediate representation into a functionally equivalent form with them absent. However, the point at which this occurs may be disjoint from when warning analyses are performed, potentially rendering them more difficult to
determine precisely.
In addition, it may also consume a portion of the budget the optimizer has allocated to attempting to improve a translation unit's performance. Given this is a function template which requires context-sensitive instantiation, this code would be more prone than most to being inlined, with a decrease in optimization budget corresponding to the effort the optimizer has already expended, having already optimized one or more calling functions. Thus, the scope for impacting the the ultimate quality of the code generated is elevated.
For this change, we rearrange things so that the location and contents of each memcpy can be computed independently, relying on a simple loop iteration counter as the only changing input between iterations.
Remove `include(default)` from `conan/profiles/libxrpl`. This means that we will now rely on compiler workarounds stored elsewhere e.g. in global.conf.
This change reverts the usage of boost::shared_mutex back to std::shared_mutex. The change was originally introduced as a workaround for a bug in glibc 2.28 and older versions, which could cause threads using std::shared_mutex to stall. This issue primarily affected Ubuntu 18.04 and earlier distributions, which we no longer support.
This change fixes the MacOS pipeline issue by limiting GitHub to choose the existing runners, ensuring the new experimental runners are excluded until they are ready.
This issue was reported on the Javascript client library: XRPLF/xrpl.js#2611
The type filter (Note: as of the latest version of rippled, type parameter is deprecated) does not work as expected. This PR removes the type filter from the ledger command.
This PR updates several dependencies to their latest versions. Not all dependencies have been updated, as some need to be patched and some require additional code changes due to backward incompatibilities introduced by the version bump.
Due to rounding, the LPTokenBalance of the last LP might not match the LP's trustline balance. This was fixed for `AMMWithdraw` in `fixAMMv1_1` by adjusting the LPTokenBalance to be the same as the trustline balance. Since `AMMClawback` is also performing a withdrawal, we need to adjust LPTokenBalance as well in `AMMClawback.`
This change includes:
1. Refactored `verifyAndAdjustLPTokenBalance` function in `AMMUtils`, which both`AMMWithdraw` and `AMMClawback` call to adjust LPTokenBalance.
2. Added the unit test `testLastHolderLPTokenBalance` to test the scenario.
3. Modify the existing unit tests for `fixAMMClawbackRounding`.
Currently there is no easy way to track MPT related transactions for the issuer. This change allows MPT transactions to show up on issuer's AccountTx RPC (to align with how IOUs work).
* Update the `account_info` API so that the `allowTrustLineLocking` flag is included in the response.
* The proposed `TokenEscrow` amendment added an `allowTrustLineLocking` flag in the `AccountRoot` object.
* In the API response, under `account_flags`, there is now an `allowTrustLineLocking` field with a boolean (`true` or `false`) value.
* For reference, the XLS-85 Token-Enabled Escrows implementation can be found in https://github.com/XRPLF/rippled/pull/5185
The current version was copied from `antithesis-sdk-cpp` but there is no logical reason to require this specific version of CMake. This change downgrades the version to make the project build with older CMake versions.
Having `boost::boost` in `self.requires` makes clio link with all boost libraries. There are additionally several Boost stacktrace backends that are both linked with, which violate ODR.
This change fixes the problem.
This PR refactors `CredentialHelpers` and removes some unnecessary dependencies as a step of modularization.
The ledger component is almost independent except that it references `MPTokenAuthorize` and `CredentialHelpers.h`, and the latter further references `Transactor.h`. This PR partially clears the path to modularizing the ledger component and decouples `CredentialHelpers` from xrpld.
This PR fixes a crash in tests when the test `Env is run at trace/debug log level.
This issue only affects tests, and only if logging at trace/debug level, so really only relevant during rippled development, and does not affect production servers.
The tests that ensure `tfInnerBatchTxn` won't block delegated transactions silently fail in `Delegate_test.cpp`. This change removes these cases from that file and adds them to `Batch_test.cpp` instead where they do not silently fail, because there the batch delegate results are explicitly checked. Moving them to that file further avoids refactoring many helper functions.
This change allows users to submit simulate requests from a multi-sign account without needing to specify the accounts that are doing the multi-signing, and fixes an error with simulate that allowed double-"signed" (both single-sign and multi-sign public keys are provided) transactions.
Multi-line log messages are hard to work with. Writing these handful of related messages as one message should make the log a tiny bit easier to manage.
The CMake statements that make it seem as if the number of cores used to build external project dependencies is halved don't actually do anything. This change removes these statements.
* Adds `tecNO_DELEGATE_PERMISSION` for unauthorized transactions sent by a delegated account.
* Returns `tecNO_TARGET` instead of `terNO_ACCOUNT` for the `DelegateSet` transaction if the delegated account does not exist.
* Fixes `tfFullyCanonicalSig` and `tfInnerBatchTxn` blocking transactions issue by adding `tfUniversal` in the permission related masks in `txFlags.h`
The change increases the default network I/O worker thread pool size from 2 to 6. This will improve stability, as worker thread saturation correlates to desyncs, particularly on high-traffic peers, such as hubs.
To be able to consume `rippled` in Conan 2, the recipe should specify transitive_headers for external libraries that are present in the exported header files. This change remains compatibility with Conan 1, where this flag was not present.
- Specification: https://github.com/XRPLF/XRPL-Standards/pull/272
- Amendment: `TokenEscrow`
- Enables escrowing of IOU and MPT tokens in addition to native XRP.
- Allows accounts to lock issued tokens (IOU/MPT) in escrow objects, with support for freeze, authorization, and transfer rates.
- Adds new ledger fields (`sfLockedAmount`, `sfIssuerNode`, etc.) to track locked balances for IOU and MPT escrows.
- Updates EscrowCreate, EscrowFinish, and EscrowCancel transaction logic to support IOU and MPT assets, including proper handling of trustlines and MPT authorization, transfer rates, and locked balances.
- Enforces invariant checks for escrowed IOU/MPT amounts.
- Extends GatewayBalances RPC to report locked (escrowed) balances.
The changes are focused on fixing NFT transactions bypassing the trustline authorization requirement and potential invariant violation when interacting with deep frozen trustlines.
* Add AMM bid/create/deposit/swap/withdraw/vote invariants:
- Deposit, Withdrawal invariants: `sqrt(asset1Balance * asset2Balance) >= LPTokens`.
- Bid: `sqrt(asset1Balance * asset2Balance) > LPTokens` and the pool balances don't change.
- Create: `sqrt(asset1Balance * assetBalance2) == LPTokens`.
- Swap: `asset1BalanceAfter * asset2BalanceAfter >= asset1BalanceBefore * asset2BalanceBefore`
and `LPTokens` don't change.
- Vote: `LPTokens` and pool balances don't change.
- All AMM and swap transactions: amounts and tokens are greater than zero, except on withdrawal if all tokens
are withdrawn.
* Add AMM deposit and withdraw rounding to ensure AMM invariant:
- On deposit, tokens out are rounded downward and deposit amount is rounded upward.
- On withdrawal, tokens in are rounded upward and withdrawal amount is rounded downward.
* Add Order Book Offer invariant to verify consumed amounts. Consumed amounts are less than the offer.
* Fix Bid validation. `AuthAccount` can't have duplicate accounts or the submitter account.
This commit changes the ledger close in env.meta to be conditional on if it hasn't already been closed (i.e. the current ledger doesn't have any transactions in it). This change will make it a bit easier to use, as it will still work if you close the ledger outside of this usage. Previously, if you accidentally closed the ledger outside of the meta function, it would segfault and it was incredibly difficult to debug.
This commit introduces the following changes:
* Renames `vp_enable config` option to `vp_base_squelch_enable` to enable squelching for validators.
* Removes `vp_squelch` config option which was used to configure whether to send squelch messages to peers or not. With this flag removed, if squelching is enabled, squelch messages will be sent. This was an option used for debugging.
* Introduces a temporary `vp_base_squelch_max_trusted_peers` config option to change the max number of peers who are selected as validator message sources. This is a temporary option, which will be removed once a good value is found.
* Adds a traffic counter to count the number of times peers ignored squelch messages and kept sending messages for squelched validators.
* Moves the decision whether squelching is enabled and ready into Slot.h.
- Specification: [XRPLF/XRPL-Standards 56](https://github.com/XRPLF/XRPL-Standards/blob/master/XLS-0056d-batch/README.md)
- Amendment: `Batch`
- Implements execution of multiple transactions within a single batch transaction with four execution modes: `tfAllOrNothing`, `tfOnlyOne`, `tfUntilFailure`, and `tfIndependent`.
- Enables atomic multi-party transactions where multiple accounts can participate in a single batch, with up to 8 inner transactions and 8 batch signers per batch transaction.
- Inner transactions use `tfInnerBatchTxn` flag with zero fees, no signature, and empty signing public key.
- Inner transactions are applied after the outer batch succeeds via the `applyBatchTransactions` function in apply.cpp.
- Network layer prevents relay of transactions with `tfInnerBatchTxn` flag - each peer applies inner transactions locally from the batch.
- Batch transactions are excluded from AccountDelegate permissions but inner transactions retain full delegation support.
- Metadata includes `ParentBatchID` linking inner transactions to their containing batch for traceability and auditing.
- Extended STTx with batch-specific signature verification methods and added protocol structures (`sfRawTransactions`, `sfBatchSigners`).
Before #5224, the pseudoaccount ID was calculated using prefix expressed in `std::uint16_t`. The refactoring to move the pseudoaccount ID calculation to View.cpp had accidentally changed the prefix type to `int` (derived from `auto i = 0`) which in turn changed the length of the input to `sha512Half` from 2 bytes to 4, altering the result.
This resulted in a different ID of the pseudoaccount calculated from the function after the refactoring, breaking the ledger. This impacts AMMCreate, even when the `SingleAssetVault` amendment is not active. This change restores the prefix type to `std::uint16_t`.
- Specification: XRPLF/XRPL-Standards#239
- Amendment: `SingleAssetVault`
- Implements a vault feature used to store a fungible asset (XRP, IOU, or MPT, but not NFT) and to receive shares in the vault (an MPT) in exchange.
- A vault can be private or public.
- A private vault can use permissioned domains, subject to the `PermissionedDomains` amendment.
- Shares can be exchanged back into asset with `VaultWithdraw`.
- Permissions on the asset in the vault are transitively applied on shares in the vault.
- Issuer of the asset in the vault can clawback with `VaultClawback`.
- Extended `MPTokenIssuance` with `DomainID`, used by the permissioned domain on the vault shares.
Co-authored-by: John Freeman <jfreeman08@gmail.com>
Using std::barrier performs extremely poorly (~1 hour vs ~1 minute to run the test suite) in certain macOS environments.
To unblock our macOS CI pipeline, std::barrier has been replaced with a custom mutex-based barrier (Barrier) that significantly improves performance without compromising correctness.
Running unit tests in parallel and multiple threads can write into one file can corrupt output files, and then gcovr won't be able to parse the corrupted file. This change adds -fprofile-update=atomic as instructed by https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68080.
This change implements the account permission delegation described in XLS-75d, see https://github.com/XRPLF/XRPL-Standards/pull/257.
* Introduces transaction-level and granular permissions that can be delegated to other accounts.
* Adds `DelegateSet` transaction to grant specified permissions to another account.
* Adds `ltDelegate` ledger object to maintain the permission list for delegating/delegated account pair.
* Adds an optional `Delegate` field in common fields, allowing a delegated account to send transactions on behalf of the delegating account within the granted permission scope. The `Account` field remains the delegating account; the `Delegate` field specifies the delegated account. The transaction is signed by the delegated account.
This change updates the squelching logic to accept squelch messages for untrusted validators. As a result, servers will also squelch untrusted validator messages reducing duplicate traffic they generate.
In particular:
* Updates squelch message handling logic to squelch messages for all validators, not only trusted ones.
* Updates the logic to send squelch messages to peers that don't squelch themselves
* Increases the threshold for the number of messages that a peer has to deliver to consider it as a candidate for validator messages.
Combines four related changes:
1. "Decrease `shouldRelay` limit to 30s." Pretty self-explanatory. Currently, the limit is 5 minutes, by which point the `HashRouter` entry could have expired, making this transaction look brand new (and thus causing it to be relayed back to peers which have sent it to us recently).
2. "Give a transaction more chances to be retried." Will put a transaction into `LedgerMaster`'s held transactions if the transaction gets a `ter`, `tel`, or `tef` result. Old behavior was just `ter`.
* Additionally, to prevent a transaction from being repeatedly held indefinitely, it must meet some extra conditions. (Documented in a comment in the code.)
3. "Pop all transactions with sequential sequences, or tickets." When a transaction is processed successfully, currently, one held transaction for the same account (if any) will be popped out of the held transactions list, and queued up for the next transaction batch. This change pops all transactions for the account, but only if they have sequential sequences (for non-ticket transactions) or use a ticket. This issue was identified from interactions with @mtrippled's #4504, which was merged, but unfortunately reverted later by #4852. When the batches were spaced out, it could potentially take a very long time for a large number of held transactions for an account to get processed through. However, whether batched or not, this change will help get held transactions cleared out, particularly if a missing earlier transaction is what held them up.
4. "Process held transactions through existing NetworkOPs batching." In the current processing, at the end of each consensus round, all held transactions are directly applied to the open ledger, then the held list is reset. This bypasses all of the logic in `NetworkOPs::apply` which, among other things, broadcasts successful transactions to peers. This means that the transaction may not get broadcast to peers for a really long time (5 minutes in the current implementation, or 30 seconds with this first commit). If the node is a bottleneck (either due to network configuration, or because the transaction was submitted locally), the transaction may not be seen by any other nodes or validators before it expires or causes other problems.
This change addresses an issue where `rippled` attempts to connect to an IPv6 address, even when the local network lacks IPv6 support, resulting in a "Network is unreachable" error.
The fix replaces the custom endpoint selection logic with `boost::async_connect`, which sequentially attempts to connect to available endpoints until one succeeds or all fail.
This PR replaces the word `failed` with `failure` in any test names and renames some test files to fix MSVC warnings, so that it is easier to search through the test output to find tests that failed.
This change fixes a number of issues involved with CTID:
* CTID is not present on all RPC tx transactions.
* rpcWRONG_NETWORK is missing in the ErrorCodes.cpp
When using subscribe at admin RPC port to send webhooks for the transaction stream to a backend, on large(r) ledgers the endpoint receives fewer HTTP POSTs with TX information than the amount of transactions in a ledger. This change removes the hardcoded queue length to avoid dropping TX notifications for the admin-only command. In addition, the per-request TTL for outgoing RPC HTTP calls has been reduced from 10 minutes to 30 seconds.
This change introduces a new fix amendment (`fixPayChanV1`) that prevents the creation of new `PaymentChannelCreate` transaction with a `CancelAfter` time less than the current ledger time. It piggy backs off of fix1571.
Once the amendment is activated, creating a new `PaymentChannel` will require that if you specify the `CancelAfter` time/value, that value must be greater than or equal to the current ledger time.
Currently users can create a payment channel where the `CancelAfter` time is before the current ledger time. This results in the payment channel being immediately closed on the next PaymentChannel transaction.
This PR splits out `ledger_entry` tests into its own file (`LedgerEntry_test.cpp`) and alphabetizes the helper functions in `LedgerEntry.cpp`. These commits were split out of #5237 to make that PR a little more manageable, since these basic trivial changes are most of the diff. There is no code change, just moving code around.
Adds metric counters for the following P2P message types:
* Untrusted proposal and validation messages
* Duplicate proposal, validation and transaction messages
It’s possible for this to happen legitimately if a set of peers, including a validator, are connected in a cycle, and the latency and message processing time between those peers is significantly less than the latency between the validator and the last peer. It’s unlikely in the real world, but obviously easy to simulate with Antithesis.
The ci pipelines are constantly hitting Docker Hub's public rate limiting since increasing the number of jobs we're running. This change switches over to images hosted in GitHub's registry.
As part of import optimization, a transitive include had been removed that defined `BOOST_COMP_MSVC` on Windows. In unity builds, this definition was pulled in, but in non-unity builds it was not - causing a compilation error. An inspection of the Boost code revealed that we can just gate the statements by `_MS_VER` instead. A `#pragma message` is added to verify that the statement is only printed on Windows builds.
The main goal of this optimisation is memory reduction in SHAMapTreeNodes by introducing intrusive pointers instead of standard std::shared_ptr and std::weak_ptr.
In preparation for a potential reference fee change we would like to verify that fee change works as expected. The first step is to fix all unit tests to be able to work with different reference fee values.
- Detects if the consensus process is "stalled". If it is, then we can declare a
consensus and end successfully even if we do not have 80% agreement on
our proposal.
- "Stalled" is defined as:
- We have a close time consensus
- Each disputed transaction is individually stalled:
- It has been in the final "stuck" 95% requirement for at least 2
(avMIN_ROUNDS) "inner rounds" of phaseEstablish,
- and either all of the other trusted proposers or this validator, if proposing,
have had the same vote(s) for at least 4 (avSTALLED_ROUNDS) "inner
rounds", and at least 80% of the validators (including this one, if
appropriate) agree about the vote (whether yes or no).
- If we have been in the establish phase for more than 10x the previous
consensus establish phase's time, then consensus is considered "expired",
and we will leave the round, which sends a partial validation (indicating
that the node is moving on without validating). Two restrictions avoid
prematurely exiting, or having an extended exit in extreme situations.
- The 10x time is clamped to be within a range of 15s
(ledgerMAX_CONSENSUS) to 120s (ledgerABANDON_CONSENSUS).
- If consensus has not had an opportunity to walk through all avalanche
states (defined as not going through 8 "inner rounds" of phaseEstablish),
then ConsensusState::Expired is treated as ConsensusState::No.
- When enough nodes leave the round, any remaining nodes will see they've
fallen behind, and move on, too, generally before hitting the timeout. Any
validations or partial validations sent during this time will help the
consensus process bring the nodes back together.
This change removes the existing undefined behavior from `LogicError`, so we can be certain that there will be always a stacktrace.
De-referencing a null pointer is an old trick to generate `SIGSEGV`, which would typically also create a stacktrace. However it is also an undefined behaviour and compilers can do something else. A more robust way to create a stacktrace while crashing the program is to use `std::abort`, which we have also used in this location for a long time. If we combine the two, we might not get the expected behaviour - namely, the nullpointer deref followed by `std::abort`, as handled in certain compiler versions may not immediately cause a crash. We have observed stacktrace being wiped instead, and thread put in indeterminate state, then stacktrace created without any useful information.
The Trustline RPC `no_ripple` flag gets set depending on `lsfDefaultRipple` flag, which is not a flag of a trustline but of the account root. The `lsfDefaultRipple` flag does not provide any insight if this particular trust line has `lsfLowNoRipple` or `lsfHighNoRipple` flag set, so it should not be used here at all. This change simplifies the logic.
The `end_marker` is used to limit the range of ledger entries to fetch. If `end_marker` is less than `marker`, a crash can occur. This change adds an additional check.
Changes the error to `malformedAddress` for `permissioned_domain` in the `ledger_entry` rpc, when the account is not a string. This change makes it more clear to a user what is wrong with their request.
What the LoadManager class does is stall detection, which is not the same as deadlock detection. In the condition of severe CPU starvation, LoadManager will currently intentionally crash rippled reporting `LogicError: Deadlock detected`. This error message is misleading as the condition being detected is not a deadlock. This change fixes and refactors the code in response.
Requiring manual updates of numFeatures is an annoying manual process that is easily forgotten, and leads to frequent merge conflicts. This change takes advantage of the `XRPL_FEATURE` and `XRPL_FIX` macros, and adds a new `XRPL_RETIRE` macro to automatically set `numFeatures`.
The codebase is filled with includes that are unused, and which thus can be removed. At the same time, the files often do not include all headers that contain the definitions used in those files. This change uses clang-format and clang-tidy to clean up the includes, with minor manual intervention to ensure the code compiles on all platforms.
- PR #5228 added assert=TRUE and werr=TRUE CMake flags to the
build/action.yml script which is used by all CI jobs to build rippled,
ensuring those flags were always set. The assumption was that only the
CI jobs used that script, so any extra time cost was offset by the
benefit of the extra checks. That assumption was incorrect. That
script is used by other downstream projects. Therefore, those flags
have been moved into the individual CI jobs' "cmake-args" parameter
passed to build/action.yml. This will have the same effect for CI jobs
without any side effects.
Combine multiple related debug log data points into a single
message. Allows quick correlation of events that
previously were either not logged or, if logged, strewn
across multiple lines, making correlation difficult.
The Heartbeat Timer and consensus ledger accept processing
each have this capability.
Also guarantees that log entries will be written if the
node is a validator, regardless of log severity level.
Otherwise, the level of these messages is at INFO severity.
* Add logging for amendment voting decision process
* When counting "received validations" to determine quorum, count the number of validators actually voting, not the total number of possible votes.
The current comment in the example cfg file incorrectly mentions both "may" and "must". This change fixes this comment to clarify that the default port of hosts is 2459 and that specifying it is therefore optional. It further sets the default port to 2459 instead of the legacy 51235.
- Drop duplicate outgoing TMGetLedger messages per peer
- Allow a retry after 30s in case of peer or network congestion.
- Addresses RIPD-1870
- (Changes levelization. That is not desirable, and will need to be fixed.)
- Drop duplicate incoming TMGetLedger messages per peer
- Allow a retry after 15s in case of peer or network congestion.
- The requestCookie is ignored when computing the hash, thus increasing
the chances of detecting duplicate messages.
- With duplicate messages, keep track of the different requestCookies
(or lack of cookie). When work is finally done for a given request,
send the response to all the peers that are waiting on the request,
sending one message per peer, including all the cookies and
a "directResponse" flag indicating the data is intended for the
sender, too.
- Addresses RIPD-1871
- Drop duplicate incoming TMLedgerData messages
- Addresses RIPD-1869
- Improve logging related to ledger acquisition
- Class "CanProcess" to keep track of processing of distinct items
---------
Co-authored-by: Valentin Balaschenko <13349202+vlntb@users.noreply.github.com>
This change enhances the filtering in the ledger, ledger_data, and account_objects methods by also supporting filtering by the canonical name of the LedgerEntryType using case-insensitive matching.
Rewrites the code so that the lock is not held during the callback. Instead it locks twice, once before, and once after. This is safe due to the structure of the code, but is checked after the second lock. This allows mutex_ to be changed back to a regular mutex.
In PeerImpl.cpp, if the function is a message handler (onMessage) or called directly from a message handler, then it should use fee_, since when the handler returns (OnMessageEnd) then the charge function is called. If the function is not a message handler, such as a job queue item, it should remain charge.
- Rename the job in missing-commits.yml from "check" to "up_to_date",
because other jobs named "check" prevent merges, but this one should
not prevent merges. How else are branches going to get caught up?
- Move the job in instrumentation.yml to nix.yml, but keep it entirely
independent.
If the permissioned domains amendment XLS-80 is enabled before credentials XLS-70, then the permissioned domain users will not be able to match any credentials. The changes here prevent the creation of any permissioned domain objects if credentials are not enabled.
Make `simulate` RPC easier to use:
* Prevent the use of `seed`, `secret`, `seed_hex`, and `passphrase` fields (to avoid confusing with the signing methods).
* Add autofilling of the `NetworkID` field.
- Also get the branch name.
- Use rev-parse instead of describe to get a clean hash.
- Return the git hash and branch name in server_info for admin
connections.
- Include git hash and branch name on separate lines in --version.
- spec: XRPLF/XRPL-Standards#220
- amendment: "DeepFreeze"
- implemented deep freeze spec to allow token issuers to prevent currency holders from being able to acquire more of these tokens.
- in combination with normal freeze, deep freeze effectively prevents any balance trust line balance change of a currency holder (except direct issuer <-> holder payments).
- added 2 new invariant checks to verify that deep freeze cannot be enacted without normal freeze and transfer is not frozen.
- made some fixes to existing freeze handling.
Co-authored-by: Ed Hennis <ed@ripple.com>
Co-authored-by: Howard Hinnant <howard.hinnant@gmail.com>
- Fix an erroneous high fee penalty that peers could incur for sending
older transactions.
- Update to the fees charged for imposing a load on the server.
- Prevent the relaying of internal pseudo-transactions.
- Before: Pseudo-transactions received from a peer will fail the signature
check, even if they were requested (using TMGetObjectByHash), because
they have no signature. This causes the peer to be charge for an
invalid signature.
- After: Pseudo-transactions, are put into the global cache
(TransactionMaster) only. If the transaction is not part of
a TMTransactions batch, the peer is charged an unwanted data fee.
These fees will not be a problem in the normal course of operations,
but should dissuade peers from behaving badly by sending a bunch of
junk.
- Improve logging: include the reason for fees charged to a peer.
Co-authored-by: Ed Hennis <ed@ripple.com>
* Has more steps, but allows merges to develop to continue when a
beta / RC is pending, increasing developer velocity.
* Add a CI job to check that no reverse merges have been missed.
* Add some useful scripts in bin/git:
* Set up upstreams as expected for safer pushes
* Squash a bunch of branches
* Set the version number
* Resolves an issue introduced in #5111, which inadvertently removed the
-Wno-maybe-uninitialized compiler option from some xrpl.libxrpl
modules. This resulted in new "may be used uninitialized" build
warnings, first noticed in the "protocol" module. When compiling with
derr=TRUE, those warnings became errors, which made the build fail.
* Github CI actions will build with the assert and werr options turned
on. This will cause CI jobs to fail if a developer introduces a new
compiler warning, or causes an assert to fail in release builds.
* Includes the OS and compiler version in the linux dependencies jobs in
the "check environment" step.
* Translates the `unity` build option into `CMAKE_UNITY_BUILD` setting.
The LEDGER_ENTRY macro now takes an additional parameter, which makes it easier to avoid missing including the new field in jss.h and to the list of account_objects/ledger_data filters.
Replace Issue in STIssue with Asset. STIssue with MPTIssue is only used in MPT tests.
Will be used in Vault and in transactions with STIssue fields once MPT is integrated into DEX.
* Rename ASSERT to XRPL_ASSERT
* Upgrade to Anthithesis SDK 0.4.4, and use new 0.4.4 features
* automatic cast to bool, like assert
* Add instrumentation workflow to verify build with instrumentation enabled
Adds two CMake functions:
* add_module(library subdirectory): Declares an OBJECT "library" (a CMake abstraction for a collection of object files) with sources from the given subdirectory of the given library, representing a module. Isolates the module's headers by creating a subdirectory in the build directory, e.g. .build/tmp123, that contains just a symlink, e.g. .build/tmp123/basics, to the module's header directory, e.g. include/xrpl/basics, in the source directory, and putting .build/tmp123 (but not include/xrpl) on the include path of the module sources. This prevents the module sources from including headers not explicitly linked to the module in CMake with target_link_libraries.
* target_link_modules(library scope modules...): Links the library target to each of the module targets, and removes their sources from its source list (so they are not compiled and linked twice).
Uses these functions to separate and explicitly link modules in libxrpl:
Level 01: beast
Level 02: basics
Level 03: json, crypto
Level 04: protocol
Level 05: resource, server
* Copy Antithesis SDK version 0.4.0 to directory external/
* Add build option `voidstar` to enable instrumentation with Antithesis SDK
* Define instrumentation macros ASSERT and UNREACHABLE in terms of regular C assert
* Replace asserts with named ASSERT or UNREACHABLE
* Add UNREACHABLE to LogicError
* Document instrumentation macros in CONTRIBUTING.md
`STNumber` lets objects and transactions contain multiple fields for
quantities of XRP, IOU, or MPT without duplicating information about the
"issue" (represented by `STIssue`). It is a straightforward serialization of
the `Number` type that uniformly represents those quantities.
---------
Co-authored-by: John Freeman <jfreeman08@gmail.com>
Co-authored-by: Howard Hinnant <howard.hinnant@gmail.com>
* 2.2.2 changed functions acquireAsync and NetworkOPsImp::recvValidation to add an item to a collection under lock, unlock, do some work, then lock again to do remove the item. It will deadlock if an exception is thrown while adding the item - before unlocking.
* Replace ScopedUnlock with scope_unlock.
Move the newest information to the top, i.e., use reverse chronological order within each of the two sections ("API Versions" and "XRP Ledger server versions")
The page_size will soon be made configurable with #5135, making this
re-ordering necessary.
When opening SQLite connection, there are specific pragmas set with
commonPragmas.
In particular, PRAGMA journal_mode creates journal file and locks the
page_size; as of this commit, this sets the page size to the default
value of 4096. Coincidentally, the hardcoded page_size was also 4096, so
no issue was noticed.
Update book_changes RPC to reduce latency, add "validated" field, and accept shortcut strings (current, closed, validated) for ledger_index.
`"validated": true` indicates that the transaction has been included in a validated ledger so the result of the transaction is immutable.
Fix#5033Fix#5034Fix#5035Fix#5036
---------
Co-authored-by: Bronek Kozicki <brok@incorrekt.com>
When rippled initiates a connection to SQLite3, rippled sends a "PRAGMA"
statement defining the maximum number of pages allowed in the database.
Update the max_page_count so it is consistent with the default for newer
versions of SQLite3. Increasing max_page_count is critical for keeping
full history servers online.
Fix#5102
* Retry some failed RPC connections / commands in unit tests
* Remove orphaned `getAccounts` function
Co-authored-by: John Freeman <jfreeman08@gmail.com>
* upstream/master:
Set version to 2.2.2
Allow only 1 job queue slot for each validation ledger check
Allow only 1 job queue slot for acquiring inbound ledger.
Track latencies of certain code blocks, and log if they take too long
* refactor filtering of validations to specifically avoid
concurrent checkAccept() calls for the same validation ledger hash.
* Log when duplicate concurrent validation requests are filtered.
* RAII for containers that track concurrent validation requests.
* Log when duplicate concurrent inbound ledger are filtered.
* RAII for containers that track concurrent inbound ledger.
* Comment on when to asynchronously acquire inbound ledgers, which
is possible to be always OK, but should have further review.
* Other small logging changes
Co-authored-by: Ed Hennis <ed@ripple.com>
Implements a CI workflow that detects when a new version of libxrpl is
proposed, uploads it to artifactory under the `clio` channel and
notifies Clio's CI to check this newly proposed version.
* Add fixNFTokenPageLinks amendment:
It was discovered that under rare circumstances the links between
NFTokenPages could be removed. If this happens, then the
account_objects and account_nfts RPC commands under-report the
NFTokens owned by an account.
The fixNFTokenPageLinks amendment does the following to address
the problem:
- It fixes the underlying problem so no further broken links
should be created.
- It adds Invariants so, if such damage were introduced in the
future, an invariant would stop it.
- It adds a new FixLedgerState transaction that repairs
directories that were damaged in this fashion.
- It adds unit tests for all of it.
* `account_objects` returns an invalid field error if `type` is not supported.
This includes objects an account can't own, or which are unsupported by `account_objects`
* Includes:
* Amendments
* Directory Node
* Fee Settings
* Ledger Hashes
* Negative UNL
The names of the files should reflect the name of the Dir class.
Co-authored-by: Zack Brunson <Zshooter@gmail.com>
Co-authored-by: Ed Hennis <ed@ripple.com>
* fix CTID in tx command returns invalidParams on lowercase hex
* test mixed case and change auto to explicit type
* add header cctype because std::tolower is called
* remove unused local variable
* change test case comment from 'lowercase' to 'mixed case'
---------
Co-authored-by: Zack Brunson <Zshooter@gmail.com>
* Add feature / amendment "InvariantsV1_1"
* Adds invariant AccountRootsDeletedClean:
* Checks that a deleted account doesn't leave any directly
accessible artifacts behind.
* Always tests, but only changes the transaction result if
featureInvariantsV1_1 is enabled.
* Unit tests.
* Resolves#4638
* [FOLD] Review feedback from @gregtatcam:
* Fix unused variable warning
* Improve Invariant test const correctness
* [FOLD] Review feedback from @mvadari:
* Centralize the account keylet function list, and some optimization
* [FOLD] Some structured binding doesn't work in clang
* [FOLD] Review feedback 2 from @mvadari:
* Clean up and clarify some comments.
* [FOLD] Change InvariantsV1_1 to unsupported
* Will allow multiple PRs to be merged over time using the same amendment.
* fixup! [FOLD] Change InvariantsV1_1 to unsupported
* [FOLD] Update and clarify some comments. No code changes.
* Move CMake directory
* Rearrange sources
* Rewrite includes
* Recompute loops
* Fix merge issue and formatting
---------
Co-authored-by: Pretty Printer <cpp@ripple.com>
* fix "account_nfts" with unassociated marker returning issue
* create unit test for fixing nft page invalid marker not returning error
add more test
change test name
create unit test
* fix "account_nfts" with unassociated marker returning issue
* fix "account_nfts" with unassociated marker returning issue
* fix "account_nfts" with unassociated marker returning issue
* fix "account_nfts" with unassociated marker returning issue
* fix "account_nfts" with unassociated marker returning issue
* fix "account_nfts" with unassociated marker returning issue
* fix "account_nfts" with unassociated marker returning issue
* fix "account_nfts" with unassociated marker returning issue
* [FOLD] accumulated review suggestions
* move BEAST check out of lambda function
---------
Authored-by: Scott Schurr <scott@ripple.com>
* fixInnerObjTemplate2 amendment:
Apply inner object templates to all remaining (non-AMM)
inner objects.
Adds a unit test for applying the template to sfMajorities.
Other remaining inner objects showed no problems having
templates applied.
* Move CMake directory
* Rearrange sources
* Rewrite includes
* Recompute loops
---------
Co-authored-by: Pretty Printer <cpp@ripple.com>
Fix interactions between NFTokenOffers and trust lines.
Since the NFTokenAcceptOffer does not check the trust line that
the issuer receives as a transfer fee in the NFTokenAcceptOffer,
if the issuer deletes the trust line after NFTokenCreateOffer,
the trust line is created for the issuer by the
NFTokenAcceptOffer. That's fixed.
Resolves#4925.
Fixes issue #4937.
The fixReducedOffersV1 amendment fixed certain forms of offer
modification that could lead to blocked order books. Reduced
offers can block order books if the effective quality of the
reduced offer is worse than the quality of the original offer
(from the perspective of the taker). It turns out that, for
small values, the quality of the reduced offer can be
significantly affected by the rounding mode used during
scaling computations.
Issue #4937 identified an additional code path that modified
offers in a way that could lead to blocked order books. This
commit changes the rounding in that newly located code path so
the quality of the modified offer is never worse than the
quality of the offer as it was originally placed.
It is possible that additional ways of producing blocking
offers will come to light. Therefore there may be a future
need for a V3 amendment.
* Add trap_tx_hash command line option
This new option can be used only if replay is also enabled. It takes a transaction hash from the ledger loaded for replay, and will cause a specific line to be hit in Transactor.cpp, right before the selected transaction is applied.
Price Oracle data-series logic uses `unordered_map` to update the Oracle object.
This results in different servers disagreeing on the order of that hash table.
Consequently, the generated ledgers will have different hashes.
The fix uses `map` instead to guarantee the order of the token pairs
in the data-series.
Due to the rounding, LPTokenBalance of the last
Liquidity Provider (LP), might not match this LP's
trustline balance. This fix sets LPTokenBalance on
last LP withdrawal to this LP's LPToken trustline
balance.
Single path AMM offer has to factor in the transfer in rate
when calculating the upper bound quality and the quality function
because single path AMM's offer quality is not constant.
This fix factors in the transfer fee in
BookStep::adjustQualityWithFees().
* Fix AMM offer rounding and low quality LOB offer blocking AMM:
A single-path AMM offer with account offer on DEX, is always generated
starting with the takerPays first, which is rounded up, and then
the takerGets, which is rounded down. This rounding ensures that the pool's
product invariant is maintained. However, when one of the offer's side
is XRP, this rounding can result in the AMM offer having a lower
quality, potentially causing offer generation to fail if the quality
is lower than the account's offer quality.
To address this issue, the proposed fix adjusts the offer generation process
to start with the XRP side first and always rounds it down. This results
in a smaller offer size, improving the offer's quality. Regardless if the offer
has XRP or not, the rounding is done so that the offer size is minimized.
This change still ensures the product invariant, as the other generated
side is the exact result of the swap-in or swap-out equations.
If a liquidity can be provided by both AMM and LOB offer on offer crossing
then AMM offer is generated so that it matches LOB offer quality. If LOB
offer quality is less than limit quality then generated AMM offer quality
is also less than limit quality and the offer doesn't cross. To address
this issue, if LOB quality is better than limit quality then use LOB
quality to generate AMM offer. Otherwise, don't use the quality to generate
AMM offer. In this case, limitOut() function in StrandFlow limits
the out amount to match strand's quality to limit quality and consume
maximum AMM liquidity.
Rounding in the payment engine is causing an assert to sometimes fire
with "dust" amounts. This is causing issues when running debug builds of
rippled. This issue will be addressed, but the assert is no longer
serving its purpose.
I am resigning from my role as maintainer of the `rippled` codebase.
Please update repository permissions accordingly, prior to merging this pull request.
Thanks to everyone who has contributed, especially those whom I had the opportunity to closely collaborate with.
The AMM has an invariant for swaps where:
new_balance_1*new_balance_2 >= old_balance_1*old_balance_2
Due to rounding, this invariant could sometimes be violated (although by
very small amounts).
This patch introduces an amendment `fixAMMRounding` that changes the
rounding to always favor the AMM. Doing this should maintain the
invariant.
Co-authored-by: Bronek Kozicki
Co-authored-by: thejohnfreeman
It can be difficult to make transaction breaking changes to low level
code because the low level code does not have access to a ledger and the
current activated amendments in that ledger (the "rules"). This patch
adds global access to the current ledger rules as a `std::optional`. If
the optional is not seated, then there is no active transaction.
The `rotateWithLock` function holds a lock while it calls a callback
function that's passed in by the caller. This is a problematic design
that needs to be used very carefully. In this case, at least one caller
passed in a callback that eventually relocks the mutex on the same
thread, causing UB (a deadlock was observed). The caller was from
SHAMapStoreImpl, and it called `clearCaches`. This `clearCaches` can
potentially call `fetchNodeObject`, which tried to relock the mutex.
This patch resolves the issue by changing the mutex type to a
`recursive_mutex`. Ideally, the code should be rewritten so it doesn't
hold the mutex during the callback and the mutex should be changed back
to a regular mutex.
Co-authored-by: Ed Hennis <ed@ripple.com>
* Amend `.codecov.yml` to disable coverage reporting of test sources
and explicitly set most parameters
* Increase codecov upload retry time to 210s (from 35s)
* Upgrade gcovr adding support for more coverage formats (lcov, clover, jacoco)
* Upgrade github actions in coverage workflow
* Explicitly disable codecov plugins (also removing `gcov` coverage, which is not
correctly handled by codecov https://github.com/codecov/feedback/issues/334)
This amendment, `fixPreviousTxnID`, adds `PreviousTxnID` and
`PreviousTxnLgrSequence` as fields to all ledger objects that did
not already have them included (`DirectoryNode`, `Amendments`,
`FeeSettings`, `NegativeUNL`, and `AMM`). This makes it much easier
to go through the history of these ledger objects.
Github Actions for the build/test jobs (nix.yml, mac.yml, windows.yml) will only run on branches that build packages (develop, release, master), and branches with names starting with "ci/". This is intended as a compromise between disabling CI jobs on personal forks entirely, and having the jobs run as a free-for-all. Note that it will not affect PR jobs at all.
A large synthetic offer was not handled correctly in the payment engine.
This patch fixes that issue and introduces a new invariant check while
processing synthetic offers.
When calculating reward shares, the amount should always be rounded
down. If the `fixUniversalNumber` amendment is not active, this works
correctly. If it is not active, then the amount is incorrectly rounded
up. This patch introduces an amendment so it will be rounded down.
This fixes a case where a peer can desync under a certain timing
circumstance--if it reaches a certain point in consensus before it receives
proposals.
This was noticed under high transaction volumes. Namely, when we arrive at the
point of deciding whether consensus is reached after minimum establish phase
duration but before having received any proposals. This could be caused by
finishing the previous round slightly faster and/or having some delay in
receiving proposals. Existing behavior arrives at consensus immediately after
the minimum establish duration with no proposals. This causes us to desync
because we then close a non-validated ledger. The change in this PR causes us to
wait for a configured threshold before making the decision to arrive at
consensus with no proposals. This allows validators to catch up and for brief
delays in receiving proposals to be absorbed. There should be no drawback since,
with no proposals coming in, we needn't be in a huge rush to jump ahead.
Remove the zaphod.alloy.ee hubs from the bootstrap and default configuration after 5 years. It has been an honor to run these servers, but it is now time for another entity to step into this role.
The zaphod servers will be taken offline in a phased manner keeping all those who have peering arrangements informed.
These would be the preferred attributes of a boostrap set of hubs:
1. Commitment to run the hubs for a minimum of 2 years
2. Highly available
3. Geographically dispersed
4. Secure and up to date
5. Committed to ensure that peering information is kept private
We do not currently enforce that incoming peer connection does not have
remote_endpoint which is already used (either by incoming or outgoing
connection), hence already stored in slots_. If we happen to receive a
connection from such a duplicate remote_endpoint, it will eventually result in a
crash (when disconnecting) or weird behavior (when updating slot state), as a
result of an apparently matching remote_endpoint in slots_ being used by a
different connection.
This amendment fixes an edge case where an empty DID object can be
created. It adds an additional check to ensure that DIDs are
non-empty when created, and returns a `tecEMPTY_DID` error if the DID
would be empty.
The witness server makes heavily use of the `account_tx` RPC command. Perf
testing showed that the SQL query used by `account_tx` became unacceptably slow
when the DB was large and there was a `marker` parameter. The plan for the query
showed only indexed reads. This appears to be an issue with the internal SQLite
optimizer. This patch rewrote the query to use `UNION` instead of `OR` and
significantly improves performance. See RXI-896 and RIPD-1847 for more details.
- Update container for Doxygen workflow. Matches Linux workflow, with newer GLIBC version required by newer actions.
- Fixes macOS workflow to install and configure Conan correctly. Still fails on tests, but that does not seem attributable to the workflow.
* telENV_RPC_FAILED is a new code, reserved exclusively
for unit tests when RPC fails. This will
make those types of errors distinct and easier to test
for when expected and/or diagnose when not.
* Output RPC command result when result is not expected.
We are currently using old version 0.6.2 of `xxhash`, as a verbatim copy and paste of its header file `xxhash.h`. Switch to the more recent version 0.8.2. Since this version is in Conan Center (and properly protects its ABI by keeping the state object incomplete), add it as a Conan requirement. Switch to the SIMD instructions (in the new `XXH3` family) supported by the new version.
This algorithm is about an order of magnitude faster than the existing
algorithm (about 10x faster for encoding and about 15x faster for
decoding - including the double hash for the checksum). The algorithms
use gcc's int128 (fast MS version will have to wait, in the meantime MS
falls back to the slow code).
* It is now an invariant that all constructed Public Keys are valid,
non-empty and contain 33 bytes of data.
* Additionally, the memory footprint of the PublicKey class is reduced.
The size_ data member is declared as static.
* Distinguish and identify the PublisherList retrieved from the local
config file, versus the ones obtained from other validators.
* Fixes#2942
The compilation fails due to an issue in the initializer list
of an optional argument, which holds a vector of pairs.
The code compiles correctly on earlier gcc versions, but fails on gcc 13.
Implement native support for Price Oracles.
A Price Oracle is used to bring real-world data, such as market prices,
onto the blockchain, enabling dApps to access and utilize information
that resides outside the blockchain.
Add Price Oracle functionality:
- OracleSet: create or update the Oracle object
- OracleDelete: delete the Oracle object
To support this functionality add:
- New RPC method, `get_aggregate_price`, to calculate aggregate price for a token pair of the specified oracles
- `ltOracle` object
The `ltOracle` object maintains:
- Oracle Owner's account
- Oracle's metadata
- Up to ten token pairs with the scaled price
- The last update time the token pairs were updated
Add Oracle unit-tests
* Commit 01c37fe introduced a change to the STTx unit test where a local
"defaultRules" object was created with a temporary inline "presets"
value provided to the ctor. Rules::Impl stores a const ref to the
presets provided to the ctor. This particular call provided an inline
temp variable, which goes out of scope as soon as the object is
created. On Windows, attempting to use the presets (e.g. via the
enabled() function) causes an access violation, which crashes the test
run.
* An audit of the code indicates that all other instances of Rules use
the Application's config.features list, which will have a sufficient
lifetime.
Add `STObject` constructor to explicitly set the inner object template.
This allows certain AMM transactions to apply in the same ledger:
There is no issue if the trading fee is greater than or equal to 0.01%.
If the trading fee is less than 0.01%, then:
- After AMM create, AMM transactions must wait for one ledger to close
(3-5 seconds).
- After one ledger is validated, all AMM transactions succeed, as
appropriate, except for AMMVote.
- The first AMMVote which votes for a 0 trading fee in a ledger will
succeed. Subsequent AMMVote transactions which vote for a 0 trading
fee will wait for the next ledger (3-5 seconds). This behavior repeats
for each ledger.
This has no effect on the ultimate correctness of AMM. This amendment
will allow the transactions described above to succeed as expected, even
if the trading fee is 0 and the transactions are applied within one
ledger (block).
Prior to this commit, `port_grpc` could not be added to the [server]
stanza. Instead of validating gRPC IP/Port/Protocol information in
ServerHandler, validate grpc port info in GRPCServer constructor. This
should not break backwards compatibility.
gRPC-related config info must be in a section (stanza) called
[port_gprc].
* Close#4015 - That was an alternate solution. It was decided that with
relaxed validation, it is not necessary to rename port_grpc.
* Fix#4557
These headers are required in the xrpl Conan package in order for
xbridge witness server (xbwd) to build. This change to libxrpl may help
any dependents of libxrpl. This addition does not change any C++ code.
Without this amendment, an NFTokenAcceptOffer transaction can succeed
even when the NFToken recipient does not have sufficient reserves for
the new NFTokenPage. This allowed accounts to accept NFT sell offers
without having a sufficient reserve. (However, there was no issue in
brokered mode or when a buy offer is involved.)
Instead, the transaction should fail with `tecINSUFFICIENT_RESERVE` as
appropriate. The `fixNFTokenReserve` amendment adds checks in the
NFTokenAcceptOffer transactor to check if the OwnerCount changed. If it
did, then it checks the new reserve requirement.
Fix#4679
Use consistent platform-agnostic library names on all platforms.
Fix an issue that prevents dependents like validator-keys-tool from
linking to libxrpl on Windows.
It is bad practice to change the binary base name depending on the
platform. CMake already manipulates the base name into a final name that
fits the conventions of the platform. Linkers accept base names on the
command line and then look for conventional names on disk.
* Disable the Windows CI unit tests "allowed to fail" workaround which
was previously introduced in #4596.
* The runner hardware was upgraded, and the unit tests have been passing
since then.
Update to #4849, using a workaround for spurious codecov upload errors.
Spurious codecov upload errors are expected in public repos which rely
on PRs via forks. Retrying uploads is a decent and easy workaround.
Clients subscribed to `transactions` over WebSocket are being
disconnected because the traffic exceeds the default `send_queue_limit`
of 100.
This commit changes the default configuration, not the default in code.
Fix#4866
Resolves a warning that was emitted from the clang compiler. Switches
usage of the sprintf function to the recommended snprintf function.
Warning was observed in Apple clang version 15.0.0 (clang-1500.0.40.1).
Fix#4569
* Add logging for Application.cpp sweep()
* Improve lifetime management of ledger objects (`SLE`s)
* Only store SLE digest in CachedView; get SLEs from CachedSLEs
* Also force release of last ledger used for path finding if there are
no path finding requests to process
* Count more ST objects (derive from `CountedObject`)
* Track CachedView stats in CountedObjects
* Rename the CachedView counters
* Fix the scope of the digest lookup lock
Before this patch, if you asked "is it caching?" It was always caching.
Prevent WebSocket connections from trying to close twice.
The issue only occurs in debug builds (assertions are disabled in
release builds, including published packages), and when the WebSocket
connections are unprivileged. The assert (and WRN log) occurs when a
client drives up the resource balance enough to be forcibly disconnected
while there are still messages pending to be sent.
Thanks to @lathanbritz for discovering this issue in #4822.
This reverts commit 002893f280.
There were two files with conflicts in the automated revert:
- src/ripple/rpc/impl/RPCHelpers.h and
- src/test/rpc/JSONRPC_test.cpp
Those files were manually resolved.
Workaround for compilation errors with gcc-13 and other compilers
relying on `libstdc++` version 13. This is temporary until actual fix is
available for us to use: https://github.com/boostorg/beast/pull/2682
Some boost.beast files (which we do use) rely on an old gcc-12 behaviour
where `#include <cstdint>` was not needed even though types from this
header were used. This was broken by a change in libstdc++ version 13:
https://gcc.gnu.org/gcc-13/porting_to.html#header-dep-changes
The necessary fix was implemented in boost.beast, however it is not yet
available. Until it is available, we can use this workaround to enable
compilation of `rippled` with gcc-13, clang-16, etc.
* Revert "Optimize calculation of close time to avoid impasse and minimize gratuitous proposal changes (#4760)"
This reverts commit 8ce85a9750.
* Revert "Several changes to improve Consensus stability: (#4505)"
This reverts commit f259cc1ab6.
* Add missing include
---------
Co-authored-by: seelabs <scott.determan@yahoo.com>
* Optimize the calculation of close time to avoid
impasse and minimize gratuitous proposal changes.
* git apply clang-format.patch
* Review (Howard) fixes.
* Review fix for impasse discovered by John.
* Review fixes (comments) from John.
* Scott S review fixes. Also clang-format.
* Promote API version 2 to supported
* Switch command line to API version 1
* Fix LedgerRequestRPC test
* Remove obsolete tx_account method
This method is not implemented, the only parts which are removed are related to command-line parsing
* Fix RPCCall test
* Reduce diff size, small test improvements
* Minor fixes
* Support for the mold linker
* [fold] handle case where both mold and gold are installed
* [fold] Use first non-default linker
* Fix TransactionEntry_test
* Fix AccountTx_test
---------
Co-authored-by: seelabs <scott.determan@yahoo.com>
* Support for the mold linker (#4807)
* Promote API version 2 to supported (#4803)
* Promote API version 2 to be supported
* Switch the command line to API version 1
* Fix LedgerRequestRPC test
* Remove obsolete tx_account method
This method is not implemented, the only parts which are removed are related to command-line parsing
* Fix RPCCall test
* Reduce diff size, small test improvements
* Minor fixes
* Support for the mold linker
* Fix TransactionEntry_test
* Fix AccountTx_test
---------
Co-authored-by: seelabs <scott.determan@yahoo.com>
* Update Linux smoketest distros (#4813)
* Fix 2.0 regression in tx method with binary output (#4812)
* Fix binary output from tx method
* Formatting fix
* Minor test improvement
* Minor test improvements
* Optimize calculation of close time to avoid impasse and minimize gratuitous proposal changes (#4760)
* Optimize the calculation of close time to avoid
impasse and minimize gratuitous proposal changes.
* git apply clang-format.patch
* Scott S review fixes. Also clang-format.
* Set version to 2.0.0-rc2
---------
Co-authored-by: manoj <mdoshi@ripple.com>
Co-authored-by: Scott Determan <scott.determan@yahoo.com>
Co-authored-by: Bronek Kozicki <brok@incorrekt.com>
Co-authored-by: Michael Legleux <legleux@users.noreply.github.com>
Co-authored-by: Mark Travis <mtrippled@users.noreply.github.com>
* Optimize the calculation of close time to avoid
impasse and minimize gratuitous proposal changes.
* git apply clang-format.patch
* Review (Howard) fixes.
* Review fix for impasse discovered by John.
* Review fixes (comments) from John.
* Scott S review fixes. Also clang-format.
* Promote API version 2 to supported
* Switch command line to API version 1
* Fix LedgerRequestRPC test
* Remove obsolete tx_account method
This method is not implemented, the only parts which are removed are related to command-line parsing
* Fix RPCCall test
* Reduce diff size, small test improvements
* Minor fixes
* Support for the mold linker
* [fold] handle case where both mold and gold are installed
* [fold] Use first non-default linker
* Fix TransactionEntry_test
* Fix AccountTx_test
---------
Co-authored-by: seelabs <scott.determan@yahoo.com>
* Remove include <ranges>
* Formatting fix
* Output for subscriptions
* Output from sign, submit etc.
* Output from ledger
* Output from account_tx
* Output from transaction_entry
* Output from tx
* Store close_time_iso in API v2 output
* Add small APIv2 unit test for subscribe
* Add unit test for transaction_entry
* Add unit test for tx
* Remove inLedger from API version 2
* Set ledger_hash and ledger_index
* Move isValidated from RPCHelpers to LedgerMaster
* Store closeTime in LedgerFill
* Time formatting fix
* additional tests for Subscribe unit tests
* Improved comments
* Rename mInLedger to mLedgerIndex
* Minor fixes
* Set ledger_hash on closed ledger, even if not validated
* Update API-CHANGELOG.md
* Add ledger_hash, ledger_index to transaction_entry
* Fix validated and close_time_iso in account_tx
* Fix typos
* Improve getJson for Transaction and STTx
* Minor improvements
* Replace class enum JsonOptions with struct
We may consider turning this into a general-purpose template and using it elsewhere
* simplify the extraction of transactionID from Transaction object
* Remove obsolete comments
* Unconditionally set validated in account_tx output
* Minor improvements
* Minor fixes
---------
Co-authored-by: Chenna Keshava <ckeshavabs@gmail.com>
The command line API still uses `apiMaximumSupportedVersion`.
The unit test RPCs use `apiMinimumSupportedVersion` if unspecified.
Context:
- #4568
- #4552
With clang 15, an unused-but-set-variable warning was emitted:
PostgresDatabase.cpp:178:14: warning: variable 'expNumResults' set but
not used [-Wunused-but-set-variable]
uint32_t expNumResults = 1;
Introduce the `fixFillOrKill` amendment.
Fix an edge case occurring when an offer with `tfFillOrKill` set (but
without `tfSell` set) fails to cross an offer with a better rate. If
`tfFillOrKill` is set, then the owner must receive the full TakerPays.
Without this amendment, an offer fails if the entire `TakerGets` is not
spent. With this amendment, when `tfSell` is not set, the entire
`TakerGets` does not have to be spent.
For details about OfferCreate, see: https://xrpl.org/offercreate.htmlFix#4684
---------
Co-authored-by: Scott Schurr <scott@ripple.com>
Remove dependency on `<ranges>` header, since it is not implemented by
all compilers which we want to support.
This code change only affects unit tests.
Resolve https://github.com/XRPLF/rippled/issues/4787
Remove `tx_history` and `ledger_header` methods from API version 2.
Update `RPC::Handler` to allow for methods (or method implementations)
to be API version specific. This partially resolves#4727. We can now
store multiple handlers with the same name, as long as they belong to
different (non-overlapping) API versions. This necessarily impacts the
handler lookup algorithm and its complexity; however, there is no
performance loss on x86_64 architecture, and only minimal performance
loss on arm64 (around 10ns). This design change gives us extra
flexibility evolving the API in the future, including other parts of
#4727.
In API version 2, `tx_history` and `ledger_header` are no longer
recognised; if they are called, `rippled` will return error
`unknownCmd`
Resolve#3638Resolve#3539
Using the "Amount" field in Payment transactions can cause incorrect
interpretation. There continue to be problems from the use of this
field. "Amount" is rarely the correct field to use; instead,
"delivered_amount" (or "DeliveredAmount") should be used.
Rename the "Amount" field to "DeliverMax", a less misleading name. With
api_version: 2, remove the "Amount" field from Payment transactions.
- Input: "DeliverMax" in `tx_json` is an alias for "Amount"
- sign
- submit (in sign-and-submit mode)
- submit_multisigned
- sign_for
- Output: Add "DeliverMax" where transactions are provided by the API
- ledger
- tx
- tx_history
- account_tx
- transaction_entry
- subscribe (transactions stream)
- Output: Remove "Amount" from API version 2
Fix#3484Fix#3902
The unity build speeds up compilation by bundling multiple source files
into one larger file. This reduces Windows CI build time by up to 50%.
As described in #4596, the automatic Windows builds take a very long
time. Unity builds are significantly faster - currently about 45 min,
much closer to the typical MacOS (35-40 minutes) and nix (~30 minutes)
run times.
This is intended as a stopgap solution until a more resourced and
reliable runner is available.
No C++ code was changed. This only affects CI.
Add a new RPC / WS call for `server_definitions`, which returns an
SDK-compatible `definitions.json` (binary enum definitions) generated by
the server. This enables clients/libraries to dynamically work with new
fields and features, such as ones that may become available on side
chains. Clients query `server_definitions` on a node from the network
they want to work with, and immediately know how to speak that node's
binary "language", even if new features are added to it in the future
(as long as there are no new serialized types that the software doesn't
know how to serialize/deserialize).
Example:
```js
> {"command": "server_definitions"}
< {
"result": {
"FIELDS": [
[
"Generic",
{
"isSerialized": false,
"isSigningField": false,
"isVLEncoded": false,
"nth": 0,
"type": "Unknown"
}
],
[
"Invalid",
{
"isSerialized": false,
"isSigningField": false,
"isVLEncoded": false,
"nth": -1,
"type": "Unknown"
}
],
[
"ObjectEndMarker",
{
"isSerialized": false,
"isSigningField": true,
"isVLEncoded": false,
"nth": 1,
"type": "STObject"
}
],
...
```
Close#3657
---------
Co-authored-by: Richard Holland <richard.holland@starstone.co.nz>
Implement native support for W3C DIDs.
Add a new ledger object: `DID`.
Add two new transactions:
1. `DIDSet`: create or update the `DID` object.
2. `DIDDelete`: delete the `DID` object.
This meets the requirements specified in the DID v1.0 specification
currently recommended by the W3C Credentials Community Group.
The DID format for the XRP Ledger conforms to W3C DID standards.
The objects can be created and owned by any XRPL account holder.
The transactions can be integrated by any service, wallet, or application.
It might be possible for the server code to indirect through certain
`end()` iterators. While a debug build would catch this problem with
`assert()`s, a release build would crash. If there are problems in this
area in the future, it is best to get a definitive indication of the
nature of the error regardless of whether it's a debug or release build.
To accomplish this, these `assert`s are converted into `LogicError`s
that will produce a reasonable error message when they fire.
In Windows, we need to call `python` in order for the `pip` upgrade
command to work.
This changes the GitHub Actions Windows CI job to use the correct
command to upgrade PIP, fixing this error:
```
ERROR: To modify pip, please run the following command:
C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe -m pip install --upgrade pip
```
A future task is to make job run on heavy Windows runners so that it
doesn't take so long.
Context: #4596
The assert is saying that the only reason `pathFinder` would be null is
if the request was aborted (connection dropped, etc.). That's what
`continueCallback()` checks. But that is very clearly not true if you
look at `getPathFinder`, which calls `findPaths`, which can return false
for many reasons.
Fix#4744
Update the nix CI runner. This commit does not modify any source code
files. The unix builds were successful, but the binaries were not
uploaded to the internal artifactory. This PR borrows an idea from
@ximinez to attempt to fix this issue.
After successful authentication, the `outcome` variable contains a
string. In the upload step, we are checking if outcome == 'success' as a
prerequisite for uploading the binary.
This commit updates the contents of the `outcome` variable.
Artifactory support was added to the `nix` builds with #4556. This
extends that support to the Windows build. Now the Windows build works;
CI will build and test a Windows release build. This only affects CI and
does not change any C++ code.
* Copy the remote setup step outcome fix from #4716 discussion
* Allow the Windows job to succeed if tests fail:
* Currently the tests do not always pass, even on a single threaded
run on the GitHub runners. So we are using parallel runs and mark
the test step as allowed to fail (continue-on-error).
* At this point, it's more important that the build succeeds than that
the tests succeed, because:
* We've got plenty of test coverage on the other jobs.
* Test failures are much rarer than build failures because of
cross-platform issues.
* Having a test failure locally doesn't interrupt a workflow nearly as
much as a build failure.
Note that Conan Center cannot hold the binaries we need. They do not
build the configurations we need, and they will not add them.
## Future Tasks
This introduces a new bottleneck since the build and test takes over an
hour. Speed up the job by:
* Making this job run on heavy Windows runners.
* Increasing the number of hardware threads.
P2P link compression is a feature added in 1.6.0 by #3287.
https://xrpl.org/enable-link-compression.html
If the default changes in the future - for example, as currently
proposed by #4387 - the comment will be updated at that time.
Fix#4656
Context: The `DisallowIncoming` amendment provides an option to block
incoming trust lines from reaching your account. The
asfDisallowIncomingTrustline AccountSet Flag, when enabled, prevents any
incoming trust line from being created. However, it was too restrictive:
it would block an issuer from authorizing a trust line, even if the
trust line already exists. Consider:
1. Issuer sets asfRequireAuth on their account.
2. User sets asfDisallowIncomingTrustline on their account.
3. User submits tx to SetTrust to Issuer.
At this point, without `fixDisallowIncomingV1` active, the issuer would
not be able to authorize the trust line.
The `fixDisallowIncomingV1` amendment, once activated, allows an issuer
to authorize a trust line even after the user sets the
asfDisallowIncomingTrustline flag, as long as the trust line already
exists.
When a new transactor is added, there are several places in applySteps
that need to be modified. This patch refactors the code so only one
function needs to be modified.
Make transactions and pseudo-transactions share the same commonFields
again. This regularizes the code in a nice way.
While this technically allows pseudo-transactions to have a
TicketSequence field, pseudo-transactions are only ever constructed by
code paths that don't add such a field, so this is not a transaction
processing change. It may be possible to add a separate check to ensure
TicketSequence (and other fields that don't make sense on
pseudo-transactions) are never added to pseudo-transactions, but that
should not be necessary. (TicketSequence is not the only common field
that can not and does not appear in pseudo-transactions.) Note:
TicketSequence is already documented as a common field.
Related: #4637Fix#4714
Address a stack-use-after-scope issue when using rvalues with
`soci::use`. Replace rvalues with lvalues to ensure the scope extends
beyond the end of the expression.
The issue arises from `soci` taking a reference to the rvalue without
copying its value or extending its lifetime. `soci` references rvalues
in `soci::use_container` and then the address in `soci_use_type`. For
types like `int`, memory access post-lifetime is unlikely to cause
issues. However, for `std::string`, the backing heap memory can be freed
and potentially reused, leading to a potential segmentation fault.
This was detected on x86_64 using clang-15 with asan. asan confirms
resolution of the issue.
Fix#4675
Update minimum compiler requirement for building the codebase. The
feature "using enum" is required. This feature was introduced in C++20.
Updating the C++ compiler to version 11 or later fixes this error:
```
Building CXX object CMakeFiles/xrpl_core.dir/src/ripple/protocol/impl/STAmount.cpp.o
/build/ripple/binary/src/ripple/protocol/impl/STAmount.cpp: In lambda function:
/build/ripple/binary/src/ripple/protocol/impl/STAmount.cpp:1577:15: error: expected nested-name-specifier before 'enum'
1577 | using enum Number::rounding_mode;
| ^~~~
```
Fix#4693
Modify the `XChainBridge` amendment.
Before this patch, two door accounts on the same chain could could own
the same bridge spec (of course, one would have to be the issuer and one
would have to be the locker). While this is silly, it does not violate
any bridge invariants. However, on further review, if we allow this then
the `claim` transactions would need to change. Since it's hard to see a
use case for two doors to own the same bridge, this patch disallows
it. (The transaction will return tecDUPLICATE).
Amendment "flapping" (an amendment repeatedly gaining and losing
majority) usually occurs when an amendment is on the verge of gaining
majority, and a validator not in favor of the amendment goes offline or
loses sync. This fix makes two changes:
1. The number of validators in the UNL determines the threshold required
for an amendment to gain majority.
2. The AmendmentTable keeps a record of the most recent Amendment vote
received from each trusted validator (and, with `trustChanged`, stays
up-to-date when the set of trusted validators changes). If no
validation arrives from a given validator, then the AmendmentTable
assumes that the previously-received vote has not changed.
In other words, when missing an `STValidation` from a remote validator,
each server now uses the last vote seen. There is a 24 hour timeout for
recorded validator votes.
These changes do not require an amendment because they do not impact
transaction processing, but only the threshold at which each individual
validator decides to propose an EnableAmendment pseudo-transaction.
Fix#4350
Fix the Windows build by using `unsigned int` (instead of `uint`).
The error, introduced by #4618, looks something like:
rpc\impl\RPCHelpers.h(299,5): error C2061: syntax error: identifier
'uint' (compiling source file app\ledger\Ledger.cpp)
A few methods, including `book_offers`, take currency codes as
parameters. The XRPL doesn't care if the letters in those codes are
lowercase or uppercase, as long as they come from an alphabet defined
internally. rippled doesn't care either, when they are submitted in a
hex representation. When they are submitted in an ASCII string
representation, rippled, but not XRPL, is more restrictive, preventing
clients from interacting with some currencies already in the XRPL.
This change gets rippled out of the way and lets clients submit currency
codes in ASCII using the full alphabet.
Fixes#4112
Currently, the `BUILD.md` instructions suggest using `.build` as the
build directory, so this change helps to reduce confusion.
An alternative would be to instruct developers to add `/.build/` to
`.git/info/exclude` or to user-level `.gitignore` (although the latter
is very intrusive). However, it is being added here because it is a good
practice to have a sensible default that's consistent with the build
instructions.
Clean up the peer-to-peer protocol start/close sequences by introducing
START_PROTOCOL and GRACEFUL_CLOSE messages, which sync inbound/outbound
peer send/receive. The GRACEFUL_CLOSE message differentiates application
and link layer failures.
* Introduce the `InboundHandoff` class to manage inbound peer
instantiation and synchronize the send/receive protocol messages
between peers.
* Update `OverlayImpl` to utilize the `InboundHandoff` class to manage
inbound handshakes.
* Update `PeerImp` for improved handling of protocol messages.
* Modify the `Message` class for better maintainability.
* Introduce P2P protocol version `2.3`.
gateway_balances
* When `account` does not exist in the ledger, return `actNotFound`
* (Previously, a normal response was returned)
* Fix#4290
* When required field(s) are missing, return `invalidParams`
* (Previously, `invalidHotWallet` was incorrectly returned)
* Fix#4548
channel_authorize
* When the specified `key_type` is invalid, return `badKeyType`
* (Previously, `invalidParams` was returned)
* Fix#4289
Since these are breaking changes, they apply only to API version 2.
Supersedes #4577
Copy the new code to `src/secp256k1` without changes:
`src/secp256k1` is identical to bitcoin-core/secp256k1@acf5c55 (v0.3.2).
We could consider changing to a Git submodule, though that would require
changes to the build instructions because we are not using submodules
anywhere else.
Remove the `verify` and `message` function declarations. The explicit
instantiation requests could not be completed because there were no
implementations for those two member functions. It is helpful that the
Microsoft (MSVC) compiler on Windows appears to be strict when it comes
to template instantiation.
This resolves the warning:
XChainAttestations.h(450): warning C4661: 'bool
ripple::XChainAttestationsBase<ripple::XChainClaimAttestation>::verify(void)
const': no suitable definition provided for explicit template
instantiation request
* For example, without this change, to run the TxQ tests, must specify
`--unittest=TxQ1,TxQ2` on the command line. With this change, can use
`--unittest=TxQ`, and both will be run.
* An exact match will prevent any further partial matching.
* This could have some side effects for different tests with a common
name beginning. For example, NFToken, NFTokenBurn, NFTokenDir. This
might be useful. If not, the shorter-named test(s) can be renamed. For
example, NFToken to NFTokens.
* Split the NFToken, NFTokenBurn, and Offer test classes. Potentially speeds
up parallel tests by a factor of 5.
A bridge connects two blockchains: a locking chain and an issuing
chain (also called a mainchain and a sidechain). Both are independent
ledgers, with their own validators and potentially their own custom
transactions. Importantly, there is a way to move assets from the
locking chain to the issuing chain and a way to return those assets from
the issuing chain back to the locking chain: the bridge. This key
operation is called a cross-chain transfer. A cross-chain transfer is
not a single transaction. It happens on two chains, requires multiple
transactions, and involves an additional server type called a "witness".
A bridge does not exchange assets between two ledgers. Instead, it locks
assets on one ledger (the "locking chain") and represents those assets
with wrapped assets on another chain (the "issuing chain"). A good model
to keep in mind is a box with an infinite supply of wrapped assets.
Putting an asset from the locking chain into the box will release a
wrapped asset onto the issuing chain. Putting a wrapped asset from the
issuing chain back into the box will release one of the existing locking
chain assets back onto the locking chain. There is no other way to get
assets into or out of the box. Note that there is no way for the box to
"run out of" wrapped assets - it has an infinite supply.
Co-authored-by: Gregory Popovitch <greg7mdp@gmail.com>
For the `account_tx` and `noripple_check` methods, perform input
validation for optional parameters such as "binary", "forward",
"strict", "transactions". Previously, when these parameters had invalid
values (e.g. not a bool), no error would be returned. Now, it returns an
`invalidParams` error.
* This updates the behavior to match Clio
(https://github.com/XRPLF/clio).
* Since this is potentially a breaking change, it only applies to
requests specifying api_version: 2.
* Fix#4543.
* Verify accepted ledger becomes validated, and retry
with a new consensus transaction set if not.
* Always store proposals.
* Track proposals by ledger sequence. This helps slow peers catch
up with the rest of the network.
* Acquire transaction sets for proposals with future ledger sequences.
This also helps slow peers catch up.
* Optimize timer delay for establish phase to wait based on how
long validators have been sending proposals. This also helps slow
peers to catch up.
* Fix impasse achieving close time consensus.
* Don't wait between open and establish phases.
Add new transaction submission API field, "sync", which
determines behavior of the server while submitting transactions:
1) sync (default): Process transactions in a batch immediately,
and return only once the transaction has been processed.
2) async: Put transaction into the batch for the next processing
interval and return immediately.
3) wait: Put transaction into the batch for the next processing
interval and return only after it is processed.
Minor refactor to `TxFormats.cpp`:
- Rename `commonFields` to `pseudoCommonFields` (since it is the common fields
that all pseudo-transactions need)
- Add a new static variable, `commonFields`, which represents all the common
fields that non-pseudo transactions need (essentially everything that
`pseudoCommonFields` contains, plus `sfTicketSequence`)
This makes it harder to accidentally leave out `sfTicketSequence` in a new
transaction.
- Verify "check", used to retrieve a Check object, is a string.
- Verify "nft_page", used to retrieve an NFT Page, is a string.
- Verify "index", used to retrieve any type of ledger object by its
unique ID, is a string.
- Verify "directory", used to retrieve a DirectoryNode, is a string or
an object.
This change only impacts api_version 2 since it is a breaking change.
https://xrpl.org/ledger_entry.htmlFix#4550
* In namespace ripple, introduces get_name function that takes a
std:🧵:native_handle_type and returns a std::string.
* In namespace ripple, introduces get_name function that takes a
std::thread or std::jthread and returns a std::string.
* In namespace ripple::this_thread, introduces get_name function
that takes no parameters and returns the name of the current
thread as a std::string.
* In namespace ripple::this_thread, introduces set_name function
that takes a std::string_view and sets the name of the current
thread.
* Intended to replace the beast utilities setCurrentThreadName
and getCurrentThreadName.
- Update amm_info to fetch AMM by amm account id.
- This is an additional way to retrieve an AMM object.
- Alternatively, AMM can still be fetched by the asset pair as well.
- Add owner directory entry for AMM object.
Context:
- Add back the AMM object directory entry, which was deleted by #4626.
- This fixes `account_objects` for `amm` type.
Modify two error cases in AMMBid transactor to return `tecINTERNAL` to
more clearly indicate that these errors should not be possible unless
operating in unforeseen circumstances. It likely indicates a bug.
The log level has been updated to `fatal()` since it indicates a
(potentially network-wide) unexpected condition when either of these
errors occurs.
Details:
The two specific transaction error cases changed are:
- `tecAMM_BALANCE` - In this case, this error (total LP Tokens
outstanding is lower than the amount to be burned for the bid) is a
subset of the case where the user doesn't have enough LP Tokens to pay
for the bid. When this case is reached, the bidder's LP Tokens balance
has already been checked first. The user's LP Tokens should always be
a subset of total LP Tokens issued, so this should be impossible.
- `tecINSUFFICIENT_PAYMENT` - In this case, the amount to be refunded as
a result of the bid is greater than the price paid for the auction
slot. This should never occur unless something is wrong with the math
for calculating the refund amount.
Both error cases in question are "defense in depth" measures meant to
protect against making things worse if the code has already reached a
state that is supposed to be impossible, likely due to a bug elsewhere.
Such "shouldn't ever occur" checks should use an error code that
categorically indicates a larger problem. This is similar to how
`tecINVARIANT_FAILED` is a warning sign that something went wrong and
likely could've been worse, but since there isn't an Invariant Check
applying here, `tecINTERNAL` is the appropriate error code.
This is "debatably" a transaction processing change since it could
hypothetically change how transactions are processed if there's a bug we
don't know about.
* Verify accepted ledger becomes validated, and retry
with a new consensus transaction set if not.
* Always store proposals.
* Track proposals by ledger sequence. This helps slow peers catch
up with the rest of the network.
* Acquire transaction sets for proposals with future ledger sequences.
This also helps slow peers catch up.
* Optimize timer delay for establish phase to wait based on how
long validators have been sending proposals. This also helps slow
peers to catch up.
* Fix impasse achieving close time consensus.
* Don't wait between open and establish phases.
Add new transaction submission API field, "sync", which
determines behavior of the server while submitting transactions:
1) sync (default): Process transactions in a batch immediately,
and return only once the transaction has been processed.
2) async: Put transaction into the batch for the next processing
interval and return immediately.
3) wait: Put transaction into the batch for the next processing
interval and return only after it is processed.
Introduce a new variadic template helper function, `forAllApiVersions`,
that accepts callables to execute a set of functions over a range of
versions - from RPC::apiMinimumSupportedVersion to RPC::apiBetaVersion.
This avoids the duplication of code.
Context: #4552
- "Rename" the type `LedgerInfo` to `LedgerHeader` (but leave an alias
for `LedgerInfo` to not yet disturb existing uses). Put it in its own
public header, named after itself, so that it is more easily found.
- Move the type `Fees` and NFT serialization functions into public
(installed) headers.
- Compile the XRPL and gRPC protocol buffers directly into `libxrpl` and
install their headers. Fix the Conan recipe to correctly export these
types.
Addresses change (2) in
https://github.com/XRPLF/XRPL-Standards/discussions/121.
For context: This work supports Clio's dependence on libxrpl. Clio is
just an example consumer. These changes should benefit all current and
future consumers.
---------
Co-authored-by: cyan317 <120398799+cindyyan317@users.noreply.github.com>
Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
Improve the checking of the path lengths during Payments. Previously,
the code that did the check of the payment path lengths was sometimes
executed, but without any effect. This changes it to only check when it
matters, and to not make unnecessary copies of the path vectors.
Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
- Replace custom popcnt16 implementation with std::popcount from C++20
- Maintain compatibility with older compilers and MacOS by providing a
conditional compilation fallback to __builtin_popcount and a lookup
table method
- Move and inline related functions within SHAMapInnerNode for
performance and readability
Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
When an AMM account is deleted, the owner directory entries must be
deleted in order to ensure consistent ledger state.
* When deleting AMM account:
* Clean up AMM owner dir, linking AMM account and AMM object
* Delete trust lines to AMM
* Disallow `CheckCreate` to AMM accounts
* AMM cannot cash a check
* Constrain entries in AuthAccounts array to be accounts
* AuthAccounts is an array of objects for the AMMBid transaction
* SetTrust (TrustSet): Allow on AMM only for LP tokens
* If the destination is an AMM account and the trust line doesn't
exist, then:
* If the asset is not the AMM LP token, then fail the tx with
`tecNO_PERMISSION`
* If the AMM is in empty state, then fail the tx with `tecAMM_EMPTY`
* This disallows trustlines to AMM in empty state
* Add AMMID to AMM root account
* Remove lsfAMM flag and use sfAMMID instead
* Remove owner dir entry for ltAMM
* Add `AMMDelete` transaction type to handle amortized deletion
* Limit number of trust lines to delete on final withdraw + AMMDelete
* Put AMM in empty state when LPTokens is 0 upon final withdraw
* Add `tfTwoAssetIfEmpty` deposit option in AMM empty state
* Fail all AMM transactions in AMM empty state except special deposit
* Add `tecINCOMPLETE` to indicate that not all AMM trust lines are
deleted (i.e. partial deletion)
* This is handled in Transactor similar to deleted offers
* Fail AMMDelete with `tecINTERNAL` if AMM root account is nullptr
* Don't validate for invalid asset pair in AMMDelete
* AMMWithdraw deletes AMM trust lines and AMM account/object only if the
number of trust lines is less than max
* Current `maxDeletableAMMTrustLines` = 512
* Check no directory left after AMM trust lines are deleted
* Enable partial trustline deletion in AMMWithdraw
* Add `tecAMM_NOT_EMPTY` to fail any transaction that expects an AMM in
empty state
* Clawback considerations
* Disallow clawback out of AMM account
* Disallow AMM create if issuer can claw back
This patch applies to the AMM implementation in #4294.
Acknowledgements:
Richard Holland and Nik Bougalis for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the project code and urge researchers to
responsibly disclose any issues they may find.
To report a bug, please send a detailed report to:
bugs@xrpl.org
Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
Introduce an API Changelog, which logs the changes that have been made
to the API.
Without this changelog, it is difficult to keep track of the changes and
additions that are made to the API. While all changes are surfaced in
PRs and Release Notes, these are mixed in with other non-API-affecting
changes. PRs that affect the API have the `API Change` label applied,
but it is hard to identify which PRs have been included in each release.
Furthermore, some API changes will take effect based on `api_version`
(starting with rippled version 1.12, which will introduce `api_version:
2`), while others are based on the `rippled` version.
The API Changelog clarifies the details of the changes in a way that is
easily understood by API consumers, and breaks down the changes to be
clear which ones are gated by api_version (versus `rippled` version).
From now on, all PR authors are responsible for updating the API
Changelog according to the additions/changes that their PR makes to the
APIs.
Reason for this change is here XRPLF/XRPL-Standards#119
We would want to be explicit that this flag is exclusively for trustline. For new token types(eg. CFT), they will not utilize this flag for clawback, instead, they will turn clawback on/off on the token-level, which is more versatile.
Use the most recent versions in ConanCenter.
* Due to a bug in Clang 16, you may get a compile error:
"call to 'async_teardown' is ambiguous"
* A compiler flag workaround is documented in `BUILD.md`.
* At this time, building this with gcc 13 may require editing some files
in `.conan/data`
* A patch to support gcc13 may be added in a later PR.
---------
Co-authored-by: Scott Schurr <scott@ripple.com>
Add AMM functionality:
- InstanceCreate
- Deposit
- Withdraw
- Governance
- Auctioning
- payment engine integration
To support this functionality, add:
- New RPC method, `amm_info`, to fetch pool and LPT balances
- AMM Root Account
- trust line for each IOU AMM token
- trust line to track Liquidity Provider Tokens (LPT)
- `ltAMM` object
The `ltAMM` object tracks:
- fee votes
- auction slot bids
- AMM tokens pair
- total outstanding tokens balance
- `AMMID` to AMM `RootAccountID` mapping
Add new classes to facilitate AMM integration into the payment engine.
`BookStep` uses these classes to infer if AMM liquidity can be consumed.
The AMM formula implementation uses the new Number class added in #4192.
IOUAmount and STAmount use Number arithmetic.
Add AMM unit tests for all features.
AMM requires the following amendments:
- featureAMM
- fixUniversalNumber
- featureFlowCross
Notes:
- Current trading fee threshold is 1%
- AMM currency is generated by: 0x03 + 152 bits of sha256{cur1, cur2}
- Current max AMM Offers is 30
---------
Co-authored-by: Howard Hinnant <howard.hinnant@gmail.com>
Improve error handling for ledger_entry by returning an "invalidParams"
error when one or more request fields are specified incorrectly, or one
or more required fields are missing.
For example, if none of of the following fields is provided, then the
API should return an invalidParams error:
* index, account_root, directory, offer, ripple_state, check, escrow,
payment_channel, deposit_preauth, ticket
Prior to this commit, the API returned an "unknownOption" error instead.
Since the error was actually due to invalid parameters, rather than
unknown options, this error was misleading.
Since this is an API breaking change, the "invalidParams" error is only
returned for requests using api_version: 2 and above. To maintain
backward compatibility, the "unknownOption" error is still returned for
api_version: 1.
Related: #4573Fix#4303
- Previously, mulDiv had `std::pair<bool, uint64_t>` as the output type.
- This is an error-prone interface as it is easy to ignore when
overflow occurs.
- Using a return type of `std::optional` should decrease the likelihood
of ignoring overflow.
- It also allows for the use of optional::value_or() as a way to
explicitly recover from overflow.
- Include limits.h header file preprocessing directive in order to
satisfy gcc's numeric_limits incomplete_type requirement.
Fix#3495
---------
Co-authored-by: John Freeman <jfreeman08@gmail.com>
* Update the `account_info` API so that the `allowClawback` flag is
included in the response.
* The proposed `Clawback` amendement added an `allowClawback` flag in
the `AccountRoot` object.
* In the API response, under `account_flags`, there is now an
`allowClawback` field with a boolean (`true` or `false`) value.
* For reference, the XLS-39 Clawback implementation can be found in
#4553Fix#4588
Enhance security during the build process:
* The '-fstack-protector' flag enables stack protection for preventing
buffer overflow vulnerabilities. If an attempt is made to overflow the
buffer, the program will terminate, thus protecting the integrity of
the stack.
* The '-Wl,-z,relro,-z,now' linker flag enables Read-only Relocations
(RELRO), a feature that helps harden the binary against certain types
of exploits, particularly those that involve overwriting the Global
Offset Table (GOT).
* This flag is only set for Linux builds, due to compatibility issues
with apple-clang.
* The `relro` option makes certain sections of memory read-only after
initialization to prevent them from being overwritten, while `now`
ensures that all dynamic symbols are resolved immediately on program
start, reducing the window of opportunity for attacks.
* Add a new YAML file (.pre-commit-config.yaml) to set up pre-commit
hook for clang-format
* The pre-commit hook is opt-in and needs to be installed in order to
run automatically
* Update CONTRIBUTING.md with instructions on how to set up and use the
clang-format linter
Automating the process of running clang-format before committing code
helps to save time by removing the need to fix formatting issues later.
This is a tooling improvement and doesn't change C++ code.
* Update the version of the checkout action (for GitHub Actions) in
`clang-format.yml` and `levelization.yml`.
* The previous version, v2, was raising deprecation warnings due to
its reliance on Node.js 12.
* The latest checkout action version, v3, uses Node.js 16.
When requesting `account_info` with an invalid `signer_lists` value, the
API should return an "invalidParams" error.
`signer_lists` should have a value of type boolean. If it is not a
boolean, then it is invalid input. The response now indicates that.
* This is an API breaking change, so the change is only reflected for
requests containing `"api_version": 2`
* Fix#4539
The debug packages were named with the extension ".ddeb", but due to a
bug in Artifactory, they need to have the ".deb" extension. Debug symbol
packages with ".ddeb" extensions are not indexed, and thus are not
visible in apt clients.
* Fix the issue by renaming the debug packages in the build script.
* Use GCC-11 and update GCC Conan profile.
* This software requires GCC 11 and C++20. However, reporting mode is
built with C++17.
This is a quick band-aid to fix the build. Later, it will be better to
remove this package-building code.
For context, a Debian (deb) package contains bundled software and
resources necessary for installing and managing software on a
Debian-based system, including Ubuntu and derivatives.
- Use powers of two to clearly indicate the bitmask
- Replace bitmask with explicit if-conditions to better indicate predicates
Change enum values to be powers of two (fix#3417) #4239
Implement the simplified condition evaluation
removes the complex bitwise and(&) operator
Implement the second proposed solution in Nik Bougalis's comment - Software does not distinguish between different Conditions (Version: 1.5) #3417 (comment)
I have tested this code change by performing RPC calls with the commands server_info, server_state, peers and validation_info. These commands worked as expected.
Certain inputs for the AccountTx method should return an error. In other
words, an invalid request from a user or client now results in an error
message.
Since this can change the response from the API, it is an API breaking
change. This commit maintains backward compatibility by keeping the
existing behavior for existing requests. When clients specify
"api_version": 2, they will be able to get the updated error messages.
Update unit tests to check the error based on the API version.
* Fix#4288
* Fix#4545
* Commits 0b812cd (#4427) and 11e914f (#4516) conflict. The first added
references to `ServerHandlerImp` in files outside of that class's
organizational unit (which is technically incorrect). The second
removed `ServerHandlerImp`, but was not up to date with develop. This
results in the build failing.
* Fixes the build by changing references to `ServerHandlerImp` to
the more correct `ServerHandler`.
Rename `ServerHandlerImp` to `ServerHandler`. There was no other
ServerHandler definition despite the existence of a header suggesting
that there was.
This resolves a piece of historical confusion in the code, which was
identified during a code review.
The changes in the diff may look more extensive than they actually are.
The contents of `impl/ServerHandlerImp.h` were merged into
`ServerHandler.h`, making the latter file appear to have undergone
significant modifications. However, this is a non-breaking refactor that
only restructures code.
Fix the libxrpl library target for consumers using Conan.
* Fix installation issues and update includes.
* Update requirements in the Conan package info.
* libxrpl requires openssl::crypto.
(Conan is a software package manager for C++.)
Replace hand-rolled code with std::from_chars for better
maintainability.
The C++ std::from_chars function is intended to be as fast as possible,
so it is unlikely to be slower than the code it replaces. This change is
a net gain because it reduces the amount of hand-rolled code.
Apply a minor cleanup in `TypedField`:
* Remove a non-working and unused move constructor.
* Constrain the remaining constructor to not be overly generic enough as
to be used as a copy or move constructor.
There is now an Artifactory (thanks @shichengripple001 and team!) to
hold dependency binaries for the builds.
* Rewrite the `nix` workflow to use it and cut the time down to a mere
21 minutes.
* This workflow should continue to work (just more slowly) for forks
that do not have access to the Artifactory.
Enhance the /crawl endpoint by publishing WebSocket/RPC ports in the
server_info response. The function processing requests to the /crawl
endpoint actually calls server_info internally, so this change enables a
server to advertise its WebSocket/RPC port(s) to peers via the /crawl
endpoint. `grpc` and `peer` ports are included as well.
The new `ports` array contains objects, each containing a `port` for the
listening port (number string), and an array `protocol` listing the
supported protocol(s).
This allows crawlers to build a richer topology without needing to
port-scan nodes. For non-admin users (including peers), the info about
*admin* ports is excluded.
Also increase test coverage for RPC ServerInfo.
Fix#2837.
Curtail the occurrence of order books that are blocked by reduced offers
with the implementation of the fixReducedOffersV1 amendment.
This commit identifies three ways in which offers can be reduced:
1. A new offer can be partially crossed by existing offers, so the new
offer is reduced when placed in the ledger.
2. An in-ledger offer can be partially crossed by a new offer in a
transaction. So the in-ledger offer is reduced by the new offer.
3. An in-ledger offer may be under-funded. In this case the in-ledger
offer is scaled down to match the available funds.
Reduced offers can block order books if the effective quality of the
reduced offer is worse than the quality of the original offer (from the
perspective of the taker). It turns out that, for small values, the
quality of the reduced offer can be significantly affected by the
rounding mode used during scaling computations.
This commit adjusts some rounding modes so that the quality of a reduced
offer is always at least as good (from the taker's perspective) as the
original offer.
The amendment is titled fixReducedOffersV1 because additional ways of
producing reduced offers may come to light. Therefore, there may be a
future need for a V2 amendment.
* Enable api_version 2, which is currently in beta. It is expected to be
marked stable by the next stable release.
* This does not change any defaults.
* The only existing tests changed were one that set the same flag, which
was now redundant, and a couple that tested versioning explicitly.
- Resolve gcc compiler warning:
AccountObjects.cpp:182:47: warning: redundant move in initialization [-Wredundant-move]
- The std::move() operation on trivially copyable types may generate a
compile warning in newer versions of gcc.
- Remove extraneous header (unused imports) from a unit test file.
Fix a bug in the `NODE_SIZE` auto-detection feature in `Config.cpp`.
Specifically, this patch corrects the calculation for the total amount
of RAM available, which was previously returned in bytes, but is now
being returned in units of the system's memory unit. Additionally, the
patch adjusts the node size based on the number of available hardware
threads of execution.
Misaligned load and store operations are supported by both Intel and ARM
CPUs. However, in C++, these operations are undefined behavior (UB).
Substituting these operations with a `memcpy` fixes this UB. The
compiled assembly code is equivalent to the original, so there is no
performance penalty to using memcpy.
For context: The unaligned load and store operations fixed here were
originally introduced in the slab allocator (#4218).
This assert was put in the wrong place, but it only triggers if shards
are configured. This change moves the assert to the right place and
updates it to ensure correctness.
The assert could be hit after the server downloads some shards. It may
be necessary to restart after the shards are downloaded.
Note that asserts are normally checked only in debug builds, so release
packages should not be affected.
Introduced in: #4319 (66627b26cf)
Global variables in different TUs are initialized in an undefined order.
At least one global variable was accessing a global switchover variable.
This caused the switchover variable to be accessed in an uninitialized
state.
Since the switchover is always explicitly set before transaction
processing, this bug can not effect transaction processing, but could
effect unit tests (and potentially the value of some global variables).
Note: at the time of this patch the offending bug is not yet in
production.
Three new fields are added to the `Tx` responses for NFTs:
1. `nftoken_id`: This field is included in the `Tx` responses for
`NFTokenMint` and `NFTokenAcceptOffer`. This field indicates the
`NFTokenID` for the `NFToken` that was modified on the ledger by the
transaction.
2. `nftoken_ids`: This array is included in the `Tx` response for
`NFTokenCancelOffer`. This field provides a list of all the
`NFTokenID`s for the `NFToken`s that were modified on the ledger by
the transaction.
3. `offer_id`: This field is included in the `Tx` response for
`NFTokenCreateOffer` transactions and shows the OfferID of the
`NFTokenOffer` created.
The fields make it easier to track specific tokens and offers. The
implementation includes code (by @ledhed2222) from the Clio project to
extract NFTokenIDs from mint transactions.
The API would allow seeds (and public keys) to be used in place of
accounts at several locations in the API. For example, when calling
account_info, you could pass `"account": "foo"`. The string "foo" is
treated like a seed, so the method returns `actNotFound` (instead of
`actMalformed`, as most developers would expect). In the early days,
this was a convenience to make testing easier. However, it allows for
poor security practices, so it is no longer a good idea. Allowing a
secret or passphrase is now considered a bug. Previously, it was
controlled by the `strict` option on some methods. With this commit,
since the API does not interpret `account` as `seed`, the option
`strict` is no longer needed and is removed.
Removing this behavior from the API is a [breaking
change](https://xrpl.org/request-formatting.html#breaking-changes). One
could argue that it shouldn't be done without bumping the API version;
however, in this instance, there is no evidence that anyone is using the
API in the "legacy" way. Furthermore, it is a potential security hole,
as it allows users to send secrets to places where they are not needed,
where they could end up in logs, error messages, etc. There's no reason
to take such a risk with a seed/secret, since only the public address is
needed.
Resolves: #3329, #3330, #4337
BREAKING CHANGE: Remove non-strict account parsing (#3330)
On macOS, if you have not installed something that depends on `xz`, then your
system may lack `lzma`, resulting in a build error similar to:
```
Downloading libarchive-3.6.0.tar.xz completed [6250.61k]
libarchive/3.6.0:
ERROR: libarchive/3.6.0: Error in source() method, line 120
get(self, **self.conan_data["sources"][self.version], strip_root=True)
ReadError: file could not be opened successfully:
- method gz: ReadError('not a gzip file')
- method bz2: ReadError('not a bzip2 file')
- method xz: CompressionError('lzma module is not available')
- method tar: ReadError('invalid header')
```
The solution is to ensure that `lzma` is installed by installing `xz`.
SOCI is the C++ database access library. The SOCI recipe was updated in
Conan Center Index (CCI), and it breaks for our choice of options. This
breakage occurs when you build with a fresh Conan cache (e.g. when you
submit a PR, or delete `~/.conan/data`).
* Add a custom Conan recipe for SOCI v4.0.3
* Update dependency building to handle exporting and installing Snappy
and SOCI
* Fix workflows to use custom SOCI recipe
* Update BUILD.md to include instruction for exporting the SOCI Conan
recipe:
* `conan export external/soci soci/4.0.3@`
This solution has been verified on Ubuntu 20.04 and macOS.
Context:
* There is a compiler error that the `sqlite3.h` header is not available
when building soci.
* When package B depends on package A, it finds the pieces it needs by
importing the Package Configuration File (PCF) that Conan generates
for package A.
* Read the CMake written by package B to check that it is importing
the PCF correctly and linking its exports correctly.
* Since this can be difficult, it is often more efficient to check
https://github.com/conan-io/conan-center-index/issues for package B
to see if anyone else has seen a similar problem.
* One of the issues points to a problem area in soci's CMake. To
confirm the diagnosis, review soci's CMake (after any patches are
applied) in the Conan build directory `build/$buildId/src/`.
* Review the Conan-generated PCF in
`build/$buildId/build/$buildType/generators/`.
* In this case, the problem was likely (re)introduced by
https://github.com/conan-io/conan-center-index/pull/17026
* If there is a problem in the source or in the Conan recipe, the
fastest fix is to copy the recipe and either:
* Add a source patch to fix any problems in the source.
* Change the recipe to fix any problems in the recipe.
* In this case, this can be done by finding soci's Conan recipe at
https://github.com/conan-io/conan-center-index/tree/master/recipes/soci
and then copying the `all` directory as `external/$packageName` in our
project. Then, make any changes.
* Test packages can be removed from the recipe folder as they are not
needed.
* If adding a patch in the `patches` directory, add a description for
it to `conandata.yml`.
* Since `conanfile.py` has no `version` property on the recipe class,
builders need to pass a version on the command line (like they do
for our `snappy` recipe).
* Add an example command to `BUILD.md`.
Future work: It may make sense to refer to recipes by revision, by
checking in a lockfile.
This change makes progress on the plan in #4371. It does not replicate
the full [matrix] implemented in #3851, but it does replicate the 1.ii
section of the Linux matrix. It leverages "heavy" self-hosted runners,
and demonstrates a repeatable pattern for future matrices.
[matrix]: d794a0f3f1/.github/README.md (continuous-integration)
Address issues related to the removal of `std::{u,bi}nary_function` in
C++17 and some warnings with Clang 16. Some warnings appeared with the
upgrade to Apple clang version 14.0.3 (clang-1403.0.22.14.1).
- `std::{u,bi}nary_function` were removed in C++17. They were empty
classes with a few associated types. We already have conditional code
to define the types. Just make it unconditional.
- libc++ checks a cast in an unevaluated context to see if a type
inherits from a binary function class in the standard library, e.g.
`std::equal_to`, and this causes an error when the type privately
inherits from such a class. Change these instances to public
inheritance.
- We don't need a middle-man for the empty base optimization. Prefer to
inherit directly from an empty class than from
`beast::detail::empty_base_optimization`.
- Clang warns when all the uses of a variable are removed by conditional
compilation of assertions. Add a `[[maybe_unused]]` annotation to
suppress it.
- As a drive-by clean-up, remove commented code.
See related work in #4486.
If `--quorum` setting is present on the command line, use the specified
value as the minimum quorum. This allows for the use of a potentially
fork-unsafe quorum, but it is sometimes necessary for small and test
networks.
Fix#4488.
---------
Co-authored-by: RichardAH <richard.holland@starstone.co.nz>
Add instructions for installing rippled using the package managers APT
and YUM. Some steps were adapted from xrpl.org.
---------
Co-authored-by: Michael Legleux <mlegleux@ripple.com>
Newer compilers, such as Apple Clang 15.0, have removed `std::result_of`
as part of C++20. The build instructions provided a fix for this (by
adding a preprocessor definition), but the fix was broken.
This fixes the fix by:
* Adding the `conf` prefix for tool configurations (which had been
forgotten).
* Passing `extra_b2_flags` to `boost` package to fix its build.
* Define `BOOST_ASIO_HAS_STD_INVOKE_RESULT` in order to build boost
1.77 with a newer compiler.
Add a `NetworkID` field to help prevent replay attacks on and from
side-chains.
The new field must be used when the server is using a network id > 1024.
To preserve legacy behavior, all chains with a network ID less than 1025
retain the existing behavior. This includes Mainnet, Testnet, Devnet,
and hooks-testnet. If `sfNetworkID` is present in any transaction
submitted to any of the nodes on one of these chains, then
`telNETWORK_ID_MAKES_TX_NON_CANONICAL` is returned.
Since chains with a network ID less than 1025, including Mainnet, retain
the existing behavior, there is no need for an amendment.
The `NetworkID` helps to prevent replay attacks because users specify a
`NetworkID` field in every transaction for that chain.
This change introduces a new UINT32 field, `sfNetworkID` ("NetworkID").
There are also three new local error codes for transaction results:
- `telNETWORK_ID_MAKES_TX_NON_CANONICAL`
- `telREQUIRES_NETWORK_ID`
- `telWRONG_NETWORK`
To learn about the other transaction result codes, see:
https://xrpl.org/transaction-results.html
Local error codes were chosen because a transaction is not necessarily
malformed if it is submitted to a node running on the incorrect chain.
This is a local error specific to that node and could be corrected by
switching to a different node or by changing the `network_id` on that
node. See:
https://xrpl.org/connect-your-rippled-to-the-xrp-test-net.html
In addition to using `NetworkID`, it is still generally recommended to
use different accounts and keys on side-chains. However, people will
undoubtedly use the same keys on multiple chains; for example, this is
common practice on other blockchain networks. There are also some
legitimate use cases for this.
A `app.NetworkID` test suite has been added, and `core.Config` was
updated to include some network_id tests.
The `Ledger` class contains two `SHAMap` instances: the state and
transaction maps. Previously, the maps were dynamically allocated using
`std::make_shared` despite the fact that they did not require lifetime
management separate from the lifetime of the `Ledger` instance to which
they belong.
The two `SHAMap` instances are now regular member variables. Some smart
pointers and dynamic memory allocation was avoided by using stack-based
alternatives.
Commit 3 of 3 in #4218.
The `SHAMapItem` class contains a variable-sized buffer that
holds the serialized data associated with a particular item
inside a `SHAMap`.
Prior to this commit, the buffer for the serialized data was
allocated separately. Coupled with the fact that most instances
of `SHAMapItem` were wrapped around a `std::shared_ptr` meant
that an instantiation might result in up to three separate
memory allocations.
This commit switches away from `std::shared_ptr` for `SHAMapItem`
and uses `boost::intrusive_ptr` instead, allowing the reference
count for an instance to live inside the instance itself. Coupled
with using a slab-based allocator to optimize memory allocation
for the most commonly sized buffers, the net result is significant
memory savings. In testing, the reduction in memory usage hovers
between 400MB and 650MB. Other scenarios might result in larger
savings.
In performance testing with NFTs, this commit reduces memory size by
about 15% sustained over long duration.
Commit 2 of 3 in #4218.
When instantiating a large amount of fixed-sized objects on the heap
the overhead that dynamic memory allocation APIs impose will quickly
become significant.
In some cases, allocating a large amount of memory at once and using
a slabbing allocator to carve the large block into fixed-sized units
that are used to service requests for memory out will help to reduce
memory fragmentation significantly and, potentially, improve overall
performance.
This commit introduces a new `SlabAllocator<>` class that exposes an
API that is _similar_ to the C++ concept of an `Allocator` but it is
not meant to be a general-purpose allocator.
It should not be used unless profiling and analysis of specific memory
allocation patterns indicates that the additional complexity introduced
will improve the performance of the system overall, and subsequent
profiling proves it.
A helper class, `SlabAllocatorSet<>` simplifies handling of variably
sized objects that benefit from slab allocations.
This commit incorporates improvements suggested by Greg Popovitch
(@greg7mdp).
Commit 1 of 3 in #4218.
Add Clio-specific JSS constants to ensure a common vocabulary of
keywords in Clio and this project. By providing visibility of the full
API keyword namespace, it reduces the likelihood of developers
introducing minor variations on names used by Clio, or unknowingly
claiming a keyword that Clio has already claimed. This change moves this
project slightly away from having only the code necessary for running
the core server, but it is a step toward the goal of keeping this
server's and Clio's APIs similar. The added JSS constants are annotated
to indicate their relevance to Clio.
Clio can be found here: https://github.com/XRPLF/clio
Signed-off-by: ledhed2222 <ledhed2222@users.noreply.github.com>
- Include NFTokenPages in account_objects to make it easier to
understand an account's Owner Reserve and simplify app development.
- Update related tests and documentation.
- Fix#4347.
For info about the Owner Reserve, see https://xrpl.org/reserves.html
---------
Co-authored-by: Scott Schurr <scott@ripple.com>
Co-authored-by: Ed Hennis <ed@ripple.com>
Change `ledger_data` to return an empty list when all entries are
filtered out.
When the `type` field is specified for the `ledger_data` method, it is
possible that no objects of the specified type are found. This can even
occur if those objects exist, but not in the section that the server
checked while serving your request. Previously, the `state` field of the
response has the value `null`, instead of an empty array `[]`. By
changing this to an empty array, the response is the same data type so
that clients can handle it consistently.
For example, in Python, `for entry in state` should now work correctly.
It would raise an exception if `state` is `null` (or `None`).
This could break client code that explicitly checks for null. However,
this fix aligns the response with the documentation, where the `state`
field is an array.
Fix#4392.
Previously, the object `account_data` in the `account_info` response
contained a single field `Flags` that contains flags of an account. API
consumers must perform bitwise operations on this field to retrieve the
account flags.
This change adds a new object, `account_flags`, at the top level of the
`account_info` response `result`. The object contains relevant flags of
the account. This makes it easier to write simple code to check a flag's
value.
The flags included may depend on the amendments that are enabled.
Fix#2457.
Log exception messages at several locations.
Previously, these were locations where an exception was caught, but the
exception message was not logged. Logging the exception messages can be
useful for analysis or debugging. The additional logging could have a
small negative performance impact.
Fix#3213.
In the release notes (current and historical), there is a link to the
`Builds` directory. By creating `Builds/README.md` with a link to
`BUILD.md`, it is easier to find the build instructions.
* Create the FeeSettings object in genesis ledger.
* Initialize with default values from the config. Removes the need to
pass a Config down into the Ledger initialization functions, including
setup().
* Drop the undocumented fee config settings in favor of the [voting]
section.
* Fix#3734.
* If you previously used fee_account_reserve and/or fee_owner_reserve,
you should change to using the [voting] section instead. Example:
```
[voting]
account_reserve=10000000
owner_reserve=2000000
```
* Because old Mainnet ledgers (prior to 562177 - yes, I looked it up)
don't have FeeSettings, some of the other ctors will default them to
the config values before setup() tries to load the object.
* Update default Config fee values to match Mainnet.
* Fix unit tests:
* Updated fees: Some tests are converted to use computed values of fee
object, but the default Env config was also updated to fix the rest.
* Unit tests that check the structure of the ledger have updated
hashes and counts.
Add the ability to mark amendments as obsolete. There are some known
amendments that should not be voted for because they are broken (or
similar reasons).
This commit marks four amendments as obsolete:
1. `CryptoConditionsSuite`
2. `NonFungibleTokensV1`
3. `fixNFTokenDirV1`
4. `fixNFTokenNegOffer`
When an amendment is `Obsolete`, voting for the amendment is prevented.
A determined operator can still vote for the amendment by changing the
source, and doing so does not break any protocol rules.
The "feature" command now does not modify the vote for obsolete
amendments.
Before this change, there were two options for an amendment's
`DefaultVote` behavior: yes and no.
After this change, there are three options for an amendment's
`VoteBehavior`: DefaultYes, DefaultNo, and Obsolete.
To be clear, if an obsolete amendment were to (somehow) be activated by
consensus, the server still has the code to process transactions
according to that amendment, and would not be amendment blocked. It
would function the same as if it had been voting "no" on the amendment.
Resolves#4014.
Incorporates review feedback from @scottschurr.
Fix a case where `ripple::Expected` returned a json array, not a value.
The problem was that `Expected` invoked the wrong constructor for the
expected type, which resulted in a constructor that took multiple
arguments being interpreted as an array.
A proposed fix was provided by @godexsoft, which involved a minor
adjustment to three constructors that replaces the use of curly braces
with parentheses. This makes `Expected` usable for
[Clio](https://github.com/XRPLF/clio).
A unit test is also included to ensure that the issue doesn't occur
again in the future.
Make it easy for projects to depend on libxrpl by adding an `ALIAS`
target named `xrpl::libxrpl` for projects to link.
The name was chosen because:
* The current library target is named `xrpl_core`. There is no other
"non-core" library target against which we need to distinguish the
"core" library. We only export one library target, and it should just
be named after the project to keep things simple and predictable.
* Underscores in target or library names are generally discouraged.
* Every target exported in CMake should be prefixed with the project
name.
By adding an `ALIAS` target, existing consumers who use the `xrpl_core`
target will not be affected.
* In the future, there can be a migration plan to make `xrpl_core` the
`ALIAS` target (and `libxrpl` the "real" target, which will affect the
filename of the compiled binary), and eventually remove it entirely.
Also:
* Fix the Conan recipe so that consumers using Conan import a target
named `xrpl::libxrpl`. This way, every consumer can use the same
instructions.
* Document the two easiest methods to depend on libxrpl. Both have been
tested.
* See #4443.
* Remove obsolete build instructions.
* By using Conan, builders can choose which dependencies specifically to
build and link as shared objects.
* Refactor the build instructions based on the plan in #4433.
Without the protocol amendment introduced by this commit, an NFT ID can
be reminted in this manner:
1. Alice creates an account and mints an NFT.
2. Alice burns the NFT with an `NFTokenBurn` transaction.
3. Alice deletes her account with an `AccountDelete` transaction.
4. Alice re-creates her account.
5. Alice mints an NFT with an `NFTokenMint` transaction with params:
`NFTokenTaxon` = 0, `Flags` = 9).
This will mint a NFT with the same `NFTokenID` as the one minted in step
1. The params that construct the NFT ID will cause a collision in
`NFTokenID` if their values are equal before and after the remint.
With the `fixNFTokenRemint` amendment, there is a new sequence number
construct which avoids this scenario:
- A new `AccountRoot` field, `FirstNFTSequence`, stays constant over
time.
- This field is set to the current account sequence when the account
issues their first NFT.
- Otherwise, it is not set.
- The sequence of a newly-minted NFT is computed by: `FirstNFTSequence +
MintedNFTokens`.
- `MintedNFTokens` is then incremented by 1 for each mint.
Furthermore, there is a new account deletion restriction:
- An account can only be deleted if `FirstNFTSequence + MintedNFTokens +
256` is less than the current ledger sequence.
- 256 was chosen because it already exists in the current account
deletion constraint.
Without this restriction, an NFT may still be remintable. Example
scenario:
1. Alice's account sequence is at 1.
2. Bob is Alice's authorized minter.
3. Bob mints 500 NFTs for Alice. The NFTs will have sequences 1-501, as
NFT sequence is computed by `FirstNFTokenSequence + MintedNFTokens`).
4. Alice deletes her account at ledger 257 (as required by the existing
`AccountDelete` amendment).
5. Alice re-creates her account at ledger 258.
6. Alice mints an NFT. `FirstNFTokenSequence` initializes to her account
sequence (258), and `MintedNFTokens` initializes as 0. This
newly-minted NFT would have a sequence number of 258, which is a
duplicate of what she issued through authorized minting before she
deleted her account.
---------
Signed-off-by: Shawn Xie <shawnxie920@gmail.com>
There were situations where `marker`s returned by `account_lines` did
not work on subsequent requests, returning "Invalid Parameters".
This was caused by the optimization implemented in "Enforce account RPC
limits by account objects traversed":
e28989638d
Previously, the ledger traversal would find up to `limit` account lines,
and if there were more, the marker would be derived from the key of the
next account line. After the change, ledger traversal would _consider_
up to `limit` account objects of any kind found in the account's
directory structure. If there were more, the marker would be derived
from the key of the next object, regardless of type.
With this optimization, it is expected that `account_lines` may return
fewer than `limit` account lines - even 0 - along with a marker
indicating that there are may be more available.
The problem is that this optimization did not update the
`RPC::isOwnedByAccount` helper function to handle those other object
types. Additionally, XLS-20 added `ltNFTOKEN_OFFER` ledger objects to
objects that have been added to the account's directory structure, but
did not update `RPC::isOwnedByAccount` to be able to handle those
objects. The `marker` provided in the example for #4354 includes the key
for an `ltNFTOKEN_OFFER`. When that `marker` is used on subsequent
calls, it is not recognized as valid, and so the request fails.
* Add unit test that walks all the object types and verifies that all of
their indexes can work as a marker.
* Fix#4340
* Fix#4354
When writing objects to the NodeStore, we need to convert them from
the in-memory format to the binary format used by the node store.
The conversion is handled by the `EncodedBlob` class, which is only
instantiated on the stack. Coupled with the fact that most objects
are under 1024 bytes in size, this presents an opportunity to elide
a memory allocation in a critical path.
This commit also simplifies the interface of `EncodedBlob` and
eliminates a subtle corner case that could result in dangling
pointers.
These changes are not expected to cause a significant reduction in
memory usage. The change avoids the use of a `std::shared_ptr` when
unnecessary and tries to use stack-based memory allocation instead
of the heap whenever possible.
This is a net gain both in terms of memory usage (lower
fragmentation) and performance (less work to do at runtime).
In rare circumstances, both `onRequestTimeout` and the response handler
(`onSiteFetch` or `onTextFetch`) can get queued and processed. In all
observed cases, the response handler processes a network error.
`onRequestTimeout` usually runs first, but on rare occasions, the
response handler runs first, which leaves `activeResource` empty.
* Prevent internal error by catching overflow exception in `gateway_balances`.
* Treat `gateway_balances` obligations overflow as max (largest valid) `STAmount`.
* Note that very large sums of STAmount are approximations regardless.
---------
Co-authored-by: Scott Schurr <scott@ripple.com>
- MSVC 19.x reported a warning about import paths in boost for
function_output_iterator class (boost::function_output_iterator).
- Eliminate that warning by updating the import paths, as suggested by
the compiler warnings.
Port numbers can now be specified using either a colon or a space.
Examples:
1.2.3.4:51235
1.2.3.4 51235
- In the configuration file, an annoying "gotcha" for node operators is
accidentally specifying IP:PORT combinations using a colon. The code
previously expected a space, not a colon. It also does not provide
good feedback when this operator error is made.
- This change simply allows this mistake (using a colon) to be fixed
automatically, preserving the intention of the operator.
- Add unit tests, which test the functionality when specifying IP:PORT
in the configuration file.
- The RPCCall test regime is not specific enough to test this
functionality, it has been tested by hand.
- Ensure IPv6 addresses are not confused for ip:port
---------
Co-authored-by: Elliot Lee <github.public@intelliot.com>
- Implement the `operator==` and the `operator<=>` (aka the spaceship
operator) in `base_uint`, `Issue`, and `Book`.
- C++20-compliant compilers automatically provide the remaining
comparison operators (e.g. `operator<`, `operator<=`, ...).
- Remove the function compare() because it is no longer needed.
- Maintain the same semantics as the existing code.
- Add some unit tests to gain further confidence.
- Fix#2525.
In Reporting Mode, a server would core dump when it is not able to read
from Cassandra. This patch prevents the core dump when Cassandra is down
for reporting mode servers. This does not fix the root cause, but it
cuts down on some of the resulting noise.
* Follow-up to #4336
* NFToken is the naming convention in the codebase (rather than NFT)
* Rename `lsfDisallowIncomingNFTOffer` to `lsfDisallowIncomingNFTokenOffer`
* Rename `asfDisallowIncomingNFTOffer` to `asfDisallowIncomingNFTokenOffer`
Partially revert the functionality introduced
with #4195 / 5a15229 (part of 1.10.0-b1).
Acknowledgements:
Aaron Hook for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.
To report a bug, please send a detailed report to:
bugs@xrpl.org
---------
Co-authored-by: Nik Bougalis <nikb@bougalis.net>
- Copies the recipe for Snappy from Conan Center, but removes three
lines that explicitly link the standard library, which prevents
builders from statically linking it.
- Removes the recipe for RocksDB now that an official recipe for version
6.27.3 is in Conan Center.
Developers will likely need to remove cached versions of both RocksDB
and Snappy:
```
conan remove -f rocksdb
conan remove -f snappy
```
---------
Co-authored-by: John Freeman <jfreeman08@gmail.com>
* Set "fail-fast: false" so that multiple jobs in one workflow can
finish independently. By default, if one job fails, other running jobs
will be aborted, even if the other jobs are working fine and are
almost done. This leads to wasted time and resources if the failure
is, for example, OS specific, or due to a flaky unit test, and the
failed job needs to be re-run, because all the jobs end up re-running.
* Put conditions back into the windows.yml job (manual, and for
a specific branch name and that job). This prevents Github Actions
from sending "No jobs were run" failure emails on every commit.
Without this amendment, for NFTs using broker mode, if the sell offer contains a destination and that destination is the buyer account, anyone can broker the transaction. Also, if a buy offer contains a destination and that destination is the seller account, anyone can broker the transaction. This is not ideal and is misleading.
Instead, with this amendment: If you set a destination, that destination needs to be the account settling the transaction. So, the broker must be the destination if they want to settle. If the buyer is the destination, then the buyer must accept the sell offer, as you cannot broker your own offers.
If users want their offers open to the public, then they should not set a destination. On the other hand, if users want to limit who can settle the offers, then they would set a destination.
Unit tests:
1. The broker cannot broker a destination offer to the buyer and the buyer must accept the sell offer. (0 transfer)
2. If the broker is the destination, the broker will take the difference. (broker mode)
Fixes#4374
It was possible for a broker to combine a sell and a buy offer from an account that already owns an NFT. Such brokering extracts money from the NFT owner and provides no benefit in return.
With this amendment, the code detects when a broker is returning an NFToken to its initial owner and prohibits the transaction. This forbids a broker from selling an NFToken to the account that already owns the token. This fixes a bug in the original implementation of XLS-20.
Thanks to @nixer89 for suggesting this fix.
Fixes 3 issues:
In the following scenario, an account cannot perform NFTokenAcceptOffer even though it should be allowed to:
- BROKER has < S
- ALICE offers to sell token for S
- BOB offers to buy token for > S
- BROKER tries to bridge the two offers
This currently results in `tecINSUFFICIENT_FUNDS`, but should not because BROKER is not spending any funds in this transaction, beyond the transaction fee.
When trading an NFT using IOUs, and when the issuer of the IOU has any non-zero value set for TransferFee on their account via AccountSet (not a TransferFee on the NFT), and when the sale amount is equal to the total balance of that IOU that the buyer has, the resulting balance for the issuer of the IOU will become positive. This means that the buyer of the NFT was supposed to have caused a certain amount of IOU to be burned. That amount was unable to be burned because the buyer couldn't cover it. This results in the buyer owing this amount back to the issuer. In a real world scenario, this is appropriate and can be settled off-chain.
Currency issuers could not make offers for NFTs using their own currency, receiving `tecINSUFFICIENT_FUNDS` if they tried to do so.
With this fix, they are now able to buy/sell NFTs using their own currency.
Three static member functions are introduced with
definitions consistent with std::numeric_limits:
static constexpr Number min() noexcept;
Returns: The minimum positive value. This is the value closest to zero.
static constexpr Number max() noexcept;
Returns: The maximum possible value.
static constexpr Number lowest() noexcept;
Returns: The negative value which is less than all other values.
You can set a thread-local flag to direct Number how to round
non-exact results with the syntax:
Number::rounding_mode prev_mode = Number::setround(Number::towards_zero);
This flag will stay in effect for this thread only until another call
to setround. The previously set rounding mode is returned.
You can also retrieve the current rounding mode with:
Number::rounding_mode current_mode = Number::getround();
The available rounding modes are:
* to_nearest : Rounds to nearest representable value. On tie, rounds
to even.
* towards_zero : Rounds towards zero.
* downward : Rounds towards negative infinity.
* upward : Rounds towards positive infinity.
The default rounding mode is to_nearest.
* Conversions to Number are implicit
* Conversions away from Number are explicit and potentially lossy
* If lossy, round to nearest, and to even on tie
* Introduces amendment `XRPFees`
* Convert fee voting and protocol messages to use XRPAmounts
* Includes Validations, Change transactions, the "Fees" ledger object,
and subscription messages
* Improve handling of 0 drop reference fee with TxQ. For use with networks that do not want to require fees
* Note that fee escalation logic is still in place, which may cause the
open ledger fee to rise if the network is busy. 0 drop transactions
will still queue, and fee escalation can be effectively disabled by
modifying the configuration on all nodes
* Change default network reserves to match Mainnet
* Name the new SFields *Drops (not *XRP)
* Reserve SField IDs for Hooks
* Clarify comments explaining the ttFEE transaction field validation
Fixes#4005
Makes it possible for internal RPC Error Codes to associate
themselves with a non-OK (200) HTTP status code. There are
quite a number of RPC responses in addition to tooBusy that
now have non-OK HTTP status codes.
The new return HTTP return codes are only enabled by including
"ripplerpc": "3.0" or higher in the original request.
Otherwise the historical value, 200, continues to be returned.
This ensures that this is not a breaking change.
featureDisallowIncoming is a new amendment that would allow users to opt-out of incoming Checks, Payment Channels, NFTokenOffers, and trust lines. This commit includes tests.
Adds four new AccountSet Flags:
1. asfDisallowIncomingNFTOffer
2. asfDisallowIncomingCheck
3. asfDisallowIncomingPayChan
4. asfDisallowIncomingTrustline
Introduces a conanfile.py (and a Conan recipe for RocksDB) to enable building the package with Conan, choosing more recent default versions of dependencies. It removes almost all of the CMake build files related to dependencies, and the configurations for Travis CI and GitLab CI. A new set of cross-platform build instructions are written in BUILD.md.
Includes example GitHub Actions workflow for each of Linux, macOS, Windows.
* Test on macos-12
We use the <concepts> library which was not added to Apple Clang until
version 13.1.6. The default Clang on macos-11 (the sometimes current
version of macos-latest) is 13.0.0, and the default Clang on macos-12 is
14.0.0.
Closes#4223.
Clang warned about the code removed in this patch with the warning:
```
warning: out-of-line definition of constexpr static data member is
redundant in C++17 and is deprecated [-Wdeprecated]
```
There's a bug in gdb where unsigned template parameters cause issues with
RTTI. This patch changes a template parameter from `size_t` to `int` to
work around this gdb bug.
Reduce the reserve requirements from 20/5 to 10/2 in line with the current network votes. The requirements of 10/2 have been on the network long enough that new nodes should not still have the old reserve amount.
Co-authored-by: Richard Holland <richard.holland@starstone.co.nz>
* Per actions/runner-images#6002, ubuntu-18.04 is being deprecated. If
latest ever fails in the future, we'll need to fix the jobs anyway, so
catch it early.
* Use long option names
* Force clang-format to ubuntu-20.04 because LLVM 10 is not available for 22.04
* Improve move semantics in Expected:
This patch unconditionally moves an `Unexpected<U>` value parameter as
long as `U` is not a reference. If `U` is a reference the code should
not compile. An error type that holds a reference is a strange use-case,
and an overload is not provided. If it is required in the future it can
be added.
The `Expected(U r)` overload should take a forwarding ref.
* Replace enable_if with concepts in Expected
* Removed a reference to the default number of workers varying based on whether a node has validation enabled. Workers default to the number of processor cores + 2: https://github.com/ripple/rippled/blob/develop/src/ripple/core/impl/JobQueue.cpp#L166
* Protobuf v2 and Ubuntu 16.04 are no longer supported.
* Updated protobuf version as v3 is now supported, fixed typos, automatically sent number of processors when building boost & rippled.
It turns out that the feature enabled by the tfTrustLine flag
on an NFTokenMint transaction could be used as a means to
attack the NFToken issuer. Details are in
https://github.com/XRPLF/rippled/issues/4300
The fixRemoveNFTokenAutoTrustLine amendment removes the
ability to set the tfTrustLine flag on an NFTokenMint
transaction.
Closes 4300.
When starting, the code generates a new ephemeral private key and
a corresponding certificate to go along with it. This process can
take time and, while this is unlikely to matter for normal server
operations, it can have a significant impact for unit testing and
development. Profiling data suggests that ~20% of the time needed
for a unit test run can be attributed to this.
This commit does several things:
1. It restructures the code so that a new self-signed certificate
and its corresponding private key are only initialized once at
startup; this has minimal impact on the operation of a regular
server.
2. It provides new default DH parameters. This doesn't impact the
security of the connection, but those who compile from scratch
can generate new parameters if they so choose.
3. It properly sets the version number in the certificate, fixing
issue #4007; thanks to @donovanhide for the report.
4. It uses SHA-256 instead of SHA-1 as the hash algorithm for the
certificate and adds some X.509 extensions as well as a random
128-bit serial number.
5. It rounds the certificate's "start of validity" period so that
the server's precise startup time cannot be easily deduced and
limits the validity period to two years, down from ten years.
6. It removes some CBC-based ciphers from the default cipher list
to avoid some potential security issues, such as CVE-2016-2107
and CVE-2013-0169.
Caching the base58check encoded version of an `AccountID` has
performance advantages, because because of the computationally
heavy cost associated with the conversion, which requires the
application of SHA-256 twice.
This commit makes the cache significantly more efficient in terms
of memory used: it eliminates the map, using a vector with a size
that is determined by the configured size of the node, and a hash
function to directly map any given `AccountID` to a specific slot
in the cache; the eviction policy is simple: in case of collision
the existing entry is removed and replaced with the new data.
Previously, use of the cache was optional and required additional
effort by the programmer. Now the cache is automatic and does not
require any additional work or information.
The new cache also utilizes a 64-way spinlock, to help reduce any
contention that the pressure on the cache would impose.
Each node on the network is supposed to have a unique cryptographic
identity. Typically, this identity is generated randomly at startup
and stored for later reuse in the (poorly named) file `wallet.db`.
If the file is copied, it is possible for two nodes to share the
same node identity. This is generally not desirable and existing
servers will detect and reject connections to other servers that
have the same key.
This commit achives three things:
1. It improves the detection code to pinpoint instances where two
distinct servers with the same key connect with each other. In
that case, servers will log an appropriate error and shut down
pending intervention by the server's operator.
2. It makes it possible for server administrators to securely and
easily generate new cryptographic identities for servers using
the new `--newnodeid` command line arguments. When a server is
started using this command, it will generate and save a random
secure identity.
3. It makes it possible to configure the identity using a command
line option, which makes it possible to derive it from data or
parameters associated with the container or hardware where the
instance is running by passing the `--nodeid` option, followed
by a single argument identifying the infomation from which the
node's identity is derived. For example, the following command
will result in nodes with different hostnames having different
node identities: `rippled --nodeid $HOSTNAME`
The last option is particularly useful for automated cloud-based
deployments that minimize the need for storing state and provide
unique deployment identifiers.
**Important note for server operators:**
Depending on variables outside of the the control of this code,
such as operating system version or configuration, permissions,
and more, it may be possible for other users or programs to be
able to access the command line arguments of other processes
on the system.
If you are operating in a shared environment, you should avoid
using this option, preferring instead to use the `[node_seed]`
option in the configuration file, and use permissions to limit
exposure of the node seed.
A user who gains access to the value used to derive the node's
unique identity could impersonate that node.
The commit also updates the minimum supported server protocol
version to `XRPL/2.1`, which has been supported since version
1.5.0 and eliminates support for `XPRL/2.0`.
We profiled different algorithms and data structures to understand which
strategy is best from a performance standpoint:
- Linear search on an array;
- Binary search on a sorted array;
- Using `std::map`; and
- Using `std::unordered_map`.
Both linear search and std::unordered_map outperformed the other alternatives
so no change to the existing data structure is justified. If more handers are
added, this should be revisited.
For some additional details and timings, please see:
https://github.com/XRPLF/rippled/issues/3298#issuecomment-1185946010
Trustlines must be between two different accounts but two trustlines exist
where an account extends trust to itself. They were created in the early
days, likely because of bugs that have been fixed. The new fixTrustLinesToSelf
amendment will remove those trustlines when it activates.
The existing spinlock code, used to protect SHAMapInnerNode
child lists, has a mistake that can allow the same child to
be repeatedly locked under some circumstances.
The bug was in the `SpinBitLock::lock` loop condition check
and would result in the loop terminating early.
This commit fixes this and further simplifies the lock loop
making the correctness of the code easier to verify without
sacrificing performance.
It also promotes the spinlock class from an implementation
detail to a more general purpose, easier to use lock class
with clearer semantics. Two different lock types now allow
developers to easily grab either a single spinlock from an
a group of spinlocks (packed in an unsigned integer) or to
grab all of the spinlocks at once.
While this commit makes spinlocks more widely available to
developers, they are rarely the best tool for the job. Use
them judiciously and only after careful consideration.
The XLS-20 implementation contained two bugs that would require the
introduction of amendments. This complicates the adoption of XLS-20
by requiring a staggered amendment activation, first of the two fix
amendments, followed by the `NonFungibleTokensV1` amendment.
After consideration, the consensus among node operators is that the
process should be simplified by the introduction of a new amendment
that, if enabled, would behaves as if the `NonFungibleTokensV1` and
the two fix amendments (`fixNFTokenDirV1` and `fixNFTokenNegOffer`)
were activated at once.
This commit implements this proposal; it does not introduce any new
functionality or additional features, above and beyond that offered
by the existing amendments.
The peer discovery protocol depends on peers exchanging messages
listing IP addresses for other peers.
Under normal circumstances, these messages should not be sent
frequently; the existing code would track the earliest time a
new message should be processed, but did not actually enforce
that limit.
An incorrect SQL query could cause the server to improperly
configure its voting state after a restart; typically, this
would manifest as an apparent failure to store a vote which
the administrator of the server had configured.
This commit fixes the broken SQL and ensures that amendment
votes are properly reloaded post-restart and closes#4220.
The existing code would, incorrectly, allow negative amounts in offers
for non-fungible tokens. Such offers would be handled very differently
depending on the context: a direct offer would fail with an error code
indicating an internal processing error, whereas brokered offers would
improperly succeed.
This commit introduces the `fixNFTokenNegOffer` amendment that detects
such offers during creation and returns an appropriate error code.
The commit also extends the existing code to allow for buy offers that
contain a `Destination` field, so that a specific broker can be set in
the offer.
A few unit tests have historically generated a lot of noise
to the console from log writes. This noise was not useful
and made it harder to locate actual test failures.
By changing the log level of these tests from
- severities::kError to
- severities::kDisabled
it was possible to remove that noise coming from the logs.
While there should never be a missing node when copying the SHAMap,
rippled should not terminate when there's an error rotating the
database. This patch aborts the database rotation rather than aborting rippled.
ThreadSafetyAnalysis was used to identify race conditions in this file.
This analysis was modivated by a (rare) crash while running unit tests.
Add locks to Shard flagged by ThreadSafetyAnalysis
The existing code properly parses the network_id parameter from the
the configuration file, but it does not properly set up the code to
use the value correctly. As a result the configured `network_id` is
ignored.
o Fixes an off-by-one when determining which NFTokenPage an
NFToken belongs on.
o Improves handling of packed sets of 32 NFTs with
identical low 96-bits.
o Fixes marker handling by the account_nfts RPC command.
o Tightens constraints of NFTokenPage invariant checks.
Adds unit tests to exercise the fixed cases as well as tests
for previously untested functionality.
The amendment increases the maximum sign of an account's signer
list from 8 to 32.
Like all new features, the associated amendment is configured with
a default vote of "no" and server operators will have to vote for
it explicitly if they believe it is useful.
* "A path is considered invalid if and only if it enters and exits an
address node through trust lines where No Ripple has been enabled for
that address." (https://xrpl.org/rippling.html#specifics)
* When loading trust lines for an account "Alice" which was reached
via a trust line that has the No Ripple flag set on Alice's side, do
not use or cache any of Alice's trust lines which have the No Ripple
flag set on Alice's side. For typical "end-user" accounts, this will
return no trust lines.
One of the two versions of the `rngfill` function accepts a pointer
to a buffer and a size (in bytes). The function aims to fill the
provided `buffer` with `size` random bytes. It does this in chunks
of 8 bytes, for long as possible, and then fills any left-over gap
one byte at a time.
To avoid an annoying and incorrect warning about a potential buffer
overflow in the "trailing write", commit 78bc2727f7
used a `#pragma` to instruct the compiler to not generate the incorrect
diagnostic. Unfortunately, this change _also_ eliminated the trailing
write code, which means that, under some cases, the `rngfill` function
would generate between 1 and 7 fewer random bytes than requested.
This problem would only manifest on builds that do not define `__GNUC__`
which, as of this writing, means MSVC.
Several hard-coded parameters control the behavior of the ledger
acquisition engine. The values of many of these parameters where
set by intuition and have complex and non-intuitive interactions
with each other and other parts of the code.
An earlier commit attempted to adjust several of these parameters
to improve syncing performance; initial testing was promising but
a number of operators reported experiencing syncing and stability
issues with their servers. As a result, this commit reverts parts
of commit 18235067af.
This commit further adjusts some tunables so as to increase the
aggressiveness of the ledger acquisition engine.
This commit addresses minor bugs introduced with commit
6faaa91850:
- The number of threads used by the database engine was
incorrectly clamped to the lower possible value, such
that the database was effectively operating in single
threaded mode.
- The number of requests to extract at once was so high
that it could result in increased latency. The bundle
size is now limited to 4 and can be adjusted by a new
configuration option `rq_bundle` in the `[node_db]`
stanza. This is an advanced tunable and adjusting it
should not be needed.
This commit modernizes the `AcceptedLedger` and `AcceptedLedgerTx`
classes, reduces their memory footprint and reduces unnecessary
dynamic memory allocations.
This commit optimizes the way asynchronous nodestore operations are
processed both by reducing the amount of time locks are held and by
minimizing the number of memory allocations and data copying.
Adds support to TaggedCache to support smart replacement
(Needed to avoid race conditions with negative caching.)
Create a "hotDUMMY" object that represents the absence
of an object.
Allow DatabaseNodeImp::asyncFetch to complete immediately
if object is in cache (positive or negative).
Fix a bug in asyncFetch where the object returned may not
be the correct canonical version because we stash the
object in the results array before we canonicalize it.
When fetching ledgers, the existing code would isolate the peer
that sent the most useful responses and issue follow up queries
only to that peer.
This commit increases the query aggressiveness, and changes the
mechanism used to select which peers to issue follow-up queries
to so as to more evenly spread the load along those peers which
provided useful responses.
The `SHAMapInnerNode` class had a global mutex to protect the
array of node children. Profiling suggested that around 4% of
all attempts to lock the global would block.
This commit removes that global mutex, and replaces it with a
new per-node 16-way spinlock (implemented so as not to effect
the size of an inner node objet), effectively eliminating the
lock contention.
* Txs with the same fee level will sort by TxID XORed with the parent
ledger hash.
* The TxQ is re-sorted after every ledger.
* Attempt to future-proof the TxQ tie breaking test
* Abort background path finding when closed or disconnected
* Exit pathfinding job thread if there are no requests left
* Don't bother creating the path find job if there are no requests
* Refactor to remove circular dependency between InfoSub and PathRequest
The existing trust line caching code was suboptimal in that it stored
redundant information, pinned SLEs into memory and required multiple
memory allocations per cached object.
This commit eliminates redundant data, reducing the size of cached
objects and unpinning SLEs from memory, and uses value_types to
avoid the need for `std::shared_ptr`. As a result of these changes, the
effective size of a cached object, includes the overhead of the memory
allocator and the `std::shared_ptr` should be reduced by at least 64
bytes. This is significant, as there can easily be tens of millions
of these objects.
This commit combines the `apply_mutex` and `read_mutex` into a single `mutex_`
var. This new `mutex_` var is a `shared_mutex`, and most operations only need to
lock it with a `shared_lock`. The only exception is `applyMutex`, which may need
a `unique_lock`.
One consequence of removing the `apply_mutex` is more than one `applyMutex`
function can run at the same time. To help reduce lock contention that a
`unique_lock` would cause, checks that only require reading data are run a
`shared_lock` (call these the "prewriteChecks"), then the lock is released, then
a `unique_lock` is acquired. Since a currently running `applyManifest` may write
data between the time a `shared_lock` is released and the `write_lock` is
acquired, the "prewriteChecks" need to be rerun. Duplicating this work isn't
ideal, but the "prewirteChecks" are relatively inexpensive.
A couple of other designs were considered. We could restrict more than one
`applyMutex` function from running concurrently - either with a `applyMutex` or
my setting the max number of manifest jobs on the job queue to one. The biggest
issue with this is if any other function ever adds a write lock for any reason,
`applyManifest` would not be broken - data could be written between the release
of the `shared_lock` and the acquisition of the `unique_lock`. Note: it is
tempting to solve this problem by not releasing the `shared_mutex` and simply
upgrading the lock. In the presence of concurrently running `applyManifest`
functions, this will deadlock (both function need to wait for the other to
release their read locks before they can acquire a write lock).
In order to preserve the Hooks ABI, it is important that field
values used for hooks be stable going forward.
This commit reserves the required codes so that they will not
be repurposed before Hooks can be proposed for inclusion in
the codebase.
* Remove Application & Database dependency in PerfLog. Replace it with
a callback passed into the constructor.
* Fixes the circular dependency between ripple/nodestore and ripple/basics
If fast loading is enabled but the last persisted ledger is not
entirely on disk, the server would fail to start without manual
intervention by the server operator.
This commit allows the server to detect this scenario and attempt
to automatically recover.
This is a refactor aimed at cleaning up and simplifying the existing
job queue.
As of now, all jobs are cancelled at the same time and in the same
way, so this commit removes the per-job cancellation token. If the
need for such support is demonstrated, support can be re-added.
* Revise documentation for ClosureCounter and Workers.
* Simplify code, removing unnecessary function arguments and
deduplicating expressions
* Restructure job handlers to no longer need to pass a job's
handle to the job.
A typographical error would mishandle the case where a caller explicitly
tries to remove a child that is not actually part of the node. This case
is never invoked in practice, and so the bug would will never trigger.
Commit bf013c02ad added support
for incorporating a commit ID into the compiled version string
but did so in a way that did not follow the semantic versioning
standard.
This commit corrects that flaw by moving the commit ID into the
"metadata" part of the version string and properly handles the
case where the commit hash cannot be retrieved.
The pathfinding engine built into the code has several configurable
parameters to adjust the depth of the paths indexed and explored.
These parameters can dramatically impact the performance and memory
consumption of a server; higher values can result in resource usage
increasing exponentially.
These default values were decided early and somewhat arbitrarily at
a time when the network and the size of the network state were much
smaller.
This commit adjusts the default values to reduce the depth of paths
to more reasonable levels; unless explicitly overriden, the changes
mean that pathfinding operations will return fewer, shallower paths
than previous versions of the software.
This commit corrects a technical flaw that was introduced with commit
7c12f01358: as written, a mutex that is
intended to help provide synchronization for multiple threads as they
are each walking the map, is declared so that each thread is passed a
dangling reference to a unique mutex.
This commit hoists the mutex outside the thread creation loop, so all
threads use a single mutex and eliminating the dangling reference.
The nodestore includes a built-in cache to reduce the disk I/O
load but, by default, this cache was not initialized unless it
was explicitly configured by the server operator.
This commit introduces sensible defaults based on the server's
configured node size.
It remains possible to completely disable the cache if desired
by explicitly configuring it the cache size and age parameters
to 0:
[node_db]
...
cache_size = 0
cache_age = 0
- Only duplicate records from archive to writable during online_delete.
- Log duration of nodestore reads.
- Include nodestore counters in perf_log output.
- Remove gratuitous nodestore activity counting.
- Report initial sync duration in server_info and perfLog.
- Report state_accounting in perfLog.
- Make state_accounting durations more accurate.
- Parallel ledger loader.
- Config parameter to load ledgers on start.
1) Don't acquire so many nodes per pass. It's likely
far more than we need.
2) Right-size the finishedReads_ vector on passes other
than just the first.
* Sort by fee level (which is the current behavior) then by transaction
ID (hash).
* Edge case when the account at the end of the queue submits a higher
paying transaction to walk backwards and compare against the cheapest
transaction from a different account.
* Use std::if_any to simplify the JobQueue::isOverloaded loop.
* Log load fee values (at debug) received from validations.
* Log remote and cluster fee values (at trace) when changed.
* Refactor JobQueue::isOverloaded to return sooner if overloaded.
* Refactor Transactor::checkFee to only compute fee if ledger is open.
This flag, if present, suppresses the output of incoming
trustlines in default state.
This is primarily motivated by observing that users of Xumm often
have many unwanted incoming trustlines in a default state, which are
not useful in the vast majority of cases.
Being able to suppress those when doing `account_lines` saves bandwidth
and resources.
This commit implements partitioned unordered maps and makes it possible
to traverse such a map in parallel, allowing for more efficient use of
CPU resources.
The `CachedSLEs`, `TaggedCache`, and `KeyCache` classes make use of the
new functionality, which should improve performance.
The pathfinding engine requires pre-building large tables which is a
resource-intensive operation. Typically, one would not expect that a
server configured as a validator would also support pathfinding APIs
and so, building those tables by default wastes resources.
This commit, if merged, will disable pathfinding on servers that are
configured as validators, unless the server operator opts to support
it explicitly, by configuring the `[path_search_max]` parameter.
Validator operators that wish to support pathfinding on a validator
and want to use the default values can add the following stanza to
their server's configuration file:
[path_search_max]
7
The priority of different types of jobs was set back in the early
days of development, based on insight and observations that don't
necessarily apply any longer.
Specifically, job types used by the server to sync to the network
were being treated as lower priority than client requests, making
it more difficult to regain sync.
This commit adjusts the priority of several jobs and should allow
servers to prioritize resynchronizing to the network over serving
clients.
The existing calculation would limit the maximum number of threads
that would be created by default to at most 6; this may have been
reasonable a few years ago, but given both the load on the network
as of today and the increase in the number of CPU cores, the value
should be revisited.
This commit, if merged, changes the default calculation for nodes
that are configured as `large` or `huge` to allow for up to twelve
threads.
The "sweep interval" is the amount of time between successive sweeps of
of various in-memory data structures to remove stale items.
Prior to this commit, the interval was automatically adjusted, based on
the value of the `[node_size]` option in a server's configuration file.
If merged, this commit introduces a new configuration option that makes
it possible for a server operator to adjust the sweep interval and make
a CPU/memory tradeoff:
[sweep_interval]
<integer>
The specified value represents the number of seconds between successive
sweeps. The range of valid values is between 10 and 600.
Important operator notes:
This is an advanced configuration option that should not be used unless
there is empirical data which suggests that the default sweep frequency
is either resulting in performance problems or is causing undue load to
the server.
Note that adjusting the sweep interval may not have the intended effect
on the server. Lower values will not always translate to a reduction of
memory usage and higher values will not always translate to a reduction
of CPU usage and/or load.
The performance characteristics of `std::unordered_map` are better
than `std::map` and the former should be preferred when the strict
ordering of the latter is not required.
* Only require adding the new feature names in one place. (Also need to
increment a counter, but a check on startup will catch that.)
* Allows rippled to have the code to support a given amendment, but
not vote for it by default. This allows the amendment to be enabled in
a future version without necessarily amendment blocking these older
versions.
* The default vote is carried with the amendment name in the list of
supported amendments.
* The amendment table is constructed with the amendment and default
vote.
* Also clean up some formatting in the Windows instructions
* Changed the recommended version for Windows to 1.1.1L after deeper
checking uncovered some build issues.
* Patch the soci unsigned-types.h file. If no changes are made, delete
the patched file and exit. If there are changes, backup the original
and replace it with the patched file.
* Fixes#3885
Patch Rocksdb only once:
* The repeated patches do not appear to affect build times, but avoiding
unnecessary copies is good for its own sake.
There are two mutexes in ValidatorSite: `site_mutex_` and `state_mutex_`. Some
function end up locking both mutexes. However, depending on the call, the
mutexes could be locked in different orders, resulting in deadlocks.
If both mutexes are locked, this patch always locks the `sites_mutex_` first.
The existing logic involves every server sending every transaction
that it receives to all its peers (except the one that it received
a transaction from).
This commit instead uses a randomized algorithm, where a node will
randomly select peers to relay a given transaction to, caching the
list of transaction hashes that are not relayed and forwading them
to peers once every second. Peers can then determine whether there
are transactions that they have not seen and can request them from
the node which has them.
It is expected that this feature will further reduce the bandwidth
needed to operate a server.
The existing license file contains copyright statements and license
snippets referencing code that has long since been removed from the
codebase and which are not necessary any longer.
The general copyright statement for the ISC License is also changed
from "Ripple Labs" which is one contributor to the codebase, to the
more general "XRP Ledger Developers".
Remaining code which was originally taken from Bitcoin includes the
relevant copyright statement(s) inline.
To aid in the automated detection of the license type by GitHub and
tools like `licensee`, the following statement which referenced the
ISC license, and which was listed at the bottom of the file, is now
incorporated at the top:
>The accompanying files incorporate work covered by the following copyright
>and previous license notice:
>
>Copyright (c) 2011 Arthur Britto, David Schwartz, Jed McCaleb,
>Vinnie Falco, Bob Way, Eric Lombrozo, Nikolaos D. Bougalis, Howard Hinnant
This commit removes the `ltINVALID` pseudo-type identifier from
`LedgerEntryType` and the `ttINVALID` pseudo-type identifier from
`TxType` and includes several small additional improvements that
help to simplify the code base.
It also improves the documentation `LedgerEntryType` and `TxType`,
which was all over the place, and highlights some important caveats
associated with making changes to the ledger and transaction type
identifiers.
The commit also adds a safety check to the `KnownFormats<>` class,
that will catch the the accidental reuse of format identifiers.
Ideally, this should be done at compile time but C++ does not (yet?)
allow for the sort of introspection that would enable this.
The legacy functions `cdirFirst` and `dirFirst` were mostly
identical; the differences were only type-related. The same
situation existed with `cdirNext` and `dirNext`.
This commit removes the duplicated code by introducing new
template functions that abstract away the differences that
are present between each pair of functions.
This commit also improves the naming of function arguments,
helping to elucidate their purpose & use and to make the
code self-documenting.
The Negative UNL is a feature of the XRP Ledger consensus protocol that
improves liveness (the network's ability to make forward progress) during
a partial outage. Using the Negative UNL, servers adjust their effective
UNLs based on which validators are currently online and operational, so
that a new ledger version can be declared validated even if several trusted
validators are offline.
The Negative UNL has no impact on how the network processes transactions
or what transactions' outcomes are, except that it improves the network's
ability to declare outcomes final during some types of partial outages.
The feature was originally introduced with version **1.6.0** but it was
only possible to manually enable this. If merged, this commit introduces
the amendment associated with the feature so that server operators can
vote on whether to enable this feature.
For more details, please see https://xrpl.org/negative-unl.html
This commit closes#3898.
Under some circumstances, it is possible to induce an out-of-bounds
memory read in the base58 decoder.
This commit addresses this issue.
Acknowledgements:
Guido Vranken for discovering and responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.
Ripple is generously sponsoring a bug bounty program for the
rippled project. For more information please visit:
https://ripple.com/bug-bounty
The HardenedValidations amendment introduces additional fields
in validations:
- `sfValidatedHash`, if present, is the hash the of last ledger that
the validator considers to be fully validated.
- `sfCookie`, if present, is a 64-bit cookie (the default
implementation selects it randomly at startup but other
implementations are possible), which can be used to improve the
detection and classification of duplicate validations.
- `sfServerVersion`, if present, reports the version of the software
that the validator is running. By surfacing this information,
server operators gain additional insight about variety of software
on the network.
If merged, this commit fixes#3797 by adding the fields to the
`validations` stream as shown below:
- `sfValidateHash` as `validated_hash`: a 256-bit hex string;
- `sfCookie` as `cookie`: a 64-bit integer as a string; and
- `sfServerVersion` as `server_version`: a 64-bit integer as
a string.
With this amendment, the CheckCash transaction creates a TrustLine
if needed. The change is modeled after offer crossing. And,
similar to offer crossing, cashing a check allows an account to
exceed its trust line limit.
The following changes were made:
- Removed dependency on template defined in beast detail namespace.
- Removed Section::find() method which had an obsolete interface.
- Made Section::get<>() easier to use for the common case of
retrieving a std::string. The revised get() method replaces old
calls to Section::find().
- Provided a default template parameter to free function
get<>(Section config, std::string name) so it stays similar to
Section::get<>().
Then the rest of the code was adapted to these changes.
- Calls to Section::find() were replaced with calls to Section::get.
- Unnecessary get<std::string>() arguments were reduced to get().
These changes dug up an interesting artifact in the SHAMap unit
tests. I'm not sure why the tests were working before, but there
was a problem with the case of a Section key. The unit test is
fixed.
The `[node_size]` configuration parameter is used to tune various
parameters based on the hardware that the code is running on. The
parameter can take five distinct values: `tiny`, `small`, `medium`,
`large` and `huge`.
The default value in the code is `tiny` but the default configuration
file sets the value to `medium`. This commit attempts to detect the
amount of RAM on the system and adjusts the node size default value
based on the amount of RAM and the number of hardware execution
threads on the system.
The decision matrix currently used is:
| | 1 | 2 or 3 | ≥ 4 |
|:-------:|:----:|:------:|:------:|
| > ~8GB | tiny | tiny | tiny |
| > ~12GB | tiny | small | small |
| > ~16GB | tiny | small | medium |
| > ~24GB | tiny | small | large |
| > ~32GB | tiny | small | huge |
Some systems exclude memory reserved by the the hardware, the kernel
or the underlying hypervisor so the automatic detection code may end
up determining the node_size to be one less than "appropriate" given
the above table.
The detection algorithm is simplistic and does not take into account
other relevant factors. Therefore, for production-quality servers it
is recommended that server operators examine the system holistically
and determine what the appropriate size is instead of relying on the
automatic detection code.
To aid server operators, the node size will now be reported in the
`server_info` API as `node_size` when the command is invoked in
'admin' mode.
A recent version of clang notes a number of places in range
for loops where the code base was making unnecessary copies
or using const lvalue references to extend lifetimes. This
fixes the places that clang identified.
* Create SQLite database for mapping transaction IDs to shard indexes
* Create SQLite database for mapping ledger hashes to shard indexes
* Create additional test cases for the shard database
* load_factor was missing from server_info when the server was running in
reporting mode. Now, the reporting mode server calls server_info on the p2p
node, and propagates the load_factor back to the client.
In order to effectively mitigate CVE-2021-3499 even when compiling
against versions of OpenSSL prior to 1.1.1k, this commit:
1) requires use of TLS 1.2 or later. Note that both TLS 1.0 and
TLS 1.1 have been officially deprecated for over a year.
2) disables renegotiation support for TLS 1.2 connections.
Lastly, this commit also changes the default list of ciphers that
the server offers, limiting it only to ciphers that are part of
TLS 1.2.
The `tx` command supports output in both "text" and "binary" modes,
controlled by the binary flag. For more details on the command and
the possible arguments, please see: https://xrpl.org/tx.html.
The existing handler would incorrectly deal with metadata when in
binary mode. This commit corrects this issue, ensuring that the
metadata is properly encoded, depending on the mode.
Typically, an RPC response contains a `result` field, which
contains details about the operation performed. For ease of
parsing, forwarded responses must look like a non-forwarded
response.
In some instances the response was incorrectly composed, so
that the actual `result` object would be encapsulated by an
outer `result` object, breaking existing code.
This commit, addresses this issue and correctly "folds" the
`result` field, ensuring a consistent schema for responses.
* Add instructions to the workflow at the point where the failure would
occur, which is where someone experiencing a failure is most likely to
look. To keep things simple, the instructions are always printed. The
assumption is that if the job succeeds, nobody is likely to look
anyway.
* Provides the diff of the failure as an artifact, so the user can apply
it directly to their repo.
* Also update the levelization/README.md to clarify the levels a little
bit.
This updates the build process to use the local Artifactory server as a docker image cache to avoid being rate limited by docker hub during the build process.
While most of the code associated with secp256k1 operations had
been migrated to libsecp256k1, the deterministic key derivation
code was still using calls to OpenSSL.
If merged, this commit replaces the OpenSSL-based routines with
new libsecp256k1-based implementations. No functional change is
expected and the change should be transparent.
This commit also removes several support classes and utility
functions that wrapped or adapted various OpenSSL types that
are no longer needed.
A tip of the hat to the original author of this truly superb
library, Dr. Pieter Wuille, and to all other contributors.
This commit expands the detection capabilities of the Byzantine
validation detector. Prior to this commit, only validators that
were on a server's UNL were monitored. Now, all the validations
that a server receives are passed through the detector.
The existing class offered several constructors which were mostly
unnecessary. This commit eliminates all existing constructors and
introduces a single new one, taking a `Slice`.
The internal buffer is switched from `std::vector` to `Buffer` to
save a minimum of 8 bytes (plus the buffer slack that is inherent
in `std::vector`) per SHAMapItem instance.
Add support to allow multiple indepedent nodes to produce a binary identical
shard for a given range of ledgers. The advantage is that servers can use
content-addressable storage, and can more efficiently retrieve shards by
downloading from multiple peers at once and then verifying the integrity of
a shard by cross-checking its checksum with the checksum other servers report.
Before this change any non-zero Sequence field was handled as
a non-ticketed transaction, even if a TicketSequence was
present. We learned that this could lead to user confusion.
So the rules are tightened up.
Now if any transaction contains both a non-zero Sequence
field and a TicketSequence field then that transaction
returns a temSEQ_AND_TICKET error code.
The (deprecated) "sign" and "submit" RPC commands are tuned
up so they auto-insert a Sequence field of zero if they see
a TicketSequence in the transaction.
No amendment is needed because this change is going into
the first release that supports the TicketBatch amendment.
* Fix bug where incorrect max amount was set for XRP
* Fix bug where incorrect source currencies were set when XRP was the dst and a
sendmax amount was set
The existing code that deserialized an STAmount was sub-optimal and performed
poorly. In some rare cases the operation could result in otherwise valid
serialized amounts overflowing during deserialization. This commit will help
detect error conditions more quickly and eliminate the problematic corner cases.
* Use theoretical quality to order the strands
* Do not use strands below the user specified quality limit
* Stop exploring strands (at the current quality iteration) once any strand is non-dry
The previous error description was focused on keys that are too long,
but this error can occur if the key is too short or does not contain
the correct prefix.
* Add a new operating mode to rippled called reporting mode
* Add ETL mechanism for a reporting node to extract data from a p2p node
* Add new gRPC methods to faciliate ETL
* Use Postgres in place of SQLite in reporting mode
* Add Cassandra as a nodestore option
* Update logic of RPC handlers when running in reporting mode
* Add ability to forward RPCs to a p2p node
- The changes to manifest relaying introduced with commit f74b469e68
will cause newly accepted manifests to be sent back to the peer from
which they were received. This no longer happens: a newly accepted
manifest is never sent back to the peer we received it from.
- When encountering a manifest without a domain set, the `manifest` and
`validator_info` commands would include an empty string as the domain
associated with the manifest. This no longer happens: if a domain is
not present, the `domain` field will not be.
The existing code attempts to validate the provided node public key
using a function that assumes that the encoded public key is for an
account. This causes the parsing to fail.
This commit fixes#3317 by letting the caller specify the type of
the public key being checked.
The manifest relay code would only ever relay manifests from validators
on a server's UNL which means that the manifests of validators that are
not broadly trusted can fail to propagate across the network, which can
make it difficult to detect and track such validators.
This commit, if merged, propagates all manifests on a best-effort basis
resulting in broader availability of manifests on the network and avoid
the need to introduce on-ledger manifest storage or to establish one or
more manifest repositories.
- Add validation/proposal reduce-relay feature negotiation to
the handshake
- Make squelch duration proportional to a number of peers that
can be squelched
- Refactor makeRequest()/makeResponse() to facilitate handshake
unit-testing
- Fix compression enable flag for inbound peer
- Fix compression algorithm parsing in the header parser
- Fix squelch duration in onMessage(TMSquelch)
This commit fixes 3624, fixes 3639 and fixes 3641
Support for 'out-of-sequence' transaction execution was introduced
in commit 7724cca384.
The changes in that commit were gated under a feature but there was
no corresponding amendment introduced that would allow the network
to vote on this amendment.
This commit introduces 'TicketBatch' amendment as the amendment
that is associated with the tickets feature. If the amendment is
enabled, it will activate support for tickets.
This commit also removes several workarounds that are no longer
needed in unit tests.
* Markdown explanation of what levelization is, the intended levels, as
well as the process used to determine dependencies
* Shell script finds all dependencies, groups them, and finds cyclic
dependencies and maps out non-cyclic dependencies.
* Github job to run the script and fail if anything changes. Should
catch introduction of new dependencies and new problems. Will also
detect changes if problems or dependencies are removed.
* Creates a version 2 of the UNL file format allowing publishers to
pre-publish the next UNL while the current one is still valid.
* Version 1 of the UNL file format is still valid and backward
compatible.
* Also causes rippled to lock down if it has no valid UNLs, similar to
being amendment blocked, except reversible.
* Resolves#3548
* Resolves#3470
* Move all the vcpkg windows dependency installations into one step.
* Move the unmodified `before_install` step above the matrix to improve
readability, because this step runs before any of the matrix steps.
Due to some quirky emergent behavior, the server can't really begin
synching until twice the default close time resolution of the genesis
ledger, which is 30 seconds, has passed. In effect, this causes a one
minute delay.
This commit adjusts the default close time resolution down to the
minimum allowed resoluion of 10 seconds, so the corresponding delay
is reduced by 67% down to 20 seconds. This should be enough time to
ensure the server has reasonable connectivity without unduly delaying
initial synch times.
This comment explains this patch and the associated patches
that should be folded into it. This paragraph should be removed
when the patches are folded after review.
This change significantly improves ledger sync and fetch
times while reducing memory consumption. The change affects
the code from that begins with SHAMap::getMissingNodes and runs
through to Database::threadEntry.
The existing code issues a number of async fetches which are then
handed off to the Database's pool of read threads to execute.
The results of each read are placed in the Database's positive
and negative caches. The caller waits for all reads to complete
and then retrieves the results out of these caches.
Among other issues, this means that the results of the first read
cannot be processed until the last read completes. Additionally,
all the results must sit in memory.
This patch changes the behavior so that each read operation has a
completion handler associated with it. The completion of the read
calls the handler, allowing the results of each read to be
processed as it completes. As this was the only reason the
negative and positive caches were needed, they can now be removed.
The read generation code is also no longer needed and is removed.
The batch fetch logic was never implemented or supported and is
removed.
gcc's implementation of `prm::synchronized_pool_resource` showed
extremely poor performance compared with
`boost::synchronized_pool_resouece`. Boost's implementation of pmr is
now used in all cases (previously it was only used when a standard
lib, like clang's, lacked an implementation of pmr).
This patch also makes a minor change where inner nodes are constructed
with sparse arrays, unless "dense" is explicitly requested.
Prior to this commit, the amendments that a server would vote in support
of or against could be configured both via the configuration file and
via the command line "feature" command. Changes made in the configuration
file would only be loaded once at server startup and changes made via the
command line take effect immediately but are not persisted across
restarts.
This commit deprecates management of amendments via the configuration
file and stores the relevant information in the `wallet.db` database
file.
1. On startup, the new code parses the configuration file.
2. If the `[veto_amendments]` or `[amendments]` sections are present,
we check if the `FeatureVotes` table is present in `wallet.db`.
3. If it is not, we create the `FeatureVotes` table and transfer the
settings from the config file.
4. Proceed normally but only reference the `FeatureVotes` table instead
of the config file.
5. Warns if the voting table already exists in `wallet.db` and there
exists voting sections in the config file. The config file is ignored
in this case.
This change addresses & closes#3366
* Found several functions called under lock that take a lock. Refactor
to require a lock as a parameter instead.
* Found several functions called under lock that don't take a lock, but
should. Refactored those as well to require a lock as a parameter.
Unit tests are counting test failures, process crashes, and process exit code
failures in the count. Since a failing tests causes the process exit code to
return failure, we get extra counts. This patch removes process exit code
failures from the count.
ReadViewFwdRange was storing a cached `end_` iterator that was lazily
created in an iterators `end()` function. When the cache is empty, and
the range it iterated from multiple threads, this creates a race
condition.
This change has performance consequences for "old style" for loops.
For example:
```
// don't do this
for(auto i = tx_range.begin(); i != tx_range.end(); ++i)
```
Can call the now expensive `end()` function more often than needed.
Range-based for loop (I.e. `for(auto const& t : tx_range)`) should be
used instead.
- Under some conditions, comparing `ReadViewFwdRange::iterators`
for equality could derefence an empty `std::unique_ptr` which
will result in a crash.
- Misuse of the `equal` API could result in a `std::bad_cast`
exception being thrown from when iterating transactions or
SLEs from the `OpenView`, `RawStateTable` and `Ledger` classes.
A large percentage of inner nodes only store a small number of children. Memory
can be saved by storing the inner node's children in sparse arrays. Measurements
show that on average a typical SHAMap's inner nodes can be stored using only 25%
of the original space.
This commit combines a number of cleanups, targeting both the
code structure and the code logic. Large changes include:
- Using more strongly-typed classes for SHAMap nodes, instead of relying
on runtime-time detection of class types. This change saves 16 bytes
of memory per node.
- Improving the interface of SHAMap::addGiveItem and SHAMap::addItem to
avoid the need for passing two bool arguments.
- Documenting the "copy-on-write" semantics that SHAMap uses to
efficiently track changes in individual nodes.
- Removing unused code and simplifying several APIs.
- Improving function naming.
- Simplify and consolidate code for parsing hex input.
- Replace beast::endian::order with boost::endian::order.
- Simplify CountedObject code.
- Remove pre-C++17 workarounds in favor of C++17 based solutions.
- Improve `base_uint` and simplify its hex-parsing interface by
consolidating the `SexHex` and `SetHexExact` methods into one
API: `parseHex` which forces callers to verify the result of
the operation; as a result some public-facing API endpoints
may now return errors when passed values that were previously
accepted.
- Remove the simple fallback implementations of SHA2 and RIPEMD
introduced to reduce our dependency on OpenSSL. The code is
slow and rarely, if ever, exercised and we rely on OpenSSL
functionality for Boost.ASIO as well.
- Provide separate functions for serializing depending on whether
one wants a "wire" version of a node, or one suitable for hashing.
- Remove unused functions
The existing SHAMapNodeID object has both a valid and an invalid state
and requirs callers to verify the state of an instance prior to using
it. A simple set of changes removes that restriction and ensures that
all instances are valid, making the code more robust.
This change also:
1. Introduces a new function to construct a SHAMapNodeID from a
serialized blob; and
2. Reduces the amount of constructors the class exposes.
- Limit the lifetime of a buffer that was only used in the early
phases of peer connection establishment but which lived on as
long as the peer was active.
- Cache the message used to transfer manifests, so it can be reused
instead of recreated for every peer connection.
- Improve the reading of partial messages by passing a hint to the
I/O layer if the number of bytes needed to complete the message
is known.
The existing code issues a PING to each peer every 8 seconds. While
frequent PINGs allow us to estimate a peer's latency with a high
degree of accuracy, this "inter-server polka dance" is inefficient
and not useful. This commit, if merged, reduces the PING frequency
to once every 60 seconds.
Additionally, this commit simplifies the PING handling logic and
merges the code used to check and disconnect peers which fail to
track the network directly into the timer callback.
When evaluating the fitness and usefulness of an outbound peer, the code
would incorrectly calculate the amount of time that the peer spent in
a non-useful state.
This commit, if merged, corrects the calculation and makes the timeout
values configurable by server operators.
Two new options are introduced in the 'overlay' stanza of the config
file. The default values, in seconds, are:
[overlay]
max_unknown_time = 600
max_diverged_time = 300
This commit replaces the `peers_max` configuration element which had
a predetermined split between incoming and outgoing connections with
two new configuration options, `peers_in_max` and `peers_out_max`,
which server operators can use to explicitly control the number of
incoming and outgoing peer slots.
There have been cases in the past where SFields have been defined
in such a way that they did not follow our conventions. In
particular, the string representation of an SField should match
the in-code name of the SField.
This change leverages the preprocessor to encourage SFields to
be properly constructed.
The suffixes of SField types are changed to be the same as
the suffixes of corresponding SerializedTypeIDs. This allows
The preprocessor to match types using simple name pasting.
Since the string representation of the SField is part of our
stable API, the name of sfPayChannel was changed to sfChannel.
This change allows sfChannel to follow our conventions while
making no changes to our external API.
* Remove DinD Service from container build template
DinD has changed how it works on GItLab due to recent docker changes such that the service no long needs to be called so long as the runner is being run on a `docker-X` tagged machine.
* refactor for docker service on normal node
* If multiple transactions are queued for the account, change the
account's sequence number in a temporary view before processing the
transaction.
* Adds a new "at()" interface to STObject which is identical to the
operator[], but easier to write and read when dealing with ptrs.
* Split the TxQ tests into two suites to speed up parallel run times.
This commit introduces a new configuration option that server
operators can set. The value is communicated to other servers
and is also reported via the `server_info` API.
The value is meant to allow third-party applications or tools
to group servers together. For example, a tool that visualizes
the network's topology can group servers together.
Similar to the "Domain" field in validator manifests, an operator
can claim any domain. Prior to relying on the value returned, the
domain should be verified by retrieving the xrp-ledger.toml file
from the domain and looking for the server's public key in the
`nodes` array.
* Increases hard-coded number of parallel unit test processes for
Windows and MacOS builds from 1 to 2.
* Reduces Travis job time to well under the timeout value of 1.5 hours.
* Continue using a hard-coded value rather than `nprocs` because higher
values cause some jobs to run out of memory.
* Jobs with no unit tests are counted as failures. Resolves#3474
* Crashed processes are counted as failures. Resolves#3600
* Any tests specified on the command line test do not have matching
suites are counted as failures.
* Remove unused CI manual test.
When processing the `tx` command, we will now load both the transaction
and its metadata directly from SQLite.
Previously the `tx` RPC call was querying SQLite for the transaction
and then separately querying the key-value store for the metadata.
Support for IPv6 messages was added with commit 08382d866b
and version 1.1.0. No peer presently connected to the network in a useful capacity fails
to understand v2 messages.
This commit removes the code that generates and processes v1 messages and deletes legacy
messages from the protocol buffer definition file.
Use C++17 constant expressions to calculate the inverse
alphabet map at compile time instead of at runtime.
Remove support for encoding & decoding tokens using the
Bitcoin alphabet.
The job queue can impose limits of how many jobs of a particular
type can be queued.
This commit makes the previously hard-coded limit associated with
transactions configurable by the server's operator. Servers that
have increased memory capacity or which expect to see an influx
of transactions can increase the number of transactions their
server will be able to queue.
This commit fixes#3556.
The "/vl" HTTP endpoint can be used to request a particular
UNL from a rippled instance.
This commit, if merged, includes the public key of the requested
list in the response.
This commit fixes#3392
* Distinguish between recent and historical shards
* Allow multiple storage paths for historical shards
* Add documentation for this feature
* Add unit tests
Some RPC commands return `ledger_index` as a quoted numeric
string. This change allows the returned value to be directly
copied and used for follow-on RPC commands.
This commit fixes#3533
When attempting to load a validator list from a configured
site, attempt to reuse the last IP that was successfully
used if that IP is still present in the DNS response.
Otherwise, randomly select an IP address from the list of
IPs provided by the DNS system.
This commit fixes#3494.
With few exceptions, servers will typically receive multiple copies
of any given message from its directly connected peers. For servers
with several peers this can impact the processing latency and force
it to do redundant work. Proposal and validation messages are often
relayed with extremely high redundancy.
This commit, if merged, introduces experimental code that attempts
to optimize the relaying of proposals and validations by allowing
servers to instruct their peers to "squelch" delivery of selected
proposals and validations. Servers making squelching decisions by
a process that evaluates the fitness and performance of a given
server and randomly selecting a subset of the best candidates.
The experimental code is presently disabled and must be explicitly
enabled by server operators that wish to test it.
Tickets are a mechanism to allow for the "out-of-order" execution of
transactions on the XRP Ledger.
This commit, if merged, reworks the existing support for tickets and
introduces support for 'ticket batching', completing the feature set
needed for tickets.
The code is gated under the newly-introduced `TicketBatch` amendment
and the `Tickets` amendment, which is not presently active on the
network, is being removed.
The specification for this change can be found at:
https://github.com/xrp-community/standards-drafts/issues/16
Commit 4dc08f8202 introduced support for
deterministic shards, which makes the sharding functionality provided
by rippled more useful.
After merging, several opportunities for further improvements to the
deterministic sharding implementation were identified and a significant
increase int memory usage during shard finalization was detected.
Because of these issues, the commit is being reverted and the feature is
being rolled back. It will be reintroduced in a future release.
* Builds Windows dependencies first.
* Builds ALL OSs in the last stage.
* Fix the MacOS builds.
* Windows dependency stages are allowed to fail so ALL configurations will
attempt to build. Windows builds will probably fail if dependencies fail
(caching may allow them to succeed), but they will at least be attempted.
* Remove broken AppVeyor config file, so it stops trying.
The checkpointer class had assumed that the database would exist for the
lifetime of the application. This is no long true. These changes resolve bugs
involving dangling pointers.
There was a race condition in `on_accept` where the object's destructor
could run while `on_accept` was called.
This patch ensures that if `on_accept` is called then the object remains
valid for the duration of the call.
* Fixes#3486
* load factor computation normalized by load_base.
* last validated ledger age set to -1 while syncing.
* Return status changed:
* healthy -> ok
* warning -> service_unavailable
* critical -> internal_server_error
This change can help improve the liveness of the network during periods of network
instability, by allowing the network to track which validators are presently not online
and to disregard them for the purposes of quorum calculations.
If the 'HardenedValidations' amendment is enabled, this commit will
track the version of the software that validators embed in their
validations.
If a server notices that at least 60% of the validators on its UNL
are running a newer version than it is running, it will periodically
print an informational message, reminding the operator to check for
update.
The tecUNFUNDED code is actively used when attempting to create payment
channels; the messages incorrectly list it as deprecated.
Meanwhile, the tecUNFUNDED_ADD code actually is an unused legacy code,
dating back to when there was a WalletAdd transactor. The terLAST and
terFUNDS_SPENT codes are also unused legacy codes.
Engine result messages are not part of the binary format and are
documented as subject to change without notice, so this should not
require an amendment nor a new API version.
Align error code table for human readability.
The amendment was partially complete, included no functional code
and, even if activated, it would result in no changes to transaction
proessing. Despite this, removing the amendment is the prudent course
of action and avoids the possibility of an accidental activation.
If additional cryptoconditions are implemented, they will be each
assigned a new, unique amendment code.
This commit, if merged, adds support to allow multiple indepedent nodes to
produce a binary identical shard for a given range of ledgers. The advantage
is that servers can use content-addressable storage, and can more efficiently
retrieve shards by downloading from multiple peers at once and then verifying
the integrity of a shard by cross-checking its checksum with the checksum
other servers report.
* Document delete_batch, back_off_milliseconds, age_threshold_seconds.
* Convert those time values to chrono types.
* Fix bug that ignored age_threshold_seconds.
* Add a "recovery buffer" to the config that gives the node a chance to
recover before aborting online delete.
* Add begin/end log messages around the SQL queries.
* Add a new configuration section: [sqlite] to allow tuning the sqlite
database operations. Ignored on full/large history servers.
* Update documentation of [node_db] and [sqlite] in the
rippled-example.cfg file.
Resolves#3321
* The amendment ballot counting code contained a minor technical
flaw, caused by the use of integer arithmetic and rounding
semantics, that could allow amendments to reach majority with
slightly less than 80% support. This commit introduces an
amendment which, if enabled, will ensure that activation
requires at least 80% support.
* This commit also introduces a configuration option to adjust
the amendment activation hysteresis. This option is useful on
test networks, but should not be used on the main network as
is a network-wide consensus parameter that should not be
changed on a per-server basis; doing so can result in a
hard-fork.
Fixes#3396
Work on a version 2 of the XRP Network API has begun. The new
API returns:
* `notSynced` in place of `noClosed`, `noCurrent`, and `noNetwork`;
* `invalidParams` in place of `lgrIdxInvalid`.
The new version 2 API cannot be selected yet, as it remains a work
in progress.
Fixes#3269
If a port number is not specified in the [ips] or [ips_fixed]
blocks, automatically add the new default peer port which was
registered with IANA: 2459. Also use 2459 if no port is specified
with manually using the `connect` command; previously it was
using 6561, which could have resulted in spurious failures.
This commit, if merged, fixes#2861.
* Gives a summary of the health of the node:
Healthy, Warning, or Critical
* Last validated ledger age:
<7s is Healthy,
7s to 20s is Warning
> 20s is Critcal
* If amendment blocked, Critical
* Number of peers:
> 7 is Healthy
1 to 7 is Warning
0 is Critical
* server state:
One of full, validating or proposing is Healthy
One of syncing, tracking or connected is Warning
All other states are Critical
* load factor:
<= 100 is Healthy
101 to 999 is Warning
>= 1000 is Critical
* If not Healthy, info field contains data that is considered not
Healthy.
Fixes: #2809
Commit e257a22 introduced changes in the logic used to acquire historical
ledgers. The logic could cause historical ledgers to be acquired only since
the last online deletion interval instead of the configured value to allow
deletion.
* Make sure variables are always initialized
* Use lround instead of adding .5 and casting
* Remove some unneeded vars
* Check for null before calling strcmp
* Remove redundant if conditions
* Remove make_TxQ factory function
* Improve documentation
* Make the ShardArchiveHandler rather than the DatabaseShardImp perform
LastLedgerHash verification for downloaded shards
* Remove ShardArchiveHandler's singleton implementation and make it an
Application member
* Have the Application invoke ShardArchiveHandler initialization
instead of clients
* Add RecoveryHandler as a ShardArchiveHandler derived class
* Improve commenting
* Add documentation for shard validation
* Retrieve last ledger hash for imported shards
* Verify the last ledger hash in Shard::finalize
* Limit last ledger hash retrieval attempts for imported shards
* Use a common function for removing failed shards
* Add new ShardInfo::State for imported shards
Identifiers for retired amendments should not generally be used
in the codebase.
This commit reduces their visibility down to one translation
unit and marks them as unused and deprecated to prevent
accidental reuse.
In deciding whether to relay a proposal or validation, a server would
consider whether it was issued by a validator on that server's UNL.
While both trusted proposals and validations were always relayed,
the code prioritized relaying of untrusted proposals over untrusted
validations. While not technically incorrect, validations are
generally more "valuable" because they are required during the
consensus process, whereas proposals are not, strictly, required.
The commit introduces two new configuration options, allowing server
operators to fine-tune the relaying behavior:
The `[relay_proposals]` option controls the relaying behavior for
proposals received by this server. It has two settings: "trusted"
and "all" and the default is "trusted".
The `[relay_validations]` options controls the relaying behavior for
validations received by this server. It has two settings: "trusted"
and "all" and the default is "all".
This change does not require an amendment as it does not affect
transaction processing.
The sfLedgerSequence field is designated as optional in the object
template but it is effectively required and validations which do not
include it were, correctly, rejected.
This commit migrates the check outside of the peer code and into the
constructor used for validations being deserialized for the network.
Furthermore, the code will generate an error if a validation that is
generated by a server does not include the field.
The existing code used std::deque along with a size check to constrain the
size of a buffer and, effectively, "hand rolled" a circular buffer. This
change simply migrates directly to boost::circular_buffer.
The unit test now verifies that if an account is not present in the
starting account_tx ledger, account_tx still iterates down and finds
the transaction that deletes the account (and earlier transactions).
This commit introduces no functional changes but cleans up the
code and shrinks the surface area by removing dead and unused
code, leveraging std:: alternatives to hand-rolled code and
improving comments and documentation.
The script, when invoked by a server operator can collect information
useful for debugging, while attempting to redact potentially sensitive
data.
It contained no explanation or other exposition to allow people who
look at the file but aren't familiar with shell scripts to understand
its purpose.
The built-in watchdog is simplistic and can, sometimes, cause problems
especially on systems that have the ability to automatically start and
monitor processes.
This commit removes the sustain system entirely, changes the handling
of the SIGTERM signal to properly terminate the process and improves
the error message reported to the user when the command line used to
start `rippled` is incorrect and malformed.
Entries in the ledger are located using 256-bit locators. The locators
are calculated using a wide range of parameters specific to the entry
whose locator we are calculating (e.g. an account's locator is derived
from the account's address, whereas the locator for an offer is derived
from the account and the offer sequence.)
Keylets enhance type safety during lookup and make the code more robust,
so this commit removes most of the earlier code, which used naked
uint256 values.
This commit removes obsolete comments, dead or no longer useful
code, and workarounds for several issues that were present in older
compilers that we no longer support.
Specifically:
- It improves the transaction metadata handling class, simplifying
its use and making it less error-prone.
- It reduces the footprint of the Serializer class by consolidating
code and leveraging templates.
- It cleanups the ST* class hierarchy, removing dead code, improving
and consolidating code to reduce complexity and code duplication.
- It shores up the handling of currency codes and the conversation
between 160-bit currency codes and their string representation.
- It migrates beast::secure_erase to the ripple namespace and uses
a call to OpenSSL_cleanse instead of the custom implementation.
A deliberately malformed token can cause the server to crash during
startup. This is not remotely exploitable and would require someone
with access to the configuration file of the server to make changes
and then restart the server.
Acknowledgements:
Guido Vranken for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.
Ripple is generously sponsoring a bug bounty program for the
rippled project. For more information please visit:
https://ripple.com/bug-bounty
Currently there is no mechanism for a validator to report the
version of the software it is currently running. Such reports
can be useful for those who are developing network monitoring
dashboards and server operators in general.
This commit, if merged, defines an encoding scheme to encode
a version string into a 64-bit unsigned integer and adds an
additional optional field to validations.
This commit piggybacks on "HardenedValidations" amendment to
determine whether version information should be propagated
or not.
The general encoding scheme is:
XXXXXXXX-XXXXXXXX-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY
X: 16 bits identifying the particular implementation
Y: 48 bits of data specific to the implementation
The rippled-specific format (implementation ID is: 0x18 0x3B) is:
00011000-00111011-MMMMMMMM-mmmmmmmm-pppppppp-TTNNNNNN-00000000-00000000
M: 8-bit major version (0-255)
m: 8-bit minor version (0-255)
p: 8-bit patch version (0-255)
T: 11 if neither an RC nor a beta
10 if an RC
01 if a beta
N: 6-bit rc/beta number (1-63)
This commit introduces the "HardenedValidations" amendment which,
if enabled, allows validators to include additional information in
their validations that can increase the robustness of consensus.
Specifically, the commit introduces a new optional field that can
be set in validation messages can be used to attest to the hash of
the latest ledger that a validator considers to be fully validated.
Additionally, the commit leverages the previously introduced "cookie"
field to improve the robustness of the network by making it possible
for servers to automatically detect accidental misconfiguration which
results in two or more validators using the same validation key.
- Add missing `#include` in `ripple/core/JobTypeInfo.h`
- Protect version string from clang-format in
`ripple/protocol/impl/BuildInfo.cpp`.
`Builds/CMake/RippledVersion.cmake` searches for this line by pattern.
Existing per-thread PRNGs are individually initialized using calls
to std::random_device.
If merged, this commit will use a single PRNG, initialized from
std::random_device on startup, to seed the thread-specific PRNGs.
Acknowledgements:
Thomas Snider, who reported this issue to Ripple on April 8, 2020.
Historically strand re-execute log messages have been treated as
errors. However in the vast majority of cases these log messages
are caused by well understood mechanics in the payment engine.
So usually these log messages should be treated as warnings.
The automated build system only builds packages signed with a list of
approved keys. This is a security measure to prevent someone who gains
push access to the repository from producing potentially malicious
packages that are signed by Ripple's trusted private keys.
Moving this list to the new location makes it easy to add and delete
new keys to the list.
* scoped_lock is now a std name with subtly different semantics
compared to lock_guard. Namely it can be used to lock 0 or
more mutexes. This is valuable, but can also be accidentally
used to lock 0 mutexes when 1 was intended, creating a
run-time error.
Therefore, if and when we use scoped_lock, extra care needs to
be taken in reviewing that code to ensure it doesn't
accidentally lock 0 mutexes when 1 was intended. To aid in
such careful reviewing, the use of the name scoped_lock should
be limited to those cases where the number of mutexes is not
exactly one.
* canonicalize_replace_cache
* canonicalize_replace_client
Now it is clear at the call site that if there are
duplicate copies of the data between the cache and
the caller, which copy gets replaced.
Additionally data parameter is now const-correct.
If it is not going to be replaced (canonicalize_replace_cache),
then the shared_ptr to the client data is const.
* The [network_id] option allows three string values:
- main: the XRP Ledger
- testnet: the Testnet operated by Ripple.
- devnet: the development test network operated by Ripple.
* Peers negotiate compression via HTTP Header "X-Offer-Compression: lz4"
* Messages greater than 70 bytes and protocol type messages MANIFESTS,
ENDPOINTS, TRANSACTION, GET_LEDGER, LEDGER_DATA, GET_OBJECT,
and VALIDATORLIST are compressed
* If the compressed message is larger than the uncompressed message
then the uncompressed message is sent
* Compression flag and the compression algorithm type are included
in the message header
* Only LZ4 block compression is currently supported
The payment engine restricts payment paths so two steps do not input the
same Currency/Issuer or output the same Currency/Issuer. This check was
skipped when the path started or ended with XRP. An example of a path
that was incorrectly accepted was: XRP -> //USD -> //XRP -> EUR
This patch enables the path loop check for paths that start or end with
XRP.
* Make ShardArchiveHandler a singleton.
* Add state database for ShardArchiveHandler.
* Use temporary database for SSLHTTPDownloader downloads.
* Make ShardArchiveHandler a Stoppable class.
* Automatically resume interrupted downloads at server start.
* Reduce lock scope on all public functions
* Use TaskQueue to process shard finalization in separate thread
* Store shard last ledger hash and other info in backend
* Use temp SQLite DB versus control file when acquiring
* Remove boost serialization from cmake files
When computing rates for offers, an STAmount's value can be out of can be out of
range (before canonicalizing). There was an assert that could incorrectly fire
in some cases. This patch removes that assert.
The newest MSVC 19.25.28610.4 does not build rocksdb. During the
Travis CI Windows job, the vs_BuildTools.exe automatically
downloads the newest version of the compiler. This fix forces the
install of MSVC 19.24.28314.0 to build rocksdb.
The fix1781 amendment was incorrectly introduced during conflict
resolution and support for it is not included at this time. This commit
removes the definition of the amendment identifier.
A review of the lag ratchet code revealed that we were using
the long-term master public keys of trusted validators, when
we should have been using the ephemeral public keys instead.
As a result, the lag ratchet code would be effectively
inoperable.
- Add support for all transaction types and ledger object types to gRPC
implementation of tx and account_tx.
- Create common handlers for tx and account_tx.
- Remove mutex and abort() from gRPC server. JobQueue is stopped before
gRPC server, with all coroutines executed to completion, so no need for
synchronization.
* Whenever a node downloads a new VL, send it to all peers that
haven't already sent or received it. It also saves it to the
database_dir as a Json text file named "cache." plus the public key of
the list signer. Any files that exist for public keys provided in
[validator_list_keys] will be loaded and processed if any download
from [validator_list_sites] fails or no [validator_list_sites] are
configured.
* Whenever a node receives a broadcast VL message, it treats it as if
it had downloaded it on it's own, broadcasting to other peers as
described above.
* Because nodes normally download the VL once every 5 minutes, a single
node downloading a VL with an updated sequence number could
potentially propagate across a large part of a well-connected network
before any other nodes attempt to download, decreasing the amount of
time that different parts of the network are using different VLs.
* Send all of our current valid VLs to new peers on connection.
This is probably the "noisiest" part of this change, but will give
poorly connected or poorly networked nodes the best chance of syncing
quickly. Nodes which have no http(s) access configured or available
can get a VL with no extra effort.
* Requests on the peer port to the /vl/<pubkey> endpoint will return
that VL in the same JSON format as is used to download now, IF the
node trusts and has a valid instance of that VL.
* Upgrade protocol version to 2.1. VLs will only be sent to 2.1 and
higher nodes.
* Resolves#2953
* Example: gcc.Debug will use the the default version of gcc installed on the
system. gcc-9.Debug will use version 9, regardless of the default. This will
be most useful when the default is older than required or desired.
* When an unknown amendment reaches majority, log an error-level
message, and return a `warnings` array on all successful
admin-level RPC calls to `server_info` and `server_state` with
a message describing the problem, and the expected deadline.
* In addition to the `amendment_blocked` flag returned by
`server_info` and `server_state`, return a warning with a more
verbose description when the server is amendment blocked.
* Check on every flag ledger to see if the amendment(s) lose majority.
Logs again if they don't, resumes normal operations if they did.
The intention is to give operators earlier warning that their
instances are in danger of being amendment blocked, which will
hopefully motivate them to update ahead of time.
* update EP and find package requirements
* minor protobuf/libarchive build fixes
* change travis release builds to nounity to
ameliorate vm memory exhaustion.
FIXES: #3223, #3232
* In and Out parameters were swapped when calculating the rate
* In and out qualities were not calculated correctly; use existing functions
to get the qualities
* Added tests to check that theoretical quality matches actual computed quality
* Remove in/out parameter from qualityUpperBound
* Rename an overload of qualityUpperBound to adjustQualityWithFees
* Add fix amendment
STAmount::soTime and soTime2 were time based "amendment like"
switches to control small changes in behavior for STAmount.
soTime2, which was the most recent, was dated Feb 27, 2016.
That's over 3 years ago.
The main reason to retain these soTimes would be to replay
old transactions. The likelihood of needing to replay a
transaction from over three years ago is pretty low. So it
makes sense to remove these soTime values.
In Flow_test the testZeroOutputStep() test is removed. That
test started to fail when the STAmount::soTimes were removed.
I checked with the original author of the test. He said
that the code being tested by that unit test has been removed,
so it makes sense to remove the test. That test is removed.
* use tagged containers for pkg build
* update build images
* continue to build container images in pipeline, but allow
failure (non-block)
* limit travis macos cache
* add vs2019 windows to travis
* remove xcode 9 travis build
* remove clang5/6 from CI and update min version of Clang required in
cmake
* break windows CI build into stages to reduce timeouts
* update datelib
* add if condition to travis builds to allow commit message to limit
builds by platform
FIXES: #2847
* Transactions that are submitted with the fail_hard flag
and that result in any TER code besides tesSUCCESS shall
be neither queued nor held.
[FOLD] Keep tec results out of the open ledger when fail_hard:
* Improve TransactionStatus const correctness, and remove redundant
`local` check
* Check open ledger tx count in fail_hard tests
* Fix some wrapping
* Remove duplicate test
Remove the implicit conversion from int64 to XRPAmount. The motivation for this
was noticing that many calls to `to_string` with an integer parameter type were
calling the wrong `to_string` function. Since the calls were not prefixed with
`std::`, and there is no ADL to call `std::to_string`, this was converting the
int to an `XRPAmount` and calling `to_string(XRPAmount)`.
Since `to_string(XRPAmount)` did the same thing as `to_string(int)` this error
went undetected.
Prior to this commit, the queue and execution times for individual jobs
were reported indepedently and could, potentially, be out of sync. This
change reports both values when either one of the exceeds the reporting
threshold.
It's possible an overloaded job queue is causing false alarms on the deadlock
detector. Log a fatal message after 90s, declare a logic error after 600s.
If merged, this commit will report additional information in the
response to the submit command; this will make it easier for developers
to accurately track the status of transaction submission.
Fixes#2851
Treat all `#` characters in config files as comments (and remove)
*unless* the `#` is immediately preceded by `\`. Write a warning
to log file when trailing comments are found/ignored in the config
to let operators know that the treatment of trailing `#` has changed.
Fixes#3121
The 'network_id' option allows an administrator to specify to which
network they intend a server to connect. Servers can leverage this
information to optimize routing and prune automatically discovered
cross-network connections.
This commit will, if merged:
- add support for the devnet keyword, which corresponds to network ID #2;
- report the network ID, if one is configured, in server_info
The `node_size` configuration option is used to automatically
configure various parameters (cache sizes, timeouts, etc) for
the server.
A previous commit included changes that caused incorrect values
to be returned which can result in sub-optimal performance that
can manifest as difficulty syncing to the network, or increased
disk I/O and/or memory usage. The problem was introduced with
commit 66fad62e66.
This commit, if merged, fixes the code to ensure that the correct
values are returned and introduces a compile-time check to prevent
this issue from reoccurring.
The existing platform detection code was derived from the old Beast
library, which was, itself, derived from JUCE.
This commit removes that code and replaces it with the Boost.Predef
library which defines a consistent set of compiler, architecture,
operating system, library, and other version numbers.
For more on Boost.Predef, please see the Boost documentation. The
documentation for the current version as of this writing is at:
https://www.boost.org/doc/libs/1_71_0/doc/html/predef.html
This commit restructures the HTTP based protocol negotiation that `rippled`
executes and introduces support for negotiation of compression for peer
links which, if implemented, should result in significant bandwidth savings
for some server roles.
This commit also introduces the new `[network_id]` configuration option
that administrators can use to specify which network the server is part of
and intends to join. This makes it possible for servers from different
networks to drop the link early.
The changeset also improves the log messages generated when negotiation
of a peer link upgrade fails. In the past, no useful information would
be logged, making it more difficult for admins to troubleshoot errors.
This commit also fixes RIPD-237 and RIPD-451
* The `tx` command now supports min_ledger and max_ledger fields.
* If the requested transaction isn't found and these fields are
provided, the error response indicates whether or not every
ledger in the the provided range was searched.
This fixes#2924
* adding package signing steps for rpm and deb
* first spike at GPG signing with CI and containers
* refine ubuntu portion
* get correct gpg package version
* adding CentOS support
* fixing errors in installing gpg on ubuntu
* base64 decode the GPG key
* fixing line continuations
* revised package signing, looking for package artifacts
* add dpkg-sig to ubuntu image
* sign all deb packges
* add passphrase to GPG process
* repeat yo slef on dpkg
* sign all the rpm packages too
* install rpm-sign in the CentOS docker image
* loop through rpm files
* no need for PIN on GPG signing
Collecting the returned and expected values in sets only works if there are no
duplicates. The implementation is changed to use sorted vectors to fix this case.
When the Env::AppBundle constructor throws an exception
it still needs to run ~AppBundle(), otherwise the JobQueue
isn't properly shut down. Specifically the JobQueue
can destruct without waiting on outstanding jobs in the
queue.
This change ensures that if Env::AppBundle constructor
throws, Env::AppBundle::~AppBundle() runs.
This fixes the unit test crash exposed by PR #3047.
FIXES: #3106
Different versions of protobuf produce subtly different
results when given invalid message payloads. This leads to
subtly different behavior when we try to deserialize these
invalid messages. As such, we can't tie success to a
particular exception.
The XRP Ledger utilizes an account model. Unlike systems based on a UTXO
model, XRP Ledger accounts are first-class objects. This design choice
allows the XRP Ledger to offer rich functionality, including the ability
to own objects (offers, escrows, checks, signer lists) as well as other
advanced features, such as key rotation and configurable multi-signing
without needing to change a destination address.
The trade-off is that accounts must be stored on ledger. The XRP Ledger
applies reserve requirements, in XRP, to protect the shared global ledger
from growing excessively large as the result of spam or malicious usage.
Prior to this commit, accounts had been permanent objects; once created,
they could never be deleted.
This commit introduces a new amendment "DeletableAccounts" which, if
enabled, will allow account objects to be deleted by executing the new
"AccountDelete" transaction. Any funds remaining in the account will
be transferred to an account specified in the deletion transaction.
The amendment changes the mechanics of account creation; previously
a new account would have an initial sequence number of 1. Accounts
created after the amendment will have an initial sequence number that
is equal to the ledger in which the account was created.
Accounts can only be deleted if they are not associated with any
obligations (like RippleStates, Escrows, or PayChannels) and if the
current ledger sequence number exceeds the account's sequence number
by at least 256 so that, if recreated, the account can be protected
from transaction replay.
* replace boost::beast::detail::iequals with boost::iequals
* replace deprecated `buffers` function with `make_printable`
* replace boost::beast::detail::ascii_tolower with lambda
* add missing includes
The validation stream only reported the ephemeral signing key for validators
which use manifests. This made tracking unnecessarily difficult for clients
processing the data stream.
With this change, the validator's long-term master public key is also
included.
This commit fixes#3005
* Provide proposing validator's master key in the validation stream
subscription JSON responses.
Implement code review changes.
FIXES: #3005
* Explain that Arch/Manjaro/etc. need `-Dstatic=OFF` during the configure step
* move configuration options closer to that step
* separate sub-headers for configuration and build
Different compilers and handling the shadow warning differently. In particular,
some are warning about types being shadowed by variables. Until these can be
resolved the shadow warning is being disabled.
Note: the shadow warning was originally enabled to help with the structured
bindings patch. As that is now complete, it's less important to keep this
warning.
FIXES: #2527
* define custom docker image for travis-linux builds based on
package build image
* add macos builds
* add windows builds (currently allowed to fail)
* improve build and shell scripts as required for the CI envs
* add asio timer latency workaround
* omit several manual tests from TravisCI which cause memory exhaustion
This commit allows server operators to reserve slots for specific
peers (identified by the peer's public node identity) and to make
changes to the reservations while the server is operating.
This commit closes#2938
- Add docker container tags for "latest_BRANCH"
- Prevent different branches from overwriting deb repo artifacts
- Manual approval always required before pushing to prod
The original intent was that RPC error codes were not stable.
But those codes were made available through the API, so some
users came to depend on the code values. This change adapts
to the current state of affairs.
Manifests which are revoked can include ephemeral keys although doing
so does not make sense: a revoked manifest isn't used for signing and
so don't need to define an ephemeral key.
A running instance of the server tracks the number of protocol messages
and the number of bytes it sends and receives.
This commit makes the counters more granular, allowing server operators
to better track and understand bandwidth usage.
* Add construction and assignment from a generic
contiguous container. Both compile-time and run time
safety checks are made to ensure the safety of this
conversion.
* Remove base_uint::copyFrom. The generic copy assignment
operator now does this functionality with enhanced
safety and better syntax.
* Remove construction from and dedendence on Blob.
The generic constructor and assignment now handle this
functionality.
* Fix client code to adhere to this new API.
* Removed the use of fromVoid in PeerImp.cpp as it was
an inappropriate use of this dangerous API. The
generic container constructors do it with enhanced
safety and better syntax.
* Rename data member pn to data_ and make it private.
* Remove constraint from hash_append
* Remove array_type alias
This PR addresses a problem where the server could hang indefinitely
on shutdown. The cause of the problem is the SNTPClock class was not
binding the socket to an endpoint on initialization. This can create
an error sent to the read handler. Unfortunately, the handler ignores
the error, reads again and enters into a loop preventing the
io_service from ever completing.
- Explain how to bind to both IPv4 and IPv6 interfaces
- Provide a hint in the default [port_peer] section
- Do not enable it by default
Note that on Linux, use of '::' and IPv4-mapped IPv6 depends on a sysctl value
setting 'net.ipv6.bindv6only = 0' which seems to be the default on most Linux
distributions.
- Use `std::lock` when grabbing multiple mutexes to ensure consistent
locking order and avoid deadlocks.
- Reduce the scope of the master mutex lock by relesing it prior to
calling setHeartbeatTimer
A tiny input amount to a payment step can cause this step to output zero. For
example, if a previous steps outputs a dust amount of 10^-80, and this step is a
IOU -> XRP offer, the offer may output zero drops. In this case, call the strand
dry. Before this patch, an error would be logged, the strand would be called
dry; in debug mode an assert triggered.
Note, this patch is not transaction breaking, as the caller did not user the ter
code. The caller only checked for success or failuer.
This patch addresses github issue issue reported here:
https://github.com/ripple/rippled/issues/2929
This patch removes calls to several deprecated asio functions.
* `io_service::post` becomes `post` (free function)
* `io_service::work` becomes `executor_work_guard`
* `io_service::wrap` becomes `bind_executor`
* `get_io_context` becomes `get_executor` or `get_executor().context()`
This patch was tested with boost 1.69 and 1.70. The functions
`ripple::get_lowest_layer` and `beast::create_waitable_timer` are required to
handle a breaking difference between these versions. When rippled no longer
needs to support pre 1.70 boost versions, both of these functions may be
removed, and the waitable timer injections may also be removed.
The new parse logic is more strict but handles more cases. If an exception
is thrown, just bail.
* Allow parsing unenclosed IPv6 addresses without port
* Improve string construction
* Reduce nesting levels of code
The XRP Ledger allows an account to authorize a secondary key pair,
called a regular key pair, to sign future transactions, while keeping
the master key pair offline.
The regular key pair can be changed as often as desired, without
requiring other changes on the account.
If merged, this commit corrects a minor technical flaw which would
allow an account holder to specify the master key as the account's
new regular key.
The change is controlled by the `fixMasterKeyAsRegularKey` amendment
which, if enabled, will:
1. Prevent specifying an account's master key as the account's
regular key.
2. Prevent the "Disable Master Key" flag from incorrectly affecting
regular keys.
Before this patch, jtx allowed non-invocable functions to be passed to
operator(). However, these arguments are ignored. This caused erronious code
code such as:
```
env (offer (account_to_test, BTC (250), XRP (1000)),
offers (account_to_test, 1));
```
While it looks like the number of offers are checked, they are not. The `offers`
funclet is never run. While we could modify jtx to make the above code correct,
a cleaner solution is to run post conditions in a `require` statement after a
transasction runs.
At this point all of the jss::* names are defined in the same
file. That file has been named JsonFields.h. That file name
has little to do with either JsonStaticStrings (which is what
jss is short for) or with jss. The file is renamed to jss.h
so the file name better reflects what the file contains.
All includes of that file are fixed. A few include order
issues are tidied up along the way.
Formerly an SOTemplate was default constructed and its elements
added using push_back(). This left open the possibility of a
malformed SOTemplate if adding one of the elements caused a throw.
With this commit the SOTemplate requires an initializer_list of
its elements at construction. Elements may not be added after
construction. With this approach either the SOTemplate is fully
constructed with all of its elements or the constructor throws,
which prevents an invalid SOTemplate from even existing.
This change requires all SOTemplate construction to be adjusted
at the call site. Those changes are also in this commit.
The SOE_Flags enum is also renamed to SOEStyle, which harmonizes
the name with other uses in the code base. SOEStyle elements
are renamed (slightly) to have an "soe" prefix rather than "SOE_".
This heads toward reserving identifiers with all upper case for
macros. The new style also aligns with other prominent enums in
the code base like the collection of TER identifiers.
SOElement is adjusted so it can be stored directly in an STL
container, rather than requiring storage in a unique_ptr.
Correspondingly, unique_ptr usage is removed from both
SOTemplate and KnownFormats.
The new 'Domain' field allows validator operators to associate a domain
name with their manifest in a transparent and independently verifiable
fashion.
It is important to point out that while this system can cryptographically
prove that a particular validator claims to be associated with a domain
it does *NOT* prove that the validator is, actually, associated with that
domain.
Domain owners will have to cryptographically attest to operating particular
validators that claim to be associated with that domain. One option for
doing so would be by making available a file over HTTPS under the domain
being claimed, which is verified separately (e.g. by ensuring that the
certificate used to serve the file matches the domain being claimed) and
which contains the long-term master public keys of validator(s) associated
with that domain.
Credit for an early prototype of this idea goes to GitHub user @cryptobrad
who introduced a PR that would allow a validator list publisher to attest
that a particular validator was associated with a domain. The idea may be
worth revisiting as a way of verifying the domain name claimed by the
validator's operator.
Resource limits were not properly applied to connections with
known IP addresses but no corresponding users.
Add unit tests for unlimited vs. limited ports.
An audit showed that a number of the RPC error codes in
ErrorCodes.h are no longer used in the code base. The unused
codes were removed from the file along with their support code
in ErrorCodes.cpp.
The ledger already declared a transaction that is both single-
and multi-signing malformed. This just adds some checking in
the signing RPC commands (like submit and sign_for) which allows
that sort of error to be identified a bit closer to the user.
In the process of adding this code a bug was found in the
RPCCall unit test. That bug is fixed as well.
If the number of peers a server has is below the configured
minimum peer limit, this commit will properly transition the
server's state to "disconnected".
The default limit for the minimum number of peers required was
0 meaning that a server that was connected but lost all its
peers would never transition to disconnected, since it could
never drop below zero peers.
This commit redefines the default minimum number of peers to 1
and produces a warning if the server is configured in a way
that will prevent it from ever achieving sufficient connectivity.
If a server is configured to support crawl, it will report the
IP addresses of all peers it is connected to, unless those peers
have explicitly opted out by setting the `peer_private` option
in their config file.
This commit makes servers that are configured as validators
opt out of crawling.
Several commands allow a user to retrieve a server's status. Commands
will typically limit disclosure of information that can reveal that a
particular server is a validator to connections that are not verified
to make it more difficult to determine validators via fingerprinting.
Prior to this commit, servers configured to operate as validators
would, instead of simply reporting their server state as 'full',
augment their state information to indicate whether they are
'proposing' or 'validating'.
Servers will only provide this enhanced state information for
connections that have elevated privileges.
Acknowledgements:
Ripple thanks Markus Teufelberger for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
The /crawl API endpoint allows developers to examine the structure of
the XRP Ledger's overlay network.
This commit adds additional information about the local server to the
/crawl endpoint, making it possible for developers to create data-rich
network-wide status dashboards.
Related:
- https://developers.ripple.com/peer-protocol.html
- https://github.com/ripple/rippled-network-crawler
When deserializing specially crafted data, the code would ignore certain
types of errors. Reserializing objects created from such data results in
failures or generates a different serialization, which is not ideal.
Also addresses: RIPD-1677, RIPD-1682, RIPD-1686 and RIPD-1689.
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these issues.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
Specially crafted messages could cause the server to buffer large
amounts of memory which could increase memory pressure.
This commit changes how messages are buffered and imposes a limit
on the amount of data that the server is willing to buffer.
Acknowledgements:
Aaron Hook for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find. For information
on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
The constructor would previously assert that the specified buffer pointer
was non-null, even if the buffer size is specified as 0. While reasonable,
this also makes it more difficult to use this API.
* Using txnsExpected_, which is influenced by both the config
and network behavior, can reserve far too much or far too
little memory, wasting time and resources.
* Not an issue during normal operation, but a user could
cause problems on their local node with extreme configuration
settings.
* initFee was using a lot of logic that could look unclear. Add
some documentation explaining why certain values were used.
* Because initFee had side effects, callers needed to repeat the
max queue size computation, making the initial problem more
likely. Instead, return the max queue size value, so the caller
can reuse it.
* A newer test (testInFlightBalance()) was incorrectly using a
hard-coded queue limit. Fix it to use initFee's new return
value.
The --rpc_port command-line option is effectively ignored. We construct
an `Endpoint` with the given port, but then drop it on the floor.
(Perhaps the author thought the `Endpoint::at_port` method is a mutation
instead of a transformation.) This small change adds the missing
assignment to hold on to the new endpoint.
Fixes#2764
This changeset ensures the preferred ledger calculation
properly distinguishes the absence of trusted validations
from a preferred ledger which is the genesis ledger.
The `STObject` member function `setType()` has been renamed to
applyTemplate() and modified to throw if there is a template
mismatch.
The error description in the exception is, in certain cases,
used, to better indicate why a particular transaction was
considered ill formed.
Fixes#2585.
`Json::Value::isConvertibleTo` indicates that unsigned integers and
reals are convertible to string, but trying to do so (with
`Json::Value::asString`) throws an exception because its internal switch
is missing these cases. This change fills them in (and adds tests).
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing this issue.
Closes#2778
Although `parseURL` used a regex to pull the authority out of the URL
being parsed, it performed manual parsing of the hostname and port.
This commit rolls the parsing of the username and password, if any,
directly into the regex. The hostname can be a name, an IPv4 or an
IPv6 address.
Fixes#2751
* Adds local file:// URL support to the [validator_list_sites] stanza.
The file:// URL must not contain a hostname. Allows a rippled node
operator to "sideload" a new list if their node is unable to reach
a validator list's web site before an old list expires. Lists
loaded from a file will be validated in the same way a downloaded
list is validated.
* Generalize file/dir "guards" from Config test so they can be reused
in other tests.
* Check for error when reading validators.txt. Saves some parsing and
checking of an empty string, and will give a more meaningful error.
* Completes RIPD-1674.
* Relevant when deciding whether an account can queue multiple
transactions. If the potential spend of the already queued
transactions would dip into the reserve, the reserve is
preserved for fees.
* Also change several direct modifications of the owner count to
call adjustOwnerCount to preserve overflow checking.
* Update related unit testcase
* Resolves#2251
Perform some extra checks on the close time and sequence number
of a candidate for network consensus ledger. This tightens
defenses against some "insane/hostile supermajority" attacks.
Under certain conditions, we could call `memcpy` or `memcmp` with a null
source pointer. Even when specifying 0 as the amount of data to copy this
could result in undefined behavior under the C and C++ standards.
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these issues.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
The XRP Ledger is designed to be censorship resistant. Any attempt to
censor transactions would require coordinated action by a majority of
the system's validators.
Importantly, the design of the system is such that such an attempt is
detectable and can be easily proven since every validators must sign
the validations it publishes.
This commit adds an automated censorship detector. While the server is
in sync, the detector tracks all transactions that, in the view of the
server, should have been included and issues warnings of increasing
severity for any transactions which, have not after several rounds.
When Ed25519 support was added to ripple-lib, a way to specify
whether a seed should be used to derive a "classic" secp256k1
keypair or a "new" Ed25519 keypair was needed, and the
requirements were that:
1. previously seeds would, correctly, generate a secp256k1
keypair.
2. users would not have to know about whether the seed was
used to generate a secp256k1 or an Ed25519 keypair.
To address these requirements, the decision was made to encode
the type of key within the seed and a custom encoding was
designed.
The encoding uses a token type of 1 and prefixes the actual
seed with a 2 byte header, selected to ensure that all such
keypairs will, when encoded, begin with the string "sEd".
This custom encoding is non-standard and was not previously
documented; as a result, it is not widely supported and other
sofware may treat such keys as invalid. This can make it
difficult for users that have stored such a seed to use
wallets or other tooling that is not based on ripple-lib.
This commit adds support to rippled for automatically
detecting and properly handling such seeds.
The 'validation_seed' RPC command was used to change the validation
key used by a validator at runtime.
Its implementation was commented out with commit fa796a2eb5
which has been included in the codebase since the 0.30.0 release
and there are no plans to reintroduce the functionality at this
point.
Validator operators should migrate to using validator manifests
instead.
This fixes#2748.
The FeeEscalation amendment has been enabled on the XRP Ledger network
since May 19, 2016. The transaction which activated this amendment is:
5B1F1E8E791A9C243DD728680F108FEF1F28F21BA3B202B8F66E7833CA71D3C3.
This change removes all conditional code based around the FeeEscalation
amendment, but leaves the amendment definition itself since removing the
definition would cause nodes to think an unknown amendment was activate
causing them to become amendment blocked.
The commit also removes the redundant precomputed hashes from the
supportedAmendments vector.
Problem:
- There are only a few call sites to cachedRead, and all of them
currently do more work than is required since we know the type in each
case.
Solution:
- "Inline" the codepath to cachedRead, but do not check if the type is
valid. In all such call sites, we know the keylet to read directly.
This fixes#2550
The WaitableEvent class was a leftover from the pre-Boost
version of Beast and used Windows- and pthread-specific
APIs.
This refactor replaces that functionality by using only
interfaces provided by the C++ standard, making the code
more portable.
Closes#2402.
Many of the warnings on Windows were not resolved, just
silenced with _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS.
They need to be resolved in a future commit.
* If rippled is not synced to the network, `fee` will return a
"no network" error instead of the possibly confusing "not enabled"
error.
* Resolves RIPD-1588
A validator that was configured to use a published validator list could
exhibit aberrent behavior if that validator list expired.
This commit introduces additional logic that makes validators operating
with an expired validator list bow out of the consensus process instead
of continuing to publish validations. Normal operation will resume once
a non-expired validator list becomes available.
This commit also enhances status reporting when using the `server_info`
and `validators` commands. Before, only the expiration time of the list
would be returned; now, its current status is also reported in a format
that is clearer.
A validator that was configured to use a published validator list could
exhibit aberrent behavior if that validator list expired.
This commit introduces additional logic that makes validators operating
with an expired validator list bow out of the consensus process instead
of continuing to publish validations. Normal operation will resume once
a non-expired validator list becomes available.
This commit also enhances status reporting when using the `server_info`
and `validators` commands. Before, only the expiration time of the list
would be returned; now, its current status is also reported in a format
that is clearer.
Problem:
- There are several specific overloads with some custom code that can be
easily replaced using Boost.Hex.
Solution:
- Introduce `strHex(itr, itr)` to return a string given a begin and end
iterator.
- Remove `strHex(itr, size)` in favor of the `strHex(T)` where T is
something that has a `begin()` member function. This allows us to
remove the strHex overloads for `std::string`, Blob, and Slice.
* For example Visual Studio, XCode. This will allow easily working with
any file in the IDE.
* Also ignore the file created by Visual Studio when using cmake
integration.
* Use conditional for unity/nounity sources (h/t @mellery451)
- allow private token for jenkins/codecov
- add custom targets for gcc/clang to generate codecov reports
- use CMake coverage target in jenkins build
- optional coverage_test argument when configuring the build
Reduces the account reserve for a multisigning SignerList from
(conditionally) 3 to 10 OwnerCounts to (unconditionally) 1
OwnerCount. Includes a transition process.
* When increasing the expected ledger size, add on an extra 20%.
* When decreasing the expected ledger size, take the minimum of the
validated ledger size or the old expected size, and subract another 50%.
* Update fee escalation documentation.
* Refactor the FeeMetrics object to use values from Setup
As described in #2314, when an offer executed with `Fill or Kill`
semantics, the server would return `tesSUCCESS` even if the order
couldn't be filled and was aborted. This would require additional
processing of metadata by users to determine the effects of the
transaction.
This commit introduces the `fix1578` amendment which, if enabled,
will cause the server to return the new `tecKILLED` error code
instead of `tesSUCCESS` for `Fill or Kill` orders that could not
be filled.
Additionally, the `fix1578` amendment will prevent the setting of
the `No Ripple` flag on trust lines with negative balance; trying
to set the flag on such a trust line will fail with the new error
code `tecNEGATIVE_BALANCE`.
Fixes: RIPD-1648
- use ExternalProject for snappy, lz4, SOCI, and sqlite3
- use FetchContent for NuDB
- update SOCI from 79e222e3c2278e6108137a2d26d3689418b37544 to
3a1f602b3021b925d38828e3ff95f9e7f8887ff7
- update lz4 from c10863b98e1503af90616ae99725ecd120265dfb to v1.8.2
- update sqlite3 from 3.21 to 3.24
- update snappy from b02bfa754ebf27921d8da3bd2517eab445b84ff9 to 1.1.7
- update NuDB from 00adc6a4f16679a376f40c967f77dfa544c179c1 to 1.0.0
When validators publish a proposal, they include the close time that they
believe the new ledger should have, and the network attempts to reach
consensus on that.
Instead of delaying consensus if no close time has the required majority
the servers can "agree to disagree"; if this happens, they switch to
proposing a close time of 0, and the network avalanches to that value.
If that occurs, determinstic rules record the new ledger's close time as
being one second later than its parent, and set a flag indicating that
no consensus on the close time was reached.
The wire protocol decoder would incorrectly filter such proposals, so
that they would not be seen by the higher level consensus engine.
This commit removes the low-level filtering, and allows higher level
code to filter out stale proposals instead.
In order to facilitate transaction signing, `rippled` offers the `sign` and
`sign_for` and `submit` commands, which, given a seed, can be used to sign or
sign-and-submit transactions. These commands are accessible from the command
line, as well as over the WebSocket and RPC interfaces that `rippled` can be
configured to provide.
These commands, unfortunately, have significant security implications:
1. They require divulging an account's seed (commonly known as a "secret
key") to the server.
2. When executing these commands against remote servers, the seeds can be
transported over clear-text links.
3. When executing these commands over the command line, the account
seed may be visible using common tools that show running processes
and may potentially be inadvertently stored by system monitoring
tools or facilities designed to maintain a history of previously
typed commands.
While this commit cannot prevent users from issuing these commands to a
server, whether locally or remotely, it restricts the `sign` and `sign_for`
commands, as well as the `submit` command when used to sign-and-submit,
so that they require administrative privileges on the server.
Server operators that want to allow unrestricted signing can do so by
adding the following stanza to their configuration file:
[signing_support]
true
Ripple discourages server operators from doing so and advises against using
these commands, which will be removed in a future release. If you rely on
these commands for signing, please migrate to a standalone signing solution
as soon as possible. One option is to use `ripple-lib`; documentation is
available at https://developers.ripple.com/rippleapi-reference.html#sign.
If the commands are administratively enabled, the server includes a warning
on startup and adds a new field in the resulting JSON, informing the caller
that the commands are deprecated and may become unavailable at any time.
Acknowledgements:
Jesper Wallin for reporting this issue to Ripple.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
Exclude several libraries from build when we are included in a
super-project (this is the case when someone only wants to use
xrpl_core). Force several target (deprecated) params to be cache
variables since they are now exposed as options.
Increase page size for SQLite transaction database upon creation
Provide diagnostics for transaction db page usage.
Shut down rippled gracefullly if transaction db is running out of pages.
Add new rippled maintenance command line option to cause new page size
to take effect.
Switch to target-oriented dependencies. Use imported targets for
dependencies (openssl, boost). Localize FindBoost to remove cmake
version dependence for latest boost support. Logically separate
"ripple-libpp" core sources and add install targets.
Add ninja build for msvc. Add two clang sanitizer builds. Misc script
changes to work with latest modernized cmake.
Certain versions of the Beast HTTP & WebSocket library can
generate exceptions, which unless caught, will result in
unexpected behavior.
Acknowledgements:
Ripple thanks Thomas Snider for originally noticing this
issue and responsibly disclosing it to Ripple.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
Based on a TODO comment in DisputedTX.h, it seems at one point the
data members of ConsensusParms were macros. Now that they are not,
we should spell them like other data members (without all uppercase).
After some discussion on https://github.com/ripple/rippled/pull/2595
we have decided that the allocator should not be plumbed through
the KeyCache class template. As such, remove the comment suggesting
to push the allocator through.
All listed validators are trusted and quorum is 80% of trusted
validators regardless of the number of:
* configured published lists
* listed or trusted validators
* recently seen validators
Exceptions:
* A listed validator whose master key has been revoked is not trusted
* Custom minimum quorum (specified with --quorum in the command line)
is used if the normal quorum appears unreachable based on the number
of recently received validators.
RIPD-1640
std::random_shuffle is deprecated in C++14 and removed completely
in C++17. The two-iterator version of std::random_shuffle usually
depends on std::rand and also on a global state. The preferred
replacement is to use std::shuffle with a pseudo-random number
generator.
- Since we require a min Boost version of 1.67 as of recently (for
Beast), we also remove the conditional checks that existed for us
to know whether Boost.Process is available or not. We can
always assume it is available now.
- Remove runtime checks for minimum Boost and OpenSSL versions
since they are checked at CMake configure time.
This changes the rules for payments in two ways:
1) It sets the maximum number of offers any book step can consume from
2000 to 1000.
2) When a strand contains a step that consumes too many offers,
currently the liquidity is not used at all and the strand will
be considered dry. This changes things so the liquidity is used,
however the strand will still be considered dry.
* mFeeDue is only used in one place by one derived class, so
only compute it as a local in that function.
* The baseFee needs to be calculated outside of the Transactor class
because, it can change during transaction processing, and the function
is static, so we need to be sure to call the right version
* Rename Transactor::calculateFee to minimumFee
Fixes: RIPD-1574
Alias beast address classes to the asio equivalents. Adjust users of
address classes accordingly. Fix resolver class so that it can support
ipv6 addresses. Make unit tests use ipv6 localhost network. Extend
endpoint peer message to support string endpoint
representations while also supporting the existing fields (both are
optional/repeated types). Expand test for Livecache and Endpoint.
Workaround some false positive ipaddr tests on windows (asio bug?)
Replaced usage of address::from_string(deprecated) with free function
make_address. Identified a remaining use of v4 address type and
replaced with the more appropriate IPEndpoint type (rpc_ip cmdline
option). Add CLI flag for using ipv4 with unit tests.
Release Notes
-------------
The optional rpc_port command line flag is deprecated. The rpc_ip
parameter now works as documented and accepts ip and port combined.
Under some corner cases, the base64 decoder would not allocate
enough memory, which could result in spurious errors.
Acknowledgements:
Ripple thanks Guido Vranken for originally noticing this issue
and responsibly disclosing it to Ripple.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
Fixes: RIPD-1575. Fix argument passing to runner. Allow multiple unit
test selectors to be passed via --unittest argument. Add optional
integer priority value to test suite list. Fix several failing manual
tests. Update CLI usage message to make it clearer.
The lsfDepositAuth flag limits the AccountIDs that can deposit into
the account that has the flag set. The original design only
allowed deposits to complete if the account with the flag set also
signed the transaction that caused the deposit.
The DepositPreauth ledger type allows an account with the
lsfDepositAuth flag set to preauthorize additional accounts.
This preauthorization allows them to sign deposits as well. An
account can add DepositPreauth objects to the ledger (and remove
them as well) using the DepositPreauth transaction.
Add a new invariant checker that verifies that we never charge a
fee higher than specified in the transaction; we will charge less
in some corner cases where the transacting account cannot afford
the fee.
Detect more anomalous conditions, and improve the logged error
messages.
Clarify the code flow associated with invoking the invariant checker
from `Transactor`, add extra comments and improve naming to make the
code self-documenting.
The six different ranges of TER codes are broken up into six
different enumerations. A template class allows subsets of
these enumerations to be aggregated. This technique allows
verification at compile time that no TEC codes are returned
before the signature is checked.
Conversion between TER instance and integer is provided by
named functions. This makes accidental conversion almost
impossible and makes type abuse easier to spot in the code
base.
* UptimeClock is a chrono-compatible seconds-precision clock.
* Like UptimeTimer, its purpose is to make it possible for clients
to query the uptime thousands of times per second without a
significant performance hit.
* UptimeClock decouples itself from LoadManager by managing its
own once-per-second update loop.
* Clients now traffic in chrono time_points and durations instead
of int.
Some classes had virtual methods, but were missing a virtual
destructor.
Technically, every unit test that inherits from the Beast test suite
would get flagged by `-Wnon-virtual-dtor` but I did not think it would
be a great idea to go sprinkle a virtual destructor for every Ripple
test suite.
* Tally and duration counters for Job Queue tasks and RPC calls
optionally rendered by server_info and server_state, and
optionally printed to a distinct log file.
- Tally each Job Queue task as it is queued, starts, and
finishes running. Track total duration queued and running.
- Tally each RPC call as it starts and either finishes
successfully or throws an exception. Track total running
duration for each.
* Track currently executing Job Queue tasks and RPC methods
along with durations.
* Json-formatted performance log file written by a dedicated
thread, for above-described data.
* New optional parameter, "counters", for server_info and
server_state. If set, render Job Queue and RPC call counters
as well as currently executing tasks.
* New configuration section, "[perf]", to optionally control
performance logging to a file.
* Support optional sub-second periods when rendering human-readable
time points.
Each validator will generate a random cookie on startup that it will
include in each of its validations. This will allow validators to detect
when more than one validator is accidentally operating with the same
validation keys.
* The compiler can provide many non-explicit constructors for
aggregate types. This is sometimes desired, but it can
happen accidentally, resulting in run-time errors.
* This commit assures that no types are aggregates unless existing
code is using aggregate initialization.
We had several hash functions implemented, including SipHash,
SpookyHash and FNV1a.
Default to using xxhash and remove the code for the remaining
hash functions.
For the functions defined in <ctype.h> the C standard requires
that the value of the int argument be in the range of an
unsigned char, or be EOF. Violation of this requirement
results in undefined behavior.
When creating an escrow, if the `CancelAfter` time is specified but
the `FinishAfter` is not, the resulting escrow can be immediately
completed using `EscrowFinish`. While this behavior is documented,
it is unintuitive and can be confusing for users.
This commit introduces a new fix amendment (fix1571) which prevents
the creation of new Escrow entries that can be finished immediately
and without any requirements.
Once the amendment is activated, creating a new Escrow will require
specifying the `FinishAfter` time explicitly or requires that a
cryptocondition be specified.
Constructing deeply nested objects could allow an attacker to
cause a server to overflow its available stack.
We now enforce a 10-deep nesting limit, and signal an error
if we encounter objects that are nested deeper.
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing this
issues.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled codebase and urge reviewers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit
https://ripple.com/bug-bounty
Constructing deeply nested objects could allow an attacker to
cause a server to overflow its available stack.
We now enforce a 10-deep nesting limit, and signal an error
if we encounter objects that are nested deeper.
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing this
issues.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled codebase and urge reviewers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit
https://ripple.com/bug-bounty
* RIPD-1617, RIPD-1619, RIPD-1621:
Verify serialized public keys more strictly before
using them.
* RIPD-1618:
* Simplify the base58 decoder logic.
* Reduce the complexity of the base58 encoder and
eliminate a potential out-of-bounds memory access.
* Improve type safety by using an `enum class` to
enforce strict type checking for token types.
* RIPD-1616:
Avoid calling `memcpy` with a null pointer even if the
size is specified as zero, since it results in undefined
behavior.
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these
issues.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
The function is non-virtual and hides the virtual function specified in
the base class.
Falling back to the virtual function in the base class is the correct
solution.
Fixes: RIPD-1601
Fix intermittent failure in server stream sub/unsub test.
Root cause is LoadManager thread *sometimes* running and causing a
fee change event which got published before our test could
unsubscribe. Fixed by explicitly stopping the LoadManager for this test.
These changes use the hash of the consensus transaction set when
characterizing the mismatch between a locally built ledger and fully
validated network ledger. This allows detection of non-determinism in
transaction process, in which consensus succeeded, but a node somehow
generated a different subsequent ledger.
When a test suite starts and ends, it informs the parent process. If the parent
has received a start message without a matching end message it reports that a
child may have crashed in that suite.
Fixes: RIPD-1521
Switch to pure doxygen HTML for developer docs. Remove docca/boostbook
system. Convert consensus document to markdown. Add existing markdown
files to doxygen input set. Fix some image paths and scale images for
use with MD links. Rename/cleanup some files for consistency.
Add pipeline logic for windows slaves. Add ninja and parallel test run
option. Add make doc target build in build-and-test.sh. Cleanup README
files. Add nounity windows build. Add link to jenkins summary table.
Add rippled_classic build (win). Improve formatting of summary table.
* Rename isArray to isArrayOrNull
* Rename isObject to isObjectOrNull
* Introduce isArray and isObject
* Change as many uses of isArrayorNull to isArray as possible
* Change as many uses of isObjectorNull to isObject as possible
* Reject null JSON arrays for subscribe and unsubscribe
* Remove all builds except cmake gcc & clang debug.
* Time some dependency and build operations, using a custom format to avoid
interfering with other timers.
* Use Travis's "trusty" infrastructure.
* Install more dependencies via apt.
* Update boost version to 1.65.1.
* Do not run unit tests under gdb - several security features prevent
it from running correctly.
* Run test job with two processes.
* Replace Ubuntu scons dependency with cmake
* Remove dependency installation support for Ubuntu 12.04 Precise Pangolin.
It is EOL'd as of 2017-04-28. http://releases.ubuntu.com/12.04/
The assert in LedgerHistory.cpp was incorrect since
getLedgerByHash() can return a null object. The assert has
been seen to fire inappropriately during online_delete.
The assert in LedgerMaster.cpp was simply misguided. If a
null hash were returned by getLedgerHashForHistory(), then we
should not forward that hash to peers. The assert only fires
in a debug build, so a release build would forward the request
for a null hash to peers.
These changes augment the Validations class with a LedgerTrie to better
track the history of support for validated ledgers. This improves the
selection of the preferred working ledger for consensus. The Validations
class now tracks both full and partial validations. Partial validations
are only used to determine the working ledger; full validations are
required for any quorum related function. Validators are also now
explicitly restricted to sending validations with increasing ledger
sequence number.
Per issue #2354, when the log level of a server was configured at
"trace", sensitive keying meterial generated by the `wallet_propose`
command could be written to the server's log file, if one was
configured.
This commit improves the log scrubbing code to account for the
sensitive information generated by a `wallet_propose`.
** Important security consideration **
We still caution everyone *against* executing this command on a
server that they do not control: a malicious server operator could
intercept the generated keypair, or operate a modified server that
returns keypairs that are not securely generated.
* This change passes detailed error messages from the JSON parser
on the server side, back to the client for inclusion into the
reply's error message.
* Errors originating from the server's inability to parse are
reclassified from rpcINTERNAL to rpcINVALID_PARAMS.
Introduce a new ledger type: ltCHECK
Introduce three new transactions that operate on checks:
- "CheckCreate" which adds the check entry to the ledger. The
check is a promise from the source of the check that the
destination of the check may cash the check and receive up to
the SendMax specified on the check. The check may have an
expiration, after which the check may no longer be cashed.
- "CheckCash" is a request by the destination of the check to
transfer a requested amount of funds, up to the check's SendMax,
from the source to the destination. The destination may receive
less than the SendMax due to transfer fees.
When cashing a check, the destination specifies the smallest
amount of funds that will be acceptable. If the transfer
completes and delivers the requested amount, then the check is
considered cashed and removed from the ledger. If enough funds
cannot be delivered, then the transaction fails and the check
remains in the ledger.
Attempting to cash the check after its expiration will fail.
- "CheckCancel" removes the check from the ledger without
transferring funds. Either the check's source or destination
can cancel the check at any time. After a check has expired,
any account can cancel the check.
Facilities related to checks are on the "Checks" amendment.
Do not process a transaction received from a peer if it has
been processed within the past ten seconds.
Increase the number of transaction handlers that can be in
flight in the job queue and decrease the relative cost for
peers to share transaction and ledger data.
Additionally, make better use of resources by adjusting the
number of threads we initialize, by reverting commit
68b8ffdb63.
Performance counter modifications:
* Create and display counters to track:
1) Pending transaction limit overruns.
2) Total peer disconnections.
3) Peers disconnections due to resource consumption.
Avoid a potential double-free in Json library.
In some cases this script did not detect properly that the subprocess is already terminated. The "append" loop afterwards caused memory to fill up and the script to fail eventually.
The DepositAuth feature allows an account to require that
it signs for any funds that are deposited to the account.
For the time being this limits the account to accepting
only XRP, although there are plans to allow IOU payments
in the future.
The lsfDepositAuth protections are not extended to offers.
If an account creates an offer it is in effect saying, “I
will accept funds from anyone who takes this offer.”
Therefore, the typical user of the lsfDepositAuth flag
will choose never to create any offers. But they can if
they so choose.
The DepositAuth feature leaves a small gap in its
protections. An XRP payment is allowed to a destination
account with the lsfDepositAuth flag set if:
- The Destination XRP balance is less than or equal to
the base reserve and
- The value of the XRP Payment is less than or equal to
the base reserve.
This exception is intended to make it impossible for an
account to wedge itself by spending all of its XRP on fees
and leave itself unable to pay the fee to get more XRP.
This commit
- adds featureDepositAuth,
- adds the lsfDepositAuth flag,
- adds support for lsfDepositAuth in SetAccount.cpp
- adds support in Payment.cpp for rejecting payments that
don't meet the lsfDepositAuth requirements,
- adds unit tests for Payment transactions to an an account
with lsfDepositAuth set.
- adds Escrow and PayChan support for lsfDepositAuth along
with as unit tests.
* Null json values can be objects or arrays.
* json arrays are now interpreted as batch commands.
* json objects are single commands.
* null jsons are ambiguous as to whether they are single or batch
commands and should be avoided.
In connection with Ripple's decentralization strategy, the example
validators.txt file is being updated, and the [validators] field,
which contained a static list of trusted validators, is being
replaced with two new fields:
- The [validator_list_keys] field which contains the keys that can be
used to sign its recommended validator list.
- The [validator_list_sites] field which specifies the URLs where
dynamic validator list can be found.
The initial configuration links to Ripple's default validator list,
which is located at https://vl.ripple.com.
To read more about Ripple's decentralization strategy, please visit
the Ripple Dev Blog:
https://ripple.com/dev-blog/decentralization-strategy-update/
Profiling and research indicates that the SQLite query planner executed
our existing SQL queries sub-optimally by not using the index efficiently.
Restructuring the SQL query works around this issue and allows queries
to be executed efficiently and without unnecessary delay.
c10863b98 fix : asan error in dctx, due to increased maximum frame header size, reported by Craig Young
1e92bb0af Merge pull request #382 from lz4/installVars
edb434365 updated lz4frame manual
a1f3a27e9 Merge branch 'dev' into installVars and fixed conflicts
af9d72b7f Merge pull request #383 from lz4/blockChecksum
8593ba883 Merge pull request #380 from lz4/dictID
01cdbfb5f lz4c legacy commands are now enabled at runtime based on link/binary name "lz4c"
731cff120 fix minor markdown display issues
aea7d521a better respect GNU standard Makefile conventions
930a69211 cli : restored command -BX to enable block checksum (#322)
77f99d292 restored block checksum capability at lz4frame API level
228e26ee6 Merge pull request #378 from deymo/dev
757497ae3 implemented lz4frame decompression API
4531637ec support dictionary compression with independent blocks
8d597d62d fixed gcc prototype warning
d8aafe2c5 dictionary compression correctly uses compression level
ca2fb166a fixed C++ conversion warnings
bf8daa2fd fixed uninitialization error in lz4frame
31f2cdf4d implemented dictionary compression in lz4frame
1d1737aaf fixed frameCompress example
a82dadfba added dictID inside LZ4F_frameInfo_t
e98a52857 updated Frame specification
76ef6d0ab Allow to predefine FORCE_INLINE macro.
9a967030d Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
c5a015bc3 Merge pull request #376 from ido/patch-1
4891f909e Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
dba4d70a1 updated man page
8aeaf6bb3 Fix typos preventing installation of static lib.
ac5ed8e4b Merge pull request #374 from ferdnyc/patch-2
63d500568 Generate updated man page from Markdown source
ef0655510 Merge pull request #373 from ferdnyc/patch-2
872f767fe Fix formatting of concatenation example
6ad3a983d fix#369
2ef4afeeb Merge pull request #371 from jku/tests-LIBDIR
e14b4c5a3 tests/Makefile: don't use LIBDIR as variable
1525fd1f5 Merge pull request #367 from Chocobo1/fallthrough
7982a905e Merge pull request #368 from Chocobo1/readme
9d453a188 Update README.md
a4ec7fa7b Fix gcc7 Wimplicit-fallthrough warnings
89b9f026c made level 10 a bit faster
a6fd0f9d0 -g compilation flag not by default for lz4 cli
bfc180660 clarified lz4frame api comment (#350)
d18084c65 report where decompression ends (#313)
139a387d1 updated NEWS
0beaa356f fixed minor scan-build warning
775e63ee0 refactored simple_buffer.c example (#363)
7e15e240a added a paragraph on overlap matches
03d8586fc cli accept block sizes with KB / MB prefixes
8c355d17f Merge pull request #360 from Chocobo1/md
1c9d4091d [Doc] Fix markdown
6614be7cc Merge pull request #359 from PierreNav/dev
4e359f5c3 Add DLL files to the INSTALL target
5c97cdfa0 Merge pull request #352 from lz4/resetDCtx
2600a154b fix (minor) g++ compatibility for frametest
e60cbb5ca added test for LZ4F_resetDecompressionContext()
2012e4de9 fixed c_standards tests
a90212761 bumped version number to 1.8.0
b8575f2d2 updated Makefile
fe932c452 expose LZ4F_resetDecompressionContext()
1efa48831 minor readability changes
a8dd86d93 changed macro HEAPMODE into LZ4_HEAPMODE
11bfedb6c Merge pull request #349 from svpv/master
6cf1f7d75 lz4hc.c: clamp compression levels > 12
a41df535f lz4cli.c: fix a comment: LZ4HC_DEFAULT_CLEVEL -> LZ4HC_CLEVEL_MAX
f3460fc14 liz4hc.h: fix a comment: LZ4HC_MAX_CLEVEL -> LZ4HC_CLEVEL_MAX
37ef330cd Merge pull request #348 from terrelln/deprecate
0b3e80704 [LZ4F] Allow users to disable LZ4F_DEPRECATE
31e9ed612 Merge pull request #347 from lz4/negativeCLevels
e2c9b1912 lz4frame : Added negative compression levels
7eecd32c0 ensure lz4f_cctx internal buffer size remain valid in case of malloc error
e169edac0 lz4frame : control lz4 context creation success
e2827775e make __packed memory access default for gcc
3d4ee35da Merge pull request #344 from lz4/LZ4F_getFrameInfo
6226d52ea fixed minor Visual warning
b88df6b1b Improved comments on LZ4F_getFrameInfo()
fc31257ab added LZ4F_resetDecompressionContext()
f0a7651fc Safer LZ4_getFrameInfo()
ab547a0ef Merge pull request #342 from iburinoc/isatty
9b1f00056 Fix inline compile errors
afde27acf Fix IS_CONSOLE returning 1 for NUL on windows
84246b974 Merge pull request #341 from iburinoc/exematch
4567d0def Merge pull request #340 from lz4/optlz4opt
e9c3b14f2 Ignore extensions in exe name matching
38362c1d6 Merge pull request #338 from ldv-alt/dev
60b6d2907 improved level 10 speed for degenerated cases
8f8fc52fc Merge branch 'dev' into optlz4opt
0863931a1 fixed API comment for LZ4F_createCompressionContext()
dab3590fc Export deprecated symbols declared in lz4hc.h
13f5cb2d5 Merge pull request #337 from ldv-alt/dev
b1daffc4e cli: add GNU separator -- specifying that all following arguments are files
68044acfe Merge branch 'optlz4opt' of github.com:Cyan4973/lz4 into optlz4opt
b8bc70022 minor refactor
cea8f60df slight btopt speed improvement
f513020a6 slight btopt speed improvement
8bd32a17b made SET_PRICE macro more usable
118b47f3d improved lz4opt speed (~4%)
eff6166eb minor price function optimization
0d073d4d2 added `extern C` for lz4.h static section
aae447fff LZ4_compress_HC_continue_destSize() works as intended up to level 10
cd35f0d98 LZ4_compress_HC_destSize() uses LZ4HC_compress_generic() code path
6b2a1b360 updated lz4 version number
430b7d32b created LZ4_HC_STATIC_LINKING_ONLY section
9144520dc fixed Visual compilation error
a33bf89c6 fix#332 : do not modify /dev/null permissions
e945a27f9 fix#333 : expose obsolete decoding functions
e1b2b370e added Oleg @remittor contribution
b9132bedc Merge pull request #331 from ldv-alt/dev
50fe87f13 Merge pull request #329 from remittor/dev-hc2
883ebdcee Export only those symbols that are part of public API
25b243588 Export deprecated symbols
7aeecbff7 Explicitly set visibility of public API functions when gcc is used
93b3b08ef Merge pull request #330 from inikep/dev
cb1671955 bench.c: respect LZ4_MAX_INPUT_SIZE limit
517d446d7 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
36842ebb1 lz4hc: Fix LZ4HC_compress_hashChain for backward compatibility
baa155088 lz4hc: Fix LZ4HC_compress_hashChain for full support destSize variant
66b26a389 tests: fuzzer: Add test for LZ4_compressHC_destSize
f007153e3 lz4hc: Add LZ4_compressHC_destSize and LZ4_compress_HC_continue_destSize
45b592b7e lz4hc: Cleanup function LZ4HC_encodeSequence
534f8fa5d lz4hc: Cleanup function LZ4HC_compress_hashChain
9141ada8d Merge pull request #328 from ldv-alt/dev
0dfb0b9da Fix test-lz4-basic
97df1c978 updated LICENSE
5bfdff96b Merge pull request #320 from inikep/dev
d3eaf7883 remove MOREFLAGS from test-lz4c32
5bb6ba794 Merge pull request #319 from inikep/dev
ca2c5fd3c CFLAGS=-I/usr/include/x86_64-linux-gnu
ace22424d circle.yml: fixed lz4c32 target (2)
0d77c85d9 circle.yml: fixed lz4c32 target
24007bb4e Merge pull request #318 from esiegerman/eks/mklib
4ee9bd5c3 improved Circle CI tests
c0a8d0ad8 added circle.yml
3ec1591db Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
68a358769 Avoid fseek()'s 2GiB barrier with MinGW
40ad1e85d upgraded util.h
eb51b2b8d Explicitly create $(DESTDIR)$(LIBDIR)/ at install time
69dbafc1c Merge pull request #317 from esiegerman/eks/no-andand-oror
b89cac7b2 Don't use "foo && false || true"
7e57c398c Merge pull request #316 from inikep/dev
f4b9c1d03 Merge pull request #315 from bebehei/patch-1
c139eb40b added OpenBSD NetBSD SunOS to list of supported `make install` OSes
5e1a925be update repolink in makefile
272699a4e Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
c7b14967a updated platform.h
4f261b828 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
04e5eaf9b added DragonFly to list of supported `make install` OS
04ec09269 updated comments on block sizes
7ed257191 Merge pull request #312 from mgorny/cmake-pc-version
96e150e7d [cmake] Fix substituting version in liblz4.pc
67abd75e6 Merge pull request #307 from inikep/dev
8e1fd97d1 lz4frame.h: added Introduction
d56ee32b3 updated gen-lz4-manual.sh
f54c7e0e7 added lz4frame_manual.html
9071df0fa lz4frame.h: prepared to generate manual
6a5633e99 lz4 manual updated to v1.7.5
f9f48f8ed lz4.h: improved manual generation
128244371 improved gen_manual
489d1618c added "This Makefile is validated for"
339e79320 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
ca083dd4d Merge pull request #304 from Chocobo1/cppcheck
db6f733ec Use logical or instead of bitwise or
44f95e92e Fix printf specifier
9683a1ae8 LZ4_MEMORY_USAGE can be modified from compilation command line
16d69c8d0 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
05e27ade6 updated NEWS
c18a81c02 Merge pull request #296 from mgorny/cmake-fixes
d7969e49a cmake: Install manpages
1380c33b7 cmake: Install lz4cat and unlz4 symlinks
28db4acc9 cmake: Support building shared & static libs simultaneously
9e867db90 cmake: Fix SOVERSION to match Makefiles
cb9599449 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
8c4de60d0 Merge pull request #295 from inikep/opt-parser
d2b51c22d fuzzer: tests more compression levels
312d88249 removed nextToUpdateBT
3d5bb3897 clean logging
1c80b9af4 LZ4HC_getSearchNum
581da2bf8 Merge pull request #293 from inikep/dev
98f9d6c72 improved logging
12aae846b fixed table update in LZ4_loadDictHC
c8b31263c added FUZ_CLEVEL_DEFAULT
7a73c5c1f changed default PREFIX and MANDIR
d57ff6456 changed default PREFIX and MANDIR
a3d61cf3d improved test-lz4-opt-parser
20381a2fc fixed -BD compression
f4575f4f1 added test-lz4-opt-parser
e6536faf3 lib\Makefile: fixed INSTALL_DATA
efaa9b7bc Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
b22e2bed5 BSD: improved "make install"
ea51ad0ce Solaris: working "make install"
19c0f21b0 updated Makefile : lz4 no longer recompiled when already up-to-date
9b4b081c1 "make test" is now compatible with Solaris
b41f9bb13 Merge pull request #292 from inikep/dev
7eb16d97d updated NEWS
52cac9a97 updated a few macros names
fea95c156 use FindFirstFileA and FindNextFileA on Windows
e6af952f8 improved comments
581245063 use fseeko for 32-bit MacOS
f084b6538 test Large File support for Mac OS-X in 32-bits mode
973bc7974 util.h and platform.h based on zstd
70526a11e fixed lib/clean
bc0839c5d minor fix for travis-install test
7cf0bb97b LZ4F_compressBound(0) provides upper bound for LZ4F_flush() and LZ4F_compressEnd() [#290, suggested by @vtermanis]
385cb4f53 minor update Makefile
4f61505cc datagen.vcxproj: CharacterSet is MultiByte
253ac12cb VS projects: CharacterSet changed to MultiByte
287f7d3d6 added IncludePath
2fd7eb554 fixed Visual Studio compilation
3c6ce9c66 fixed gcc 4.4 support
64cbc4e1d improved MinGW support
9546ba62d executables use basic types from util.h
452cc970d reordered definitons in platform.h
090cb887a improved formatting (2)
e1332241a improved formatting
5f0cc8e2d fixed gcc warnings (2)
2efedaf7d fixed gcc warnings
6adf05d1e improved platform.h
2cfef1395 executables use platform.h
bb69cdf25 updated util.h
45b14ab8e added platfom.h
913b98fd6 fix `make` concurrency build (#277)
fec57e49e Merge pull request #287 from inikep/dev
4f050b68d tree update dependent from compression level
838ed341f fullUpdate as a parameter
b3ecc3e3e last match starts at least 12 bytes before end of block
d694bf911 faster opt compression
e7c42d757 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
a22e71d4a full binary tree update
626552671 updated NEWS
de93e9e5d fix#285 : lz4cat doesn't work with stdin (reported by @beiDei8z)
fb6c98c85 slightly improved lz4opt.h
20183ad7a Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
0280cf40d fixed path-1, as suggested by @t-mat
7e2d4c4c0 fixed test-lz4c32
750a50cf6 fix#284, reported by @beiDei8z : added test cases for path/to/lz4cat
01ffed25a fix : invocation of path/to/unlz4 (#284), reported by @beiDei8z
8c2f23c13 fixed LZ4_STREAMHCSIZE
76228f064 LZ4HC_DICTIONARY_LOGSIZE 17
8bfa42e61 improved ratio
128903824 15-bit LZ4HC_HASH_LOG
410ba53db improved LZ4HC_BinTree_GetAllMatches
1833be1cf fixed gcc warnings
f2ebf37bf slightly improved ratio
77b051ed7 improve code formatting
e3fee9474 LZ4HC_MAX_CLEVEL = 12
757ef1d5f removed LZ4HC_GetAllMatches
cfe6a24b0 updated LZ4HC_get_price
2113ead17 17-bit LZ4HC_MAXD
c1ef7a177 introduced LZ4HC_compress_hashChain
3f430daf7 added lz4opt.h
5ae0e789f Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
5eb547a60 fix#283 : implement LZ4_versionString().
9a6f7dfc5 updated NEWS
a9a8dbf7c fixed minor display error when using -r
6b90cfbea Updated man page
009f06e01 minor man page update
b3bc054d0 Merge pull request #279 from t-mat/md-to-man
544290d58 Add "man" target to Makefile
018ddf799 numbering
605512a7b updated NEWS
f4c74e6fa library release build compatible with environment variable
edc132b69 lz4 release build compatible with PIE
7819d5878 new test case with fPIE
de0cf5de6 fixed LZ4IO_compressMultipleFilenames
65a542579 updated README.md
2cbd0233b Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
cc21115cb minor warning fix
8875e7dbb fix minor analyzer warning
8b233b228 added a few dates
ba7a85b46 fixed minor analyzer warning
3bde02a05 added gcc 4.4 test
2fe3aa985 added examples to make all
742f2b683 added doc in code comments
2dbcebfda changes travis test order
b4db9d7e3 Add C++14 standard [[deprecated]] attribute
a38cbf9f6 fixed : shadow global variable on gcc 4.4 (minor)
19df3029e fixed#272 (compilation fails on gcc 4.4), reported by @totaam
519932afa attempt -mx32 tests on Travis CI
a91983daa highly improved speed on -mx32 mode Now -mx32 is fastest mode on x64 CPU
6cabcd151 removed usan32 tests. Let's summarize : gcc-5 and gcc-6 on Travis do not support UB sanitizer. And clang on Travis is not able to generate -m32 code. Great. It makes the combination impossible to test on Travis ... :(
7a22904a7 moved usan32 tests to gcc-6
ffc8c18c1 usan32 relies on default compiler (by default)
a4f4bf21e fixed gcc-5 conf
71814a26f switch usan32 tests to gcc-5
8e1ea59ad attempt to fix sanitize32 package dependency
8f536a43b added sanitize32 tests to Travis
3f6f57768 fixed strict aliasing warnings in fasttest
c0e55ec05 fixed versionsTest
7d5354847 added v1.x.y to versions tests
279344a17 fixed Travis test cases
1abecbc33 fix 32-bits mode. Large File support for Mac OS-X in 32-bits mode Fixed potential undefined behavior Changed makefile for 32-bits mode
7fde7438d Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
a22ca9378 updated README for v1.7.3
e9bec0fdc Merge pull request #271 from inikep/dev
b1519f861 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
2dee7bdaf include NEWS in lz4_x64.zip and lz4_x86.zip
632ab3b8d updated programs\README.md
8b48d5e44 updated example\README.md
11f6217cc created dll\example directory
873206e70 appveyor.yml: create only lz4_x64.zip and lz4_x86.zip
7b6a9d863 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
2e10aab20 MinGW doesn't require the import library at all
7cecc7b08 fixed a limit case scenario
00fa1b9c0 fixed double ;; for Visual
929badd61 fixed sign conversion warning
d613e2494 restored .travis.yml
d2be69b14 fixed minor coverity warnings
8c32a12f1 .travis.yml for coverity
2eddc8ea2 removed .travis.yml, for better rebase
2906c373c fixed appveyor badges
0d6dc079f update appveyor tokens
b44790cf9 Merge pull request #270 from inikep/dev
4235e5e25 updated visual\README.md
257b65a1d ToolsVersion="4.0"
200746427 removed _x64 and _x86
e583dd056 ImageHasSafeExceptionHandlers=false
eb160e1a2 appveyor.yml: added liblz4_x86.zip
45b8623af updated lib\dll\README.md
5625cc21e appveyor.yml: generate DLL package
c093e256d added files used by DLL package
be471c69d added fullbench-dll.sln that works with gcc/MinGW
f71340ce1 DLL dependencies moved to lib/dll/
f3f59bef4 updated appveyor.yml
b0e994cc5 upload executables to GitHub
d84a80e66 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
923cf4447 bump version number
4c5c711b7 fixed minor conversion warning
f094f5314 silence a minor msan warning
0fe5c187a fixed minor msan warning
201149f8e fixed conversion warnings
fa158b411 fixed nameless union warnings
84af32d49 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
5a5ebeec1 clang and g++ tests restricted to native only on Travis CI Ubuntu v12 environment
569897ddd Merge pull request #269 from inikep/dev
1b24cc115 fixed conversion warnings
9ad7508db Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
ecc55d19b fixed uintptr_t on pre-c99 compilers
3d3ccc154 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
d5c9b39dd minor code refactoring
4bd9db1dd Merge pull request #268 from nemequ/dev
f42130ee4 cli : add --help command
84b97f90b fuzzer code refactoring fix g++ conversion warning
151021a3a cmake: add public domain dedication, and a note about the maintainer
b6bcb84ea cmake: make PIC optional when generating a static library
94542d845 made lz4hc explicitly dependent on lz4
bb97e8a6f fixed __GNUC__ macro
cb2230826 fixed fuzzer error
3580d9698 enabled deprecation warnings on remaining obsolete functions
874f3e095 update code comments
e528a82f0 Merge pull request #267 from terrelln/strict-aliasing
3d456bdf2 fixed conversion warnings
bf6dda69e updated frametest
85aeb0e4b Expose internal types to remove strict aliasing
dbfdd5131 Merge pull request #266 from terrelln/frame-ex
7aad356ff Add frame decompression example
ec6fb477b added visual\README.md
9d075580d Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
4f9db1383 updated xxhash lib
c72d2f5b1 minor typo
7d2e4b3cf Merge pull request #265 from terrelln/rm-zstd
5b37837e6 Fix license and remove references to zstd
1f246a989 Fixed#178 fullbench on small input
5e13a6ec4 improved man on compression level
f9c9bb364 Merge pull request #263 from terrelln/dict-ex
cd4462629 Merge pull request #264 from inikep/dev
52251d970 Using MinGW+MSYS to create DLL
72a4a84df udated README.md
2ecce6f15 lz4.h: added Introduction section
93e2477ca added lz4_manual.html
a340b734b added contrib/gen_manual
2278d1f02 custom block size
d57d3e4b4 better MSYS/MinGW support
74f127787 NB_LOOPS ?= -i1
cf8ca581d reordered Travis CI tests
767a23160 changed FUZZER_TIME
6a160b72e Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
924f00c86 fix empty body
94917c9a0 Add dictionary random access example
f34808e4a fixed some static analyzer warning
46f74d79b fixed sparse on stdout
fe48cef50 fixed multiples files on lz4cat (#184, by @libor-m)
1b1680594 fix unused variable warning
1b95d60c6 fixed synthetic benchmark mode slightly optimized test mode
acc48a3f8 fixed lz4cat changed lz4-test order
0c5e2e4d1 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
fafba3973 updated NEWS
2987eb417 Merge pull request #262 from inikep/dev
063b4d92e fixed make install
773ae50b4 DLL exports only functions defined in liblz4.def
ca80a7c8f create DLL with MinGW/MSYS
67fd5dffd updated appveyor.yml
c794829a2 update .gitignore
9e7fe1062 removed <PlatformToolset>
dd14dca15 LZ4_DLL_IMPORT
325babddf added lz4.rc
ef2fdfe3d rename liblz4-dll to lz4-dll
7df658ec1 removed LZ4_DLL_EXPORT=1 (2)
4a6e7327c removed LZ4_DLL_EXPORT=1
bde0d340b added fullbench-dll.vcxproj
da48dbb57 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
fe5148417 introduced LZ4_COMPILED_AS_DLL
d5903c8e5 Merge pull request #261 from terrelln/clean-fasttest
d594d9d87 Remove fasttest when cleaning
bd88e4007 Merge pull request #260 from nemequ/dev
661c48040 cmake: significant changes to accomodate a new "bundled" mode.
ce2202690 Merge pull request #259 from inikep/dev
96ea2e363 updated IntDir
39157c9f3 added lz4-dll project
ec63747de VS project: added IntDir
c6fd78f4f VS project: added setargv.obj
1d22fae1e added test-lz4-speed.py
85f861fc4 bench.c: support notification level
2c788b372 joined arm and aarch64 tests
0a22bf7d6 err0r(LZ4F_errorCodes code) uses ptrdiff_t
d06ab4f41 test powerpc64
230264913 tests\Makefile: added test-platform
c201327f1 added platformTest
0f73ed572 new Travis tests
90981a35c Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
c7ab95faa slightly improved HC compression ratio
e9089bcbe small compression ratio and speed improvement on small files
00c6370ad Fix#243 : detect and report fread() errors, by @iyokan
f6a7c6b57 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
301c4e94e Merge pull request #258 from inikep/dev2
839085f05 Merge pull request #257 from inikep/dev
9b3dcb325 MOREFLAGS="-Werror"
7fbb163f3 added "make c_standards"
b6c407886 condensed travis.yml
2fbd7a512 updated travis.yml
ef54f9f9d added --rm option
d61bb74f2 updated declaration of LZ4F_freeDecompressionContext
7a98db266 added datagen.c to lz4.vcxproj
4b29af350 TreatWarningAsError=true only for debug
90d4401fe fixed appveyor.yml (3)
434d1230e fixed appveyor.yml (2)
58ead8dd2 fixed appveyor.yml
98be00847 Copy owner, file permissions and modification time
158156729 util.h requires CharacterSet=MultiByte
ef9d1b2d4 fixed --test
112499a73 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
58d7a619a Merge pull request #254 from terrelln/dev
920bf2171 Fix LZ4_decompress_fast_continue() bug
207f478ba minor refactor
8195ba8f7 Updated man page
96565816b restricted deprecation warning to fairly recent gcc versions (>=6.0)
eaad740ac lz4frame obsolete enum should trigger deprecation warnings (on supported compilers)
079d5dd54 removed support for old error code enums from lz4frame_static.h, to free up namespace. note : lz4frame_static.h does not guaranteed API stability. note 2 : the macro to enable old error code enums is still present. Just needs to comment one line to re-enable them.
473acf5aa Merge pull request #252 from inikep/dev
10c5d5eee Merge pull request #253 from terrelln/exit-code
86a24c80f Fix typo in lz4 manpage
f30c56c08 Quiet gcc-4.6.3 narrowing warning
374090c7a Return error if input file does not exist.
fbede33fd fixed Travis tests
8ddaddc2d updated #include in util.h
f878c08b7 better correctness on big-endian 64-bits platforms
26dec498c small compression speed improvement on 64-bits systems
e8a6067e8 updated lz4cli.c
8b8c726a5 bench.c based on zstd
e06fcd903 redefined VOID for Windows/MSYS
e7648f4fc added -r option
6ebf8859e Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
4c496faba Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
6d6a3e0fb Merge pull request #250 from terrelln/test-mode
4c6610709 minor code refactoring
136caa552 Rename passThrough to testMode
eda8c4eb1 fix#198 : no longer requires to restart streaming decompression from where it stopped. It permits relocating remaining data into another memory space. Still, the same content must be presented.
04d728b87 Fix tests for test mode
4791cda04 fixed strict warnings
258a5e7fa updated comments
3dcafd35d Fix test mode and write to null
6f68a1692 fixed travis badges
99d11120b Merge pull request #249 from inikep/dev
46b49af37 fixed clang conversion warning
c66108ceb check fread() operation results (#243)
8847faffb updated programs/README.md
f38adde9e versionsTests moved to tests/
74ba377d7 updated .travis.yml
9f15a4c6e updated Visual Studio projects
84cedb463 updated links to LZ4 repository
e624978ea test programs moved to tests/
88cef262e documentation moved to doc/
b768da8ad Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
59adfd82d fix for ARM platform
67500142c Clarified license (#115, #244)
f17302769 moved cmake and debian directories to contrib (#245)
d05ca2ff6 fixed#247, reported by Felix Bolte
7e112cfa5 fixes static compilation under Windows (#246)
e64a92692 Merge pull request #242 from vapier/dev
92408d338 README: fix minor typo
52cdb4a5f Merge pull request #241 from vapier/dev
c5cb4aa52 Merge pull request #238 from kou/add-lz4lib-api
544abaced Merge pull request #240 from kou/add-missin-ext
43b631223 add a flag to disable static libs
829941e5a Add missing $(EXT) to lz4 program name
59771f41c Use different prefix for each file to avoid conflict
1ca408105 Merge pull request #239 from kou/use-lower-case-for-header-name
012c66536 Add LZ4LIB_API
46a7b94e8 Use lower case for header file name
3b9439ccd updated uninstall
8eae1320c Merge pull request #236 from inikep/dev
33993542a removed test artefacts
3eb143fff fix bug #232 at https://github.com/Cyan4973/lz4/issues/232
c8a78dba1 datagen depends on lz4.h for version string
53515b05e Merge pull request #235 from inikep/dev
92ee3e0ea fixed LZ4 repository links
39be6f16a versionTests require 32-bit gcc libraries
122fdd13a versionTests require python3
6adf7392a updated .travis.yml
8ac619afe Merge pull request #233 from inikep/dev
58a1616d7 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev
c77fdb44f disabled <RunCodeAnalysis> for Win32
28f98647e lz4 version source from lz4.h
dd51ca630 frametest depends on standard C time.h only
f327a8bdd fullbench depends on standard C time.h only
32a31663c fuzzer depends on standard C time.h only
64f556e61 lz4cli version number from lz4.h
5871585b6 coding style change
e5b0f1ea1 CFLAGS += $(MOREFLAGS)
829ae7587 remove EnablePREfast=true for Win32
547e1943c set OutDir
ab707e8b0 update IncludePath
3a7b4457a restored lz4.vcxproj
81d90609b added appveyor.yml
c200f7e09 renamed VS project
f4111cf04 Merge pull request #231 from inikep/dev
b3ae49c7b added <string.h> for strlen()
2cf32ef0f -b# and -e# options from zstd
c1864f69e fix mingw64 compilation warnings
7f08131f9 Merge pull request #226 from gsauthof/port-make
3fa60044e don't use variable eval syntax in if()
e1e3305db don't link the tools against the shared library
b0d42cb7e remove additional -fPIC switch because
681d5010d use gnu c99 to get both c99 and POSIX
584e1fe92 request POSIX.2001 definitions for fileno()
5bd3eaa48 minor comments
9578ff539 minor refactor
a7672dc4f minor decompression speed gains
182645ee0 lz4 as phony target
0214638eb Makefile : lz4 at root
63d22e7a1 minor comments clarifications
74e6afc5d minor comments clarification
2ac9ecec6 Fixed#157 : LZ4F_getFrameInfo() fails on valid null-content frame
45ea27949 Minor fuzzer code refactoring
de69eca47 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
828332d23 fixed : fuzzer test, reported by @setharnold (#203)
b130fd592 minor refactoring changes
3c295ca4a Merge pull request #222 from kou/fix-build-error-with-mingw
0c63b7cbd Fix build error with MinGW on Linux
7316584ca Merge pull request #219 from BenWiederhake/dev
d3e2d8026 Use https wherever possible
cec38d1b7 Correctly reference self (github, not googlecode)
654003003 Use correct 'repository' hostname
91cce757f Updated xxhash library to v0.6.1
5540f4f93 stronger compilation flags
19e585cc4 modified : some more deprecated functions from lz4hc.h generate warnings
6b1600f41 changed : default benchmark LZ4_decompress_safe()
6923f3e23 Minor refactoring (code style)
410ec8154 Fixed : alignment warning
bc342ab94 minor refactoring
c8a573d32 very minor decompression speed gain
a5508b1ad fixed display result issue when benchmarking multiple files
e27e7316d minor code refactoring
2d7df8b30 Reduced bench dependency to standard C <time.h> Faster tests More accurate results on systems with high resolution clocks
8938e1074 minor compression speed improvement
c77c0c108 minor refactoring
3c0332600 minor refactoring (coding style)
1f47f3f74 Merge pull request #208 from Nimloth/dev
7bfbccb97 Add FreeBSD to install targets
5bb5d6ee1 Merge pull request #204 from irwand/dev
bc70a4b72 fixed non-C99 compilers
10e3eecc5 fixed non-C99 compiler
973baa2c1 Merge branch 'heapmode_alloc_failure_fix' into dev
3e5bb6753 alloc failure fix
8a42ee10a Merge pull request #201 from cernekee/cmake-pc
4bda09fc8 cmake: Install liblz4.pc
8b562a4f3 Merge pull request #197 from Cyan4973/fix-readme-travis
5cdf9a657 Fix wrong link for Travis
220e0b606 Merge pull request #194 from jzhuge/dev
cf6652f9b lz4cli: print library version
60ba8638c Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
219da951d Added zstd
2995a45e5 Merge pull request #185 from jwerner-chromium/dev
dc868cd5b Allow for safe in-place decoding
4fcb2e17f Remove whitespace from ends of lines
ad564c872 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
db53bbb94 updated hyperlink
b84eb3349 Merge pull request #182 from kou/add-missing-ext
b8abffc39 Merge pull request #181 from kou/fix-build-failure-on-mingw
ab3e1d253 Add missing $(EXT)
14b40fb6d Fix build error on MinGW
62bbd102f Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
2785eeedf clarified man page as suggested in #170
631dc8f2b Merge pull request #173 from flandr/liblz4
d67ae8fe4 Merge pull request #172 from psteinb/add_static_cmake
4c9d851eb added static build target for OSX and Linux to cmake
495ea43d9 Merge pull request #162 from KyleJHarper/20151021_examples
7aa004377 Forgot to make clean before commiting. Removing binary.
7f96babc4 Several changes to address a few concerns from Yann. See Google Group LZ4c topic 'Reusing compression/decompression resources' for details.
d1be164e4 Added a note about why char* is used. Also removed a binary that wasn't supposed to be in there.
0d2518567 Finished the two example files. Decided to avoid adding anything to lz4.c/h to expose LZ4_compress_generic().
02be6631b Took out the basics and placed them into basics.c. Added decompression and a wrapper for the generic call. I will likely break this file up into 2 examples before submission.
67b3a2470 Final tests and reporting are done. As expected there isn't much to be gained by jumping the chain. In most of my tests I did see a moderate performance gain when invoking LZ4_compress_generic() directly with normal text. This could very easily be an edge case. Either way it's interesting and worth sharing.
bdd9143e0 Didn't add my file for some reason. Odd.
73dd936b9 Wrote the skeleton of the performance test. Need to finish tests for the rest of the call stack and then my own copy of generic().
2c8013812 heapmode macro option for lz4hc
ff3324b8d Fix : compiler-independent macro to remove deprecation warnings
64f8b037e clarified lz4 license
2df602568 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
7d4575e54 fixed link typo
012ab2f52 Merge pull request #152 from dcolascione/dev
94a0954b1 Remove .dir-locals.el
4f1ff58f8 Tweak error message
975c06b1a Improve error messages for attempted console IO
6f1fe1b70 Add .dir-locals.el for setting tab-width
0011033e3 fixed#150 : blank lines when converting lz4.1 to .ps or .pdf
248553c5e minor : show in table that frame checksum is optional
cd18ed308 create link to dynamic library without using `cp -a` (#147)
fb6fece77 Updated Makefile and .travis
8a750aa35 Add static library target for msbuild
13f12aa02 Wrapped possibly empty string (Emscripten) with quotes (@Fordi)
4a5a003c4 updated cli help
9aa5504f0 New unalign access method
e5aee601e updated xxhash
e64345506 Merge pull request #142 from bioothod/master
d15ac1f1a Slightly improved compression speed on ARMv6
0f2bf0c54 Improved performance on ARMv6
fb4d3ef2c Added .gitignore
4b4384772 Updated xxhash to r41
2a6ad2a7f debian: changed name lz4->liblz4
2f3e4bf32 debian: changed name lz4->liblz4
31fed3167 debian: changed name lz4->liblz4
309d1e087 debian: remove sbin dir
2131eedf7 debian: renamed lz4->liblz4, updated installed library files
b47786b9e debian: updated debianization to cdbs
a8f33076a cmake: when building library make it shared
33fa0bfb7 Added debianization
b4b739c4b cmake: bump version to the latest release
5dd12b4a7 faster sanitize tests
60119e258 Makefile : generates *.o for faster processing
9517bd923 minor simplification lz4_readarch()
a60d11e46 update on decompression speed
00eedb37d minor refactor
3e579afa6 Slightly improved decompression speed Slightly decreased decompression stack usage in 64-bits mode
795a2cdb1 sligthly improved decompression speed
a3c6daba6 liblz4 : Moved XXH_NAMESPACE to CPPFLAGS
d86dc9167 Merge pull request #131 from Cyan4973/dev
dfabec2d1 Fixed a few minor visual analyzer warnings
4abc86e0b Increased stack memory considered normal by Visual static analyzer
db59c7d33 Merge pull request #130 from Cyan4973/dev
8901dcdc9 removed one malloc
1098d897a lz4io : removed one malloc
81f19264a lz4cli : add one malloc test
aa1350e13 lz4hc : added one malloc test
6fe48b918 changed datagen to remove one malloc
15603d956 Added static analyzer test in Visual
48feb909d Merge pull request #129 from Cyan4973/dev
57e14d4c1 renamed directory to versionsTest
14064beaf Updated NEWS
7273da38b minor example clarification
0cc38f9eb Updated lib readme
454d109c5 Fixed : fuzzer display messages
caba2cc0f staticAnalyze use same compilation flags as normal build
8e8dc17ca Ensured make test is not confused with directory test
192ee724a Added namespace ability to xxhash
c04df7e1b Changed : static library is no longer compiled with -fPIC by default (this option can still be added on the command line) See #53
cdef03389 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
992faef5a Merge pull request #124 from toddlipcon/master
3d9adf7af Merge pull request #122 from keszybz/master
02ff90ac9 Merge pull request #121 from keszybz/add-gitignore
91138150c Fix cmake build to use gnu flags on clang
2ce2e6008 Minor doc clarification on sparse mode support
b17d3220d Minor lib readme update
3bcf68ff3 Use lz4 binary to check output
386d216cd Add example which uses the frame api through the library
0e0c32dfb Make git ignore generated files
7a532eedd Minor update on source ports
8a7bcead9 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
1cec6cb03 Updated badges
7f3d82def Merge pull request #114 from lpsantil/djgpp
f0a0e4430 READM fixup
a3a479fc2 djgpp port now using the proposed contrib structure
00b6535b4 Reverted all `Makefile` changes
936f75b4d Renamed the djgpp README so that it gets converted to HTML by github. Fixed breakage of `make install`.
81d6b5634 Makefile cleanups to make building with Andrew Wu's build-djgpp cross compilers (and everyone else) simpler.
faa0bce4e Fixup lib/Makefile so that is builds a library for djgpp using Andrew Wu's build-djgpp cross compilers
b2bbb2790 Fixup programs/{Makefile,lz4cli.c,lz4io.c} to build with Andrew Wu's build-djgpp cross compilers. Add some documentation in README.DJ
git-subtree-dir: src/lz4
git-subtree-split: c10863b98e1503af90616ae99725ecd120265dfb
b02bfa754 Tag open source release 1.1.7.
824e6718b Add a loop alignment directive to work around a performance regression.
55924d110 Add GNUInstallDirs to CMake configuration.
632cd0f12 Use 64-bit optimized code path for ARM64.
77c12adc1 Add unistd.h checks back to the CMake build.
c8049c582 Replace getpagesize() with sysconf(_SC_PAGESIZE).
18e2f220d Add guidelines for opensource contributions.
f0d3237c3 Use _BitScanForward and _BitScanReverse on MSVC.
71b8f8688 Add SNAPPY_ prefix to PREDICT_{TRUE,FALSE} macros.
be6dc3db8 Redo CMake configuration.
e4de6ce08 Small improvements to open source CI configuration.
c756f7f5d Support both static and shared library CMake builds.
038a3329b Inline DISALLOW_COPY_AND_ASSIGN.
a8b239c3d snappy: Remove autoconf build configuration.
27671c6ae Clean up CMake header and type checks.
548501c98 zippy: Re-release snappy 1.1.5 as 1.1.6.
513df5fb5 Tag open source release 1.1.5.
5bc9c82ae Set minimum CMake version to 3.1.
e9720a001 Update Travis CI config, add AppVeyor for Windows CI coverage.
f24f9d2d9 Explicitly copy internal::wordmask to the stack array to work around a compiler optimization with LLVM that converts const stack arrays to global arrays. This is a temporary change and should be reverted when https://reviews.llvm.org/D30759 is fixed.
82deffcde Remove benchmarking support for fastlz.
18488d621 Use 64 bit little endian on ppc64le.
7b9532b87 Improve the SSE2 macro check on Windows.
7dadceea5 Check for the existence of sys/uio.h in autoconf build.
83179dd8b Remove quicklz and lzf support in benchmarks.
c8131680d Provide a CMakeLists.txt.
ed3b7b242 Clean up unused function warnings in snappy.
8b60aac4f Remove "using namespace std;" from zippy-stubs-internal.h.
7d7a8ec80 Add Travis CI configuration to snappy and fix the make build.
1cd3ab02e Rename README to README.md. It already in markdown, we might as well let github know so that it renders nicely.
597fa795d Delete UnalignedCopy64 from snappy-stubs since the version in snappy.cc is more robust and possibly faster (assuming the compiler knows how to best copy 8 bytes between locations in memory the fastest way possible - a rather safe bet).
039b3a7ac Add std:: prefix to STL non-type names.
3c706d223 Make UnalignedCopy64 not exhibit undefined behavior when src and dst overlap.
d3c6d20d0 Add compression size reporting hooks.
626e1b9fa Use #ifdef __SSE2__ for the emmintrin.h include, otherwise snappy.cc does not compile with -march=prescott.
2d99bd14d 1.1.4 release.
8bfb028b6 Improve zippy decompression speed.
818b58338 adds std:: to stl types (#061)
27c5d8652 Re-work fast path for handling copies in zippy decompression.
4a7409408 Speed up Zippy decompression in PIE mode by removing the penalty for global array access.
38a5ec5fc Re-work fast path that emits copies in zippy compression.
094c67de8 Speed up the EmitLiteral fast path, +1.62% for ZFlat benchmarks.
fce661fa8 Speed up zippy decompression by removing some zero-extensions.
e788e527d Avoid calling memset when resizing the buffer.
32d6d7d8a Merge pull request #6 from deviance/provide-pkg-config-data
971613510 Add #ifdef to guard against macro redefinition if this is included in another Google project that also defines this.
0000f997d Merge pull request #13 from huachaohuang/patch-1
d53de1879 Make heuristic match skipping more aggressive.
2b9152d9c Default to glibtoolize instead of libtoolize if it exists, and also make it customizable through the environment variable $LIBTOOLIZE.
0800b1e4c Work around an issue where some compilers interpret <:: as a trigraph. Also correct the namespace name.
e7d2818d1 Unbreak the open-source build for ARM due to missing ATTRIBUTE_PACKED declaration.
7525a1600 Fix an issue where the ByteSource path (used for parsing std::string) would incorrectly accept some invalid varints that the other path would not, causing potential CHECK-failures if the unit test were run with --write_uncompressed and a corrupted input file.
ef5598aa0 Make UNALIGNED_LOAD16/32 on ARMv7 go through an explicitly unaligned struct, to avoid the compiler coalescing multiple loads into a single load instruction (which only work for aligned accesses).
b8cd908a8 Allow to compile in nested packages.
96a2e340f Update URLs in the Snappy README to reflect the move to GitHub.
0852af760 Move the logic from ComputeTable into the unit test, which means it's run automatically together with the other tests, and also removes the stray function ComputeTable() (which was never referenced by anything else in the open-source version, causing compiler warnings for some) out of the core library.
d80342922 Fix signed-vs.-unsigned comparison warnings.
d2cb73b6a Provide pkg-config data
efb39e81b Release Snappy 1.1.3; getting the new Uncompress variant in a release is nice, and it's also good to finally get an official release out after the migration to GitHub.
eb66d8176 Initialized members of SnappyArrayWriter and SnappyDecompressionValidator. These members were almost surely initialized before use by other member functions, but Coverity was warning about this. Eliminating these warnings minimizes clutter in that report and the likelihood of overlooking a real bug.
b2312c4c2 Add support for Uncompress(source, sink). Various changes to allow Uncompress(source, sink) to get the same performance as the different variants of Uncompress to Cord/DataBuffer/String/FlatBuffer.
b2ad96006 Changes to eliminate compiler warnings on MSVC
e7a897e18 Fixed unit tests to compile under MSVC.
86eb8b152 Change a few branch annotations that profiling found to be wrong. Overall performance is neutral or slightly positive.
11ccdfb86 Sync with various Google-internal changes.
22acaf438 Change some internal path names.
git-subtree-dir: src/snappy/snappy
git-subtree-split: b02bfa754ebf27921d8da3bd2517eab445b84ff9
* Update unity build for RocksDB changes
* Log RocksDB options on startup
* Support RocksDB option strings
* Support full file bloom filters
You can now configure most RocksDB options with RocksDB's option
string scheme.
Set "filter_full" to 1 to make bloom filters for an
entire file rather than each block. More memory will be
needed during compaction but less memory will be needed
during fetching for large databases. Does nothing unless
bloom filters are enabled with "filter_bits".
Example:
options = max_compaction_bytes=64;max_bytes_for_level_multiplier=64
clock_cache_mb = 96
filter_bits = 10
filter_full = 1
Do not dispatch a transaction received from a peer for
processing, if it has already been dispatched within the
past ten seconds.
Increase the number of transaction handlers that can be in
flight in the job queue and decrease the relative cost for
peers to share transaction and ledger data.
Additionally, make better use of resources by adjusting the
number of threads we initialize, by reverting commit
68b8ffdb63.
aead40417 fix HISTORY.md typo
a0cdc3cec Bump version to 5.8.7
7513f6350 Fix IOError on WAL write doesn't propagate to write group follower
9e47084ce Bump version to 5.8.6
36074ba5d Enable cacheline_aligned_alloc() to allocate from jemalloc if enabled.
aa00523e0 Add -DPORTABLE=1 to MSVC CI build
cf2b98237 Bump version to 5.8.5
e8c9350f2 Blob DB: not using PinnableSlice move assignment
4907d2463 Bump version to 5.8.4
5d928c795 Blob DB: Fix race condition between flush and write
725bb9d66 Blob DB: Fix release build
b7367fe84 Bump version to 5.8.3
13b2a9b6f Blob DB: use compression in file header instead of global options
5dc70a15c Fix PinnableSlice move assignment
9019e9125 dynamically change current memtable size
7f1815c37 Bump version to 5.8.2
2584a18ef Blob DB: Fix BlobDBTest::SnapshotAndGarbageCollection asan failure
17f67b546 PinnableSlice move assignment
6fb56c582 Blob DB: Add compaction filter to remove expired blob index entries
f90ced92f Blob DB: fix snapshot handling
632f36dcd Blob DB: option to enable garbage collection
11bacd578 Blob DB: Fix flaky BlobDBTest::GCExpiredKeyWhileOverwriting test
f98efcb1e Blob DB: Evict oldest blob file when close to blob db size limit
c1e99eddc Blob DB: cleanup unused options
ffc3c62ca Blob DB: Initialize all fields in Blob Header, Footer and Record structs
9e8254090 Blob DB: update blob file format
d66bb21e1 Blob DB: Inline small values in base DB
05d5c575a Return write error on reaching blob dir size limit
2b8893b9e Blob DB: Store blob index as kTypeBlobIndex in base db
419b93c56 Blob DB: not writing sequence number as blob record footer
8afb0036c fix lite build
dded348dd Blob DB: Move BlobFile definition to a separate file
374736123 add GetLiveFiles and GetLiveFilesMetaData for BlobDB
8cff6e945 Enable WAL for blob index
c29347290 Add ValueType::kTypeBlobIndex
eae53de3b Make it explicit blob db doesn't support CF
65aec19df Fix memory leak on blob db open
30b38c98c TableProperty::oldest_key_time defaults to 0
2879f4beb Bump version to 5.8.1
88595c882 Add DB::Properties::kEstimateOldestKeyTime
266ac245a Bumping version to 5.8
64185c23a update HISTORY.md for DeleteRange bug fix
e83d6a02e Not using aligned_alloc with gcc4 + asan
0980dc6c9 Fix wrong smallest key of delete range tombstones
b76797231 avoid use-after-move error
c41744270 CMake: Fix formatting
c21ea8f7a CMake: Add support for CMake packages
544434558 add Erlang to the list of language bindings
2972a702a Minor updates to FlushWAL blog
fbfa3e7a4 WriteAtPrepare: Efficient read from snapshot list
b01f426f5 Blog post for FlushWAL
503db684f make blob file close synchronous
3c840d1a6 Allow DB reopen with reduced options.num_levels
92bfd6c50 Fix DropColumnFamily data race
7fdf735d5 Pinnableslice examples and blog post
7fbb9ecca support disabling checksum in block-based table
19cc66dc4 fix clang bug in block-based table reader
7eba54eb9 test compaction input-level split range tombstone assumption
cd26af347 Add unit test for WritePrepared skeleton
a12479819 Improved transactions support in C API
c10b39131 LANGUAGE-BINDINGS.md: add another rust binding
90177432e Remove leftover references to phutil_module_cache
234f33a3f allow nullptr Slice only as sentinel
ccf7f833e Use PinnableSlice in Transactions
1dfcdb15f Extend pin_l0 to filter partitions
39ef90055 stop calling memcmp with nullptrs
78cb6b611 Provide byte[] version of SstFileWriter.merge to reduce GC Stall
867fe92e5 Scale histogram bucket size by constant factor
f004307e9 CMake improvements
09ac6206a Circumvent ASAN false positive
5b68b114f Blob db create a snapshot before every read
4624ae52c GC the oldest file when out of space
8ace1f79b add counter for deletion dropping optimization
0d8e992b4 Revert the mistake in version update
5358a8056 add VerifyChecksum to HISTORY.md
ed0a4c93e perf_context measure user bytes read
1efc600dd Preload l0 index partitions
bddd5d363 Added mechanism to track deadlock chain
c1384a707 fix db_stress uint64_t to int32 cast
29877ec7b Fix blob db crash during calculating write amp
8f2598ac9 Enable Cassandra merge operator to be called with a single merge operand
9a44b4c32 Allow merge operator to be called even with a single operand
ac8fb77af fix some misspellings
23593171c minor improvements to db_stress
af012c0f8 fix deleterange with memtable prefix bloom
1c8dbe2aa update scores after picking universal compaction
eb6425303 Update WritePrepared with the pseudo code
132306fbf Remove PartialMerge implementation from Cassandra merge operator
71598cdc7 Fix false removal of tombstone issue in FIFO and kCompactionStyleNone
3204a4f64 Fix missing stdlib include required for abort()
7aa96db7a db_stress rolling active window
dfa6c23c4 Update RocksDBCommonHelper to use escapeshellarg
e367774d1 Overload new[] to properly align LRUCacheShard
ad42d2fcb Remove residual arcanist_util directory
279296f4d properly set C[XX]FLAGS during CMake configure-time checks
c5f0c6cc6 compile with correct flags to determine SSE4.2 support
185ade4c0 cmake: support more compression type
5449c0990 rocksdb: make buildable on aarch64
a144a9782 Fix for CMakeLists.txt on Windows for RocksJava
acf935e40 fix deletion dropping in intra-L0
8254e9b57 make sst_dump compression size command consistent
74f18c130 db_bench support for non-uniform column family ops
5de98f2d5 approximate histogram stats to save cpu
3f5888430 Fix c_test ASAN failure
e5a1b727c Fix blob DB transaction usage while GC
6f051e0c7 fix corruption_test valgrind
ac098a462 expose set_skip_stats_update_on_db_open to C bindings
666a005f9 Support prefetch last 512KB with direct I/O in block based file reader
ad77ee0ea Revert "Makefile: correct faligned-new test"
b87ee6f77 Use more keys per lock in daily TSAN crash test
25df24254 Add column families related functions (C API)
64f848435 block_cache_tier: fix gcc-7 warnings
0cecf8155 Write batch for `TransactionDB` in C API
6a9de4347 Windows.h macro call fix
23c7d1354 fix comment
1fbad84b6 Makefile: correct faligned-new test
7848f0b24 add VerifyChecksum() to db.h
47ed3bfc3 fix WinEnv assertions
d97a72d63 Try to repair db with wal_dir option, avoid leak some WAL files
36375de76 gcc-7/i386: markup intentional fallthroughs
bdc056f8a Refactor PessimisticTransaction
a9a4e89c3 Fix valgrind complaint about initialization
4ca11b4b0 Update USERS.md
c9804e007 Refactor TransactionDBImpl
20dc5e74f Optimize range-delete aggregator call in merge helper.
0d4a2b733 Avoid blob db call Sync() while writing
627c9f1ab Don't add -ljemalloc when DISABLE_JEMALLOC is set
dce6d5a83 db_bench background work thread pool size arguments
4f81ab38b Makefile: fix for GCC 7+ and clang 4+
92afe830f Update all blob db TTL and timestamps to uint64_t
5883a1ae2 Fix /bin/bash shebangs
cc01985db Introduce bottom-pri thread pool for large universal compactions
0b814ba92 Allow concurrent writes to blob db
2c45ada4c Blob DB garbage collection should keep keys with newer version
58410aee4 Fix the overflow bug in AwaitState
c3d5c4d38 Refactor TransactionImpl
060ccd4f8 support multiple CFs with OPTIONS file
345387067 Fix statistics in RocksJava sample
1900771bd Dump Blob DB options to info log
3218edc57 Fix universal compaction bug
6a36b3a7b fix db get/write stats
a84cee812 Add a missing "once" in .h
21696ba50 Replace dynamic_cast<>
e85f2c64c Prevent empty memtables from using a lot of memory
ac748c57e Fix FIFO Compaction with TTL tests
aaf42fe77 Move blob_db/ttl_extractor.h into blob_db/blob_db.h
aace46516 Fix license headers in Cassandra related files
50a969131 CacheActivityLogger, component to log cache activity into a file
6083bc79f Blob DB TTL extractor
710411aea fix asan/valgrind for TableCache cleanup
3a3fb00b7 TARGETS file not setting sse explicitly
fca4d6da1 Build fewer tests in Travis platform_dependent tests
8f553d3c5 remove unnecessary internal_comparator param in newIterator
7f6d012d7 "ccache -C" in Travis
d12691b86 move TableCache::EraseHandle outside of db mutex
f33f11368 fix db_bench argument type
e7697b8ce Fix LITE unit tests
3ce20e985 Fix use of RocksDBCommonHelper in cont_integration.sh
c281b4482 Revert "CRC32 Power Optimization Changes"
9980de262 Fix FIFO compaction picker test
2289d3811 CRC32 Power Optimization Changes
30b58cf71 Remove the orphan assert on !need_log_sync
fe1a5559f Fix flaky write_callback_test
addbd279c 5.6.1 release blog post
30edff308 buckification: remove explicit `-msse*` compiler flags
2b259c9d4 Lower num of iterations in DeadlockCycle test
277f6f23d Release note for partitioned index/filters
5e731a138 Remove unused rocksdb arcanist lib
9b11d4345 Fix broken links
06f191744 add vcpkg as an windows option
ea8ad4f67 Fix compaction div by zero logging
34112aeff Added db paths to c
1d8aa2961 Gcc 7 ParsedInternalKey replace memset with clear function.
a4c42e800 Fix UBSAN issue of passing nullptr to memcmp
16e038820 LRUCacheShard cache line size alignment
216644c61 enable UBSAN macro in TARGETS
e67b35c07 Add Iterator::Refresh()
a34b2e388 Fix caching of compaction picker's next index
72502cf22 Revert "comment out unused parameters"
1d7048c59 comment out unused parameters
534c255c7 Cassandra compaction filter for purge expired columns and rows
63163a8c6 Remove make_new_version.sh
0302da47a Reduce blob db noisy logging
3e5ea29a8 Fix Flaky DeleteSchedulerTest::ImmediateDeleteOn25PercDBSize
a22b9cc6f overlapping endpoint fixes in level compaction picker
ffd2a2eef delete ExpandInputsToCleanCut failure log
3e6e863b1 Remove arcanist_util directory
36651d14e Moving static AdaptationContext to outside function
6e3ee015f Update java/rocksjni.pom
ecff9d5e3 Include write_buffer_manager in ImmutableDBOptions::Dump
ae28634e9 Remove some left-over BSD headers
33b1de82a Remove format compatibility hack
2f375154e checkout local branch in check_format_compatible.sh
ddb22ac59 avoid collision with master branch in check format
0c03a7f17 set the remote for git checkout
7ac184c6d Revert cmake -DNDEBUG for non-MSVC
0655b5858 enable PinnableSlice for RowCache
00464a314 Fix column_family_test with LITE build
b2dd192fe tools/write_stress.cc: Correct "1204" typos.
cbaab3044 table/block.h: change memset
f1a056e00 CodeMod: Prefer ADD_FAILURE() over EXPECT_TRUE(false), et cetera
4a2e4891f Add back the LevelDB license file
a7321fc97 Remove the licensing description in CONTRIBUTING.md
3c327ac2d Change RocksDB License
132013366 Make TARGETS file portable
ccf5f08f8 Set CACHE_LINE_SIZE for s390, PPC, ARM64
67510eeff db_crashtest.py: remove need for shell
4267eb00d Remove punit tests
5bfb67d90 Enable write rate limit for updaterandom benchmark
20a691d98 Update HISTORY to release 5.7
98d1a5510 db_bench to by default verify checksum
26ce69b19 Update blob db to use ROCKS_LOG_* macro
43e4eef77 remove unnecessary fadvise
21b17d768 Fix BlobDB::Get which only get out the value offset
70440f7a6 Add virtual func IsDeleteRangeSupported
7550255c5 Add JAVAC_ARGS as Makefile variable
7a0b5de77 Gcc 7 ignored quantifiers
000bf0af3 Improve the design and native object management of Stats in RocksJava
269d383d5 Bump version to 5.7
c32f27223 Fixes db_bench with blob db
fcd99d27c db_bench_tool: fix buffer size
87128bd5c fix regression test
8f927e5f7 Fix undefined behavior in Hash
643b787c7 Added a note about LZ4 compression dependency
56656e12d Temporarily disable FIFOCompactionWithTTLTest
b5fb85ec5 fix valgrind init complaint
657df29ea Add max_background_jobs to db_bench
a43c053ad remove duplicated utilities/merge_operators/cassandra/test_utils.cc in src.mk
7c4a9e6c9 Initialize a variable in ldb to make code analysis tool happy
98669b535 init filters_in_partition_
0013bf14e fix asan and valgrind leak report in test
521b4c28b rocksdb 5.5.1 release post
33042573d Fix GetCurrentTime() initialization for valgrind
f6b9d9355 Fix clang error in PartitionedFilterBlockBuilder
45b9bb033 Cut filter partition based on metadata_block_size
f4ae1bab0 update history for OnBackgroundError and DeleteRange fix
1cb8c6de6 Add -enable_pipelined_write to db_bench and add two defaults
7604b463b Update the AddDBStats in LITE
1e34d07e1 Simplify and document sync rules for logs_ etc
d310e0f33 Regression test for empty dedicated range deletion file
e9f91a517 Add a fetch_add variation to AddDBStats
c1b375e96 skip generating empty sst
67b417d62 fix format compatible test
afbef6518 Bug fix: Fast CRC Support printing is not honest
397ab1115 Improve Status message for block checksum mismatches
18c63af6e Make "make analyze" happy
01534db24 Fix the reported asan issues
1cd45cd1b FIFO Compaction with TTL
982cec22a Fix TARGETS file tests list
b49b37109 allow numa >= 2.0.8
e517bfa2c CLANG Tidy
dc3d2e4d2 update compatible test
89468c01d Fix Windows build broken by 5c97a7c0664d4071768113814e9ba71fe87e18cf
51778612c Encryption at rest support
7061912c2 Trivial typo in HISTORY.md
2a9cd8799 Fix jni WriteBatchThreadedTest
0025a3640 revert perf_context and io_stats to __thread
5c97a7c06 Unit Tests for sync, range sync and file close failures
4cee11f4e Intra-L0 blog post
857e9960b Improve the error message for I/O related errors.
d757355cb Fix bug that flush doesn't respond to fsync result
8e6345d2d Update rename of ParanoidCheck
499ebb3ab Optimize for serial commits in 2PC
0ac4afb97 Sanitize partitioning options
521724ba8 fixed wrong type for "allow_compaction" parameter
71f5bcb73 Introduce OnBackgroundError callback
88cd2d96e Downgrade option sanitiy check level for prefix_extractor
6837a1762 Fix Data Race Between CreateColumnFamily() and GetAggregatedIntProperty()
af1746751 WriteBufferManager will not trigger flush if much data is already being flushed
9467eb614 Fix flush assertion with tsan
048446fc7 Fix cassandra ASAN use-after-free
a21db161c Implement ReopenWritibaleFile on Windows and other fixes
c430d69ee fix coredump for release nullptr
0d278456c default implementation for InRange
cbd825dee Create a MergeOperator for Cassandra Row Value
2c98b06bf Remove pin_slice option by making it the default
c80c6115d add db_bench options for partitioning
6a3377f45 Synchronize statistic enumeration values between statistics.h and java API
53dda8797 Do not run RateLimiterTest.Rate test on Travis+Mac OSX.
ae8571f5c Fix blob db compression bug
7a380deff Update blob_db_test
89ad9f3ad Allow ignoring unknown options when loading options from a file
6b5a5dc5d fixed typo
0f228be3b fixed typo in util/dynamic_bloom.h
c217e0b9c Call RateLimiter for compaction reads
91e2aa3ce write exact sequence number for each put in write batch
6f4154d69 record index partition properties
5d5a28a98 Fix Clang release build broken by 5582123dee8426a5191dfd5e846cea8c676c793c
0175d58c3 Make direct I/O write use incremental buffer
7a270069b GNU C library for struct tm has 2 additional fields.
d713471da Limit trash directory to be 25% of total DB
9bb91e932 Dedup release
27b450165 Update HistogramTypes in the Java API
e97304c68 update history for 5.6
5582123de Sample number of reads per SST file
db818d2d1 Fix RocksDB Lite build with CLANG
a472c4ae4 update 5.5 change log
bc09c8a0d Fix crash in PosixWritableFile::Close() when fstat() fails
6d0f22e42 Fix mock_env.cc uninitialized variable
c2012d488 Java APIs for put, merge and delete in file ingestion
85dace2af Disable DBRangeDelTest::TailingIteratorRangeTombstoneUnsupported for ubsan
d4f7731b6 fix travis error with init time in mockenv
550a1df72 Fix clang errors by asserting the precondition
cc5f9339e Fix concurrency issue with filter_block_set_
2e64f450d bump version to 5.6
afbc2d0d2 Force travis to build with clang on MacOS
b172a3f1a Fix warnings while generating RocksJava documentation
52a7f38b1 WriteOptions.low_pri which can throttle low pri writes if needed
26a8a8071 Switch from CentOS 5 to CentOS 6 for crossbuilding RocksJava
dba9f3722 Fix db_write_test clang/windows build failure
c7662a44a fixed typo
7e8d95cc0 Fix the Java build which was broken by a4d9c02
7e5fac2c3 remove test dir before exit when current regression is running
7f6c02dda using ThreadLocalPtr to hide ROCKSDB_SUPPORT_THREAD_LOCAL from public…
138b87eae Fix interaction between CompactionFilter::Decision::kRemoveAndSkipUnt…
95b0e89b5 Improve write buffer manager (and allow the size to be tracked in block cache)
a4d9c0251 Pass CF ID to MemTableRepFactory
f68d88be5 Fix DBWriteTest::ReturnSequenceNumberMultiThreaded data race
215076ef0 Fix TSAN: avoid arena mode with range deletions
3a8a848a5 account for L0 size in estimated compaction bytes
0fae3f5dd codemod: format TARGETS with buildifier [5/5] (D5092623)
872199606 add checkpoint support for single db in regression test
5a9b4d743 Retire memenv https://github.com/facebook/rocksdb/pull/2082
d6019651b sync internal/external TARGETS
bbaba51bb Add missing index type to C-API
292edfd51 travis: test with xcode8.3 (OS X 10.12)
0dc3040d5 db: avoid `#include`ing malloc and jemalloc simultaneously
9b3ed8350 fix regression test
9c9909bf7 Support ingest file when range deletions exist
ad19eb868 Fixing blob db sequence number handling
51ac91f58 Histogram of number of merge operands
345878a7f update blob_db_test
cbc821c25 change regression rebuild to one level
103d0692e Avoid unsupported attributes when not building with UBSAN
5fd04566c travis: reduce the number of travis builders
2d05002b9 RocksDB 5.4.5 release blog post
7eca90f18 Update RocksDB blog authors
d03c34497 update comment of GetNextFile
f7bb1a006 support merge and delete in file ingestion
c2c62ad4e Reorder variables of ReadOptions
7bb1f5d48 Increase of compaction threads should be logged at info level instead of a warning
6c456ecae Clean zstd files
347e16f83 codemod: replace `headers = AutoHeaders.*` with `auto_headers`
0be636bf7 Fix db_bench build break with blob db
135ee6a3f fix tsan crash data race
a99fb9928 fix column_family_test asan
f41bffb3d travis: clang-3.6 -> clang-4.0
e7612798b update buckifer/TARGETS
bb01c1880 Introduce max_background_jobs mutable option
5a2530462 Fix the CMakeLists for RocksJava
41cbb7274 options.delayed_write_rate use the rate of rate_limiter by default.
506803466 range sync should be enabled
02594b5f1 Fix build errors in blob_dump_tool with GCC 4.8
52d9e5f7b Fix column family seconds_up accounting
7d8207f1f Fix errors in clang-analyzer builds
85b8569ae Fix release build on Linux
69ec8356b Allow SstFileWriter to use the rate limiter
6cc9aef16 New API for background work in single thread pool
9d0a07ed5 Fix rocksdb.estimate-num-keys DB property underflow
578fb0b1d Simple blob file dumper
ac39d6bec Core-local statistics
3e86c0f07 disable direct reads for log and manifest and add direct io to tests
15ba4d6c4 Address MS Visual Studio 2017 issue with autovector
88c818e43 Replace deprecated RocksDB#addFile with RocksDB#ingestExternalFile
228f49d20 Fix data races caught by tsan
4c9d2b104 remove #include port/port.h in public header file
07bdcb91f New WriteImpl to pipeline WAL/memtable write
d746aead1 Suppress clang-analyzer false positive
217b866f4 column_family_test: EnvCounter::num_new_writable_file_ to be atomic
9f839a7f6 keep util/build_version.cc when make clean
7eecd40a4 add emacs tags file - etags
9bbba4fec Remoe unused BlockBasedTable::compaction_optimized_
f5ba131bf Fixed some spelling mistakes
146b7718f Fix mingw compilation with -DNDEBUG
a36220ccf fix unity test
0ebdd7057 fixed typo
8032f4cb3 Remove -pie in TSAN
362ba9b02 Release RocksDB 5.5.0
ba685a472 Support ingest_behind for IngestExternalFile
01ab7b528 Add ROCKSDB_LIBRARY_API macro to a few C APIs, to fix windows build
cb9392a09 add Transactions and Checkpoint to C API
445f1235b s/std::snprintf/snprintf
cd593c283 Fix travis java_test
f720796e2 fixed typo
a48a62d5b define NDEBUG in CMake non-debug builds
1ca723dbd C API: support pinnable get
2ef15b85d Core-local stats blog post
4f9e69ccf fix log err
11c5d4741 cross-platform compatibility improvements
d00433302 Put lib files into suitable path in RPM package
86d549253 Fix build error with blob DB.
254c46800 Fix the RocksJava Release on Windows
7a47b431f Fix .gitignore pattern
fa5a15ceb Make sure that zstd is statically linked correctly in the Java static build
3fa9a39c6 Add GetAllKeyVersions API
1a60982a5 Simplified instructions for CentOS
a5cc7ecec Facility for cross-building RocksJava using Docker
ccd3dddf5 Blog post for partitioned index/filters
b145c34d7 Update blog authors
bbe9ee7dd core-local array type conversions
c2be43430 Build and link with ZStd when creating the static RocksJava build
c61e72c42 Add missing files of blob_db to CMake file
3907c94ff Fix ColumnFamilyTest:BulkAddDrop
cda5fde2d CoreLocalArray class
93949667c update TARGETS
4e83b8001 title: Bulkoading -> title: Bulkloading
d85ff4953 Blob storage pr
492fc49a8 fix readampbitmap tests
be421b0b1 portable sched_getcpu calls
0f559abdb Add NO_UPDATE_BUILD_VERSION option to makefile
3a04a254a Flink state
35df23fe8 Fix suite exclisions
e7cea86f7 Fixes the CentOS 5 cross-building of RocksJava
259a00eac unbiase readamp bitmap
a62096696 port: updated PhysicalCoreID()
df035b682 Print compaction_options_universal.stop_style in LOG file
4897eb250 dont skip IO for filter blocks
3f73d54bb Add C API to set max_file_opening_threads option
0b69e5079 Define CACHE_LINE_SIZE only when it's not defined
2cd00773c Add bulk create/drop column family API
40af2381e Object lifetime in cache
fdaefa030 travis: add Windows cross-compilation
a30a69603 do not read next datablock if upperbound is reached
2d42cf5ea Roundup read bytes in ReadaheadRandomAccessFile
264d3f540 Allow IntraL0 compaction in FIFO Compaction
8c3a180e8 Set lower-bound on dynamic level sizes
7c1c8ce5a Avoid calling fallocate with UINT64_MAX
a45e98a5b max_open_files dynamic set, follow up
6b99dbe04 fix memory alignment with logical sector size
e7ae4a3a0 Max open files mutable
60847a3b0 port: updated PhysicalCoreID()
b551104e0 support PopSavePoint for WriteBatch
498693cf3 Remove orphaned Java classes
5e2ebf2bd travis: add CMake compilation
af6fe69e4 Fix an issue of manual / auto compaction data race
6798d1f3b Revert "Delete filter before closing the table"
89833577a Delete filter before closing the table
47a09b0a8 Avoid pinning when row cache is accessed
aeaba07b2 Remove an assert that causes TSAN failure.
0b90aa951 Set VALGRIND_VER
a2b05210e Make PosixLogger::flush_pending_ atomic
da4b2070b Fix WriteBatchWithIndex address use after scope error
d616ebea2 Add GPLv2 as an alternative license.
4439b4596 Add documentation license
0ca3ead0c add GetRootDB() in DeleteFilesInRange
cdad04b05 Remove double buffering on RandomRead on Windows.
e15382c09 Disable two flaky tests
2150cc1f3 fix WritableFile buffer size in direct IO
efc361ef7 Add user stats Reset API
6616e4d62 add prefetch to PosixRandomAccessFile in buffered io
f6a27d0bc Extract statistics tests into separate file
7eddecce1 support bulk loading with universal compaction
3b4d1b7a4 add <sys/sysmacros.h> to avoid warning with glibc 2.25
e5e545a02 Reunite checkpoint and backup core logic
72c21fb3f call GetRootDB() before cast to DBImpl* in CancelAllBackgroundWork
4c9447d88 Add erase option to release cache
04d58970c AIX and Solaris Sparc Support
afff9951e Respect deprecated flag in table options
066cfbacc Adding -noprofile to CMakeLists for Windows
cb885bccf set compaction_iterator earliest_snapshot to max if no snapshot
7534ba7bd StackableDB should pass ResetStats()
c1fbf91b2 Fixing Solaris Sparc crash due to cached TLS
963eeba48 Revert how check_format_compatible.sh checkout release branches.
97005dbd5 tools/check_format_compatible.sh to cover option file loading too
8f6196788 Add cpu usage to regression benchmarks (4th attempt)
df74b775e Limit backups opened
1dd776051 Change L0 compaction score using level size
927bbab25 Revert "Add cpu usage to regression benchmarks (3rd attempt)"
8e84a388e Re-add index_per_partition but as deprecated
1553659d6 Add more recent versions to tools/check_format_compatible.sh
966ebb02f Hide event listeners from lite build
476e80be8 Add cpu usage to regression benchmarks (3rd attempt)
c49d70465 Add DB:ResetStats()
0fcdccc33 Blob storage helper methods
a6439d797 CMake: compile with -O2
e67f0adf3 enable O2 optimization for lz4
bc3973259 CMake: add support for SSE4.2
7d5f5aa97 Separate compile and link for shared library
071652734 remove warning
6e8d6f429 readahead backwards from sst end
ca96654d8 Change Build Env to gcc-5
e9e6e5324 Simplify write thread logic
6799c7e00 Pass in remote as a param to branch creation script
44fa8ece9 change use_direct_writes to use_direct_io_for_flush_and_compaction
13b50358f add space for buggy kernel warning
b6b9359ec Fix BYTES_WRITTEN accounting
13369fbd4 Update ShipIt to honor TARGETS updates
f2449ce92 Remove .deprecated_arcconfig
415be221c RocksDB Release 5.4 : Update HISTORY.md and build version.
3eab41d7c java dependencies test -s -> use test -d
a22ed4eab internal_repo_rocksdb to build Java and RocksDB LITE
9f2cc59ec sync TARGETS file
10d754696 set readahead buffer size from roundup(user_size) + 4k to roundup(use…
ba7da434a fix db_stress crash caused by buggy kernel warning
6257837d8 Add ROCKSDB_JAVA_NO_COMPRESSION flag
6a6723ee1 Move MergeOperatorPinning tests to be with other merge operator tests
6a8d5c015 Revert "Report cpu usage using time command"
8f47a9751 File level histogram should be printed per CF, not per DB
9300ef545 Fix shared lock upgrades
1f8b119ed Limit maximum memory used in the WriteBatch representation
97ec8a134 Report cpu usage using time command
20778f2f9 Adding comments to the write path
7124268a0 Reduce the number of params needed to construct DBIter
04abb2b2d FreeBSD only requires WITH_JEMALLOC, not the rest
61730186d dummy diff
360e9960f Summary: Remove .arcconfig
69a5e6461 Deprecate .arcconfig
9690653db Add a verify phase to benchmarks
dd8f9e38e Fix compilation for GCC-5
c2954f9b6 Add Travis job to build examples
72fc1e9d0 avoid non-existent O_DIRECT on OpenBSD
ff9728701 Refactor compaction picker code
9e7293902 only FALLOC_FL_PUNCH_HOLE when ftruncate is buggy
343b59d6e Move various string utility functions into string_util
1d068f606 Fix CompactRange incorrect buffer release
df6f5a377 Move memtable related files into memtable directory
107c5f6a6 CMake: more MinGW fixes
d2dce5611 Move some files under util/ to separate dirs
c50e3750d Use a human readable size for level report
ce64b8b71 Divide db/db_impl.cc
02799ad77 Revert "delete fallocate with punch_hole"
e2a7b202c Release note for partition filters
af256eb2b build db every monday
e5a1372b2 Rework test running script.
d659faad5 Level-based L0->L0 compaction
a12306fab Add a notice on gflags installation in INSTALL.md
43010a929 Revert "[rocksdb][PR] CMake: more MinGW fixes"
a30b75cdc Add buckifier script to github repo
3450ac8c1 CMake: more MinGW fixes
90cfd4645 update IterKey that can get user key and internal key explicitly
e2c6c0636 add TimedEnv
9e4453180 Refactor WriteImpl (pipeline write part 1)
6ef8c620d Move auto_roll_logger and filename out of db/
a1c469d71 Add release notes for PinnableSlice
0537f515c fix run_remote with strong quoting
72e600094 fixed misses on Centos library installation instructions
88cc81df5 auto_roll_logger_test to move away from real sleep
d25e28d58 replace sometimes-undefined uint type with unsigned int
a1d7e487b Add L0 write-amp to compaction level stats
b6d609063 CMake: support AVX2 in MinGW
bd7d13835 test remote instead run remote in regression test
c81a805fe test db existence on the remote host
5fc1e6765 add -rf when remove db in regression test
4ab4049f2 gflags has moved to GitHub
4e0065015 make all DB::Get overloads virtual
6401a8b76 Fix build with MinGW
80fe5b385 disable test: DeleteSchedulerTest.DynamicRateLimiting1
a9c86f51b backup garbage collect shared_checksum tmp files
da175f7ec exit with code 2 when there is already a db_bench running in regression test
0ee7f0403 Added missing options to RocksJava
c6d04f2ec Option to fail a request as incomplete when skipping too many internal keys
58179ec4a Cleanup of ThreadStatusUtil structures should use the DB's reference
f3607640a add ldb build to regression test
8a8c96746 Enable Fast CRC32 for Win64
f9813b853 Added SstFileWriter construtor without explicit comparator to JNI api
8d3cb4f20 Added naming of backup engine threads
67d762379 Expose the stalling information through DB::GetProperty()
0fd574926 delete fallocate with punch_hole
41fe9ad75 Hide usage of compaction_options_fifo from lite build
e7731d119 Configure index partition size
69c8d524a Fix jni library name for PowerPC Architecture
34a70859b Fix segmentation fault caused by #1961
8dee8cad9 Enable fifo compaction benchmark to db_bench
a5c8b5434 fix a header include
91b5feb37 Fix Windows Build broken by a recent commit
41ccae6d2 Add C API functions (and tests) for WriteBatchWithIndex
88bb6f6bf non_shm CI should run tests on /tmp
8888de2b1 Update .gitignore file in examples
203136e79 Fix Compilation errors when using IBM Java
f4fce4751 Fix clang compile error - [-Werror,-Wunused-lambda-capture]
a084b26a5 Blog post for releasing 5.2.1
15950fe3a Remove ASSERT_EQ(boolean, ...)
3e56c7e0c make total_log_size_ atomic
909028e21 HISTORY.md for log_size_for_flush in CreateCheckpoint()
be723a8d8 Optionally construct Post Processing Info map in MemTableInserter
e474df947 db_bench: not need to check mmap for PlainTable
8b0097b49 Readers for partition filter
9ef3627fd Allow checkpointing without flushing
17866ecc3 Allow Users to change customized ldb tools' header in help printing
a2a883318 remove deleted option from benchmark.sh
78cb19559 add checkpoint to ldb
4b04addfc updated solution if "make format" command fails
8f5bf0446 Flush triggered by DB write buffer size picks the oldest unflushed CF
6908e24b5 dynamic setting of stats_dump_period_sec through SetDBOption()
93c68b642 change regression bash file with debug mode
21d8c3179 remove LIKELY from public headers
36ad75778 INSTALL: document USE_SSE
9272e12f1 avoid ftruncate twice in buffered io
d52f334cb Break stalls when no bg work is happening
e66221add fix db_bench rate limiter callsites
dbae438a0 Replace "DEPRECATED" comment to "not supported"
995618a82 Support SstFileManager::SetDeleteRateBytesPerSecond()
e19163688 Add macros to include file name and line number during Logging
d525718a9 cleanup direct io flag in WritableFileWriter
5fa927aa9 Add Xpress and ZSTD CompressionType values to C header
11526252c Pinnableslice (2nd attempt)
e5bd8def1 update history.md for fixing the bug that skips keys
1ffbdfd9a Add a new SstFileWriter constructor without explicit comparator
ebd5639b6 Add ability to search for key prefix in sst_dump tool
e6725e8c8 Fix some bugs in MockEnv
900c62be6 fix compile for VS2015
fe1835617 release 5.3
5dae01947 Revert "Report cpu usage using time command"
f2817fb7f avoid ASSERT_EQ(false, ...);
5b11124e3 add max to histogram stats
d43adf21b Report cpu usage using time command
18fc1bc0e minor changes for rate limiter test flakiness
12ba00ea6 Reset DBIter::saved_key_ with proper user key anywhere before pass to DBIter::FindNextUserEntry
c9df05d1e Fix random access alignment
f64991537 Add Bulkoading IngestExternalFile blog post
54b434110 Builders for partition filter
97edc72d3 Add a memtable-only iterator
72202962f fix db_sst_test flakiness
5f65dc877 Expose DB::DeleteRange and WriteBath::DeleteRange in Java
58b12dfe3 Set logs as getting flushed before releasing lock, race condition fix
f8a4ea020 Move db_test and external_sst_file_test out of Travis's MAC OS run
534581a35 Fix a bug in tests in options operator=
a2f7a514d Refactoring
2a5daa06f Add stderr log level for ldb backup commands
4561275c2 fix rate limiter test flakiness
7c80a6d7d Statistic for how often rate limiter is drained
0ad5af42d Clarify VerifyBackup behavior
6fb901344 sanitize readahead when direct read enabled
f89b3893c Remove skip_table_builder_flush and default it to true
cc253982d Use more default options in db_bench
8432bcf55 Make compaction_pri settable through option string
d5b607a43 Make db_wal_test slightly faster
ba4c77bd6 Divide external_sst_file_test
e877afa08 Remove bulk loading and auto_roll_logger in rocksdb_lite
90d835507 Fix the wrong address for PREFETCH in DynamicBloom::Prefetch
08864df21 Move advanced column family options to advanced_options.h
2ca2059f6 Get unique_ptr to use delete[] for char[] in DumpMallocStats
253799c06 Add missing include for `abort()`
c6d464a9d Fixed various memory leaks and Java 8 JNI Compatibility
be3e5568b Fix unaligned reads in read cache
8ad0fcdf9 Separate small subset tests in DBTest
6c951c43c Run fewer tests in OSX
f7997f134 add direct I/O to version notes 5.2.0
3b8ba703c Fix flaky DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest2
96c7e1504 Fix Java build
e0b87afc7 Black list some slow valgrind tests
e67232cff Handle failed Finish() in SST file writer
8efb5ffa2 [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp…
1ba2804b7 Remove XFunc tests
e7d902e69 add direct_io and compaction_readahead_size in db_stress
1ef5f50e8 detect logical sector size
ed50308d2 check backup directory exists before listing children
4d7c06ced Make WriteBatchWithIndex moveble
5040414e6 Gracefully handle previous backup interrupted
f206af56f add use_direct_io() to ReadaheadRandomAccessFile
082493442 truncate patch
286a36db7 posix writablefile truncate
f0879e4c3 Page size isn't always 4k on linux
18eeb7b90 Fix interference between max_total_wal_size and db_write_buffer_size checks
1560b2f5f Temporarly return deprecated functions to fix MongoRocks build
2a0f3d0de level compaction expansion
ebc8a7998 alignment is on in ReadaheadRandomAccessFile::Read()
381fd3224 Remove timeout_hint_us from WriteOptions
fce7a6e19 Fail IngestExternalFile when bg_error_ exists
a618a16f4 New subcode for IOError to detect the ESTALE errno
7ab005183 Remove deprecated DB::AddFile and DB::CompactRange
401667366 Adding Dlang to the list
756c5924e Allow adding external v1 sst file with no global seqno support
aa0298fa9 Remove trailing whitespace from examples Makefile
db2b4eb50 avoid direct io in rocksdb_lite
43e9f01c2 Fix repair_test on ROCKSDB_LITE
7106a994f Use monotonic time points in write_controller.cc and rate_limiter.cc
c2247dc1c Make DBImpl::has_unpersisted_data_ atomic
eb912a927 Remove disableDataSync option
0227c16d6 Update static library versions and add checksums
b3aae4d07 Add repair_test to make check
421ce7c2b Add support for JNI Library on Linux on PowerPC.
9afa20cf2 Increase build version and HISTORY.md for releasing 5.2
a5adda064 Fix repair issues
b48e4778b Consolidate file cutting logic in compaction loop
ac2a77a74 Announce the experimetnal two-level index feature in HISTORY.md
c4a37dcb4 Print the missed last layer in cfstats
a12818afc Blog post for 5.1.2 release
3b4ac8076 Clarify ldb column family argument
d70ce7ee0 Move db_bench flags out of unnamed namespace
186c7eedb Remove incorrect statistics warning
53bb01516 [rocksdb][PR] compaction_style and compaction_pri should output their value as a st…
69d5262c8 Two-level Indexes
0a4cdde50 Windows thread
1aaa898cf Adding GetApproximateMemTableStats method
9fc23c55f Use gcc-4.9-glibc-2.20-fb python in precommit_checker
b797e4215 Dump compression dictionary meta-block
036d668b1 Fix wrong result in data race case related to Get()
574b543f8 Rename merger.h -> merging_iterator.h
add8b50cc Move ThreadLocal implementation into .cc
71d2496af Fix arc setting for Facebook internal tools
f289d9f4a Fix OSX build break after the fallocate change
4a3e7d320 Change the default of delayed slowdown value to 16MB/s
0513e21f9 RangeSync() should work with ROCKSDB_FALLOCATE_PRESENT not set
8b369ae5b Cleaner default options using C++11 in-class init
ec79a7b53 Dedup code in option.cc and db_options.cc
b96372dea improving the C wrapper
04c4ec41d Change corruption_test to use 4 bits.
2d75cd40d NewLRUCache() to pick number of shard bits based on capacity if not given
f25f1ec60 Add test DBTest2.GetRaceFlush which can expose a data race bug
37d4a79e9 Deserialize custom Statistics object in db_bench
3b35134e4 Avoid cache lookups for range deletion meta-block
94a0c32e7 Fix LRU Ref() for handles with external references only
17c118060 Generalize Env registration framework
07dddd5f7 EnvPosixTestWithParam should wait for all threads to finish
5dad9d6d2 Avoid logs_ operation out of DB mutex
a7b13919b Fix CompactFiles() bug when used with CompactionFilter using SuperVersion
616a1464e Fix DeleteRange including sentinels in output files
c918c4b76 Update USERS.md add user Pika
03ca2ac8a Remove function from DBImpl that are not used anywhere
b0029bc7f Test merge op covered by range deletion in memtable
d438e1ec1 Test range deletion block outlives table reader
fba726e55 Version librocksdb.so
9da4d542f Range deletions unsupported in tailing iterator
f2b4939da fixed typo
973f1b78f memtable: delete merge value for range deleteion
aebfd1703 fix non-portable behavior in encoder
753ff84a3 Fix get approx size
d7ea44f2f Fixup a couple of builds errors on Linux.
537da370d c: allow set savepoint to writebatch
af6ec4d78 fix batchresult handle leak
e29bb934f Zlib 1.2.8 is no longer available, switched to 1.2.10
5ac97314e Fix std::out_of_range when DBOptions::keep_log_file_num is zero
4e35ffdfa cmake: check -momit-leaf-frame-pointer before using it
3c0852d1d Make ingest external file backward compatible
0e8dfd606 Fix OptimizeForPointLookup()
e840213d6 Change DB::GetApproximateSizes for more flexibility needed for MyRocks
9239103cd Flush job should release reference current version if sync log failed
da54d36a9 Disable IngestExternalFile in ReadOnly mode
5cf176ca1 Fix for 2PC causing WAL to grow too large
4a73bb0b4 Split travis jobs
c70d3c7ad Enable DBTest.GroupCommit as it runs in a reasonlable time now.
602c13a96 Remove fadvise with direct IO read
f9d18e22d Fix DeleteRange file boundary correctness issue with max_compaction_bytes
3ce091fd7 Add KEEP_DB env var option
77b480662 Fix 2PC with concurrent memtable insert
e8a096000 util/thread_local.h: silence a clang-build warning
324a0f988 Follow up for DirectIO refactor
bc5d7b702 travis: For linux, do all tests under gcc
3e6899d11 change UseDirectIO() to use_direct_io()
d4e07a845 fix warning of unused direct io helper functions
dc2584eea direct reads refactor
d18dd2c41 Abort compactions more reliably when closing DB
62384ebe9 Guarding extra fallocate call with TRAVIS because its not working pro…
9f246298e Performance: Iterate vector by reference
fe395fb63 Allow incrementing refcount on cache handles
2172b660e Fix build on FreeBSD
3c233ca4e Fix Windows environment issues
763173456 Fix the error in ColumnFamiliesTest
7a02ad070 Update travis to ubuntu trusty
60c509ff1 Fix valgrind failure in test CurrentFileModifiedWhileCheckpointing2PC
d0ba8ec8f Revert "PinnableSlice"
54d94e9c2 PinnableSlice
e04480fae Fix MS warnings. Use ROCKSDB_Prsz for size_t.
c081f7215 5.0.1 release blog post
ac73d7558 Add GetSupportedCompressions() convenience function
b104b8781 Maintain position in range deletions map
640d72480 Update db_bench and sst_dump to test with block cache mid-point inser?
653ac1f9c C API: support total_order_mode
85ac1a320 Fix rocksdb::Status::getState
76711b6e7 Make ExternalSSTFileTest::CompactionDeadlock more deterministic
c963460db Fix tests under GCC_481
33c86d677 Fix backupable db test
e425ec116 utilities/backupable: backup should limit the copy size of wal.
0712d541d Delegate Cleanables
d58ef52ba Allow SstFileWriter to Fadvise the file away from page cache
17a4b75cc Always fsync the file after file copying
2fb70dc79 examples: Add options_file_example to target all
a738af8f8 db/pinned_iterators_manager.h: bugfix
906523d98 Add description to the 2PC checkpooint bug in HISTORY.md
438f22bc5 Fix bug of Checkpoint loses recent transactions with 2PC
335981d47 Fix the directory path for RocksDB repo
548b62805 Enable conditionally using adaptive mutexes
4e07b08ef include/rocksdb/utilities/env_librados: fix typo
ab48c165a Print cache options to info log
972f96b3f direct io write support
989e644ed Remove sst_file_manager option from LITE
1beef6569 Fix c_test
3d692822f persistent_cache: fix two timer
046099c9b The array is malloced by backtrace_symbols(), and must be freed
6ff2c8d7f Remove gflags as travis build dependency
3cd9ed1c3 Show sandcastle URL in phabricator
50e305de9 Collapse range deletions
5d1457dbb Dump persistent cache options
7bd725e96 db_bench: introduce --benchmark_read_rate_limit
296691847 Update Netflix section of USERS.md
342370f1d Simplify MemTable::Update
1a136c1f1 Expose file size
fbff4628a Reduce compaction iterator status checks
bd6cf7b51 WritableFileWriter: default buffer size equal min(64k,options.writabl?
fc0c6fd98 "make format" format diff since last commit from master
816c1e30c gcc-7 requires include <functional> for std::function
c27073586 Iterator should be in corrupted status if merge operator return false
a8bf4d63f Make format shows wrong curl command for retrieving clang-format-diff.py
8f5d24ae6 C API: support get usage and pinned_usage for cache
0ab6fc167 Gcc-7 buffer size insufficient
b7239bf7e Gcc 7 fallthrough
477b6ea57 std::remove_if requires <algorithm>
83f9a6fd2 Fail BackupEngine::Open upon meta-file read error
a79eae4b0 Add pcache documentation images
d71e728c7 Print user collected properties in sst_dump
7004a6f7b Add missing copyright header
3cdfaeca3 Fixes for MSVC compilation
e097222e6 util/logging.cc: buffer of insufficient size (gcc-7 -Werror=format-length)
cfc34d7c4 Missing break in case in DBTestBase::CurrentOptions
bfbcec233 Gcc 7 error expansion to defined
6653e32ac build: make it easier to pass PORTABLE
67adc937b intentional fallthough (prevents gcc-7/clang-4 error)
1a146f89c break Flush wait for dropped CF
c3e5ee715 util/histogram.cc: HistogramStat::toString buffer insufficient
5334d8b44 table/block_based_table_builder.cc: intentional fallthrough - comment to match gcc pattern
36d42e65d Disable test to unblock travis build
b57dd9262 C API: support writebatch delete range
2ba59b5a1 Disallow ingesting files into dropped CFs
1f6f7e3e8 cast to signed char in ldb_cmd_test for ppc64le
243975d5d More accurate error status for BackupEngine::Open
f0c509e2c Return finer-granularity status from Env::GetChildren*
dc64f46b1 Add db_bench option for stderr logging
2cabdb8f4 Increase buffer size
4a17b47bb Remove unnecessary header include
8c2b921fd Fixed a crash in debug build in flush_job.cc
20ce081fa Fix issue where IngestExternalFile insert blocks in block cache with g_seqno=0
5241e0dbf fix db_bench argument type
c04f6a0b4 Specify shell in makefile
45c7ce137 CompactRangeOptions C API
2c2ba6824 db_stress support for range deletions
b821984d3 DeleteRange read path end-to-end tests
2f4fc539c Compaction::IsTrivialMove relaxing
1dce75b2e Update USERS.md
304b3c706 Update USERS.md
fa50fffaf Option to expand range tombstones in db_bench
c26a4d8e8 Fix compile error in trasaction_lock_mgr.cc
ed8fbdb56 Add EventListener::OnExternalFileIngested() event
2005c88a7 Implement non-exclusive locks
0b0f23572 Mention IngestExternalFile changes in HISTORY.md
23db48e8d Update HISTORY.md for 5.0 branch
beb36d9c1 Fixed CompactionFilter::Decision::kRemoveAndSkipUntil
67f37cf19 Allow user to specify a CF for SST files generated by SstFileWriter
9053fe2a5 Made delete_obsolete_files_period_micros option dynamic
edde954e7 fix clang build
56281f3a9 Add memtable_insert_with_hint_prefix_size option to db_bench
4a21b1402 Cache heap::downheap() root comparison (optimize heap cmp call)
e39d08087 Fix travis (compile for clang < 3.9)
3f407b065 Kill flashcache code in RocksDB
b77007df8 Bug: paralle_group status updated in WriteThread::CompleteParallelWorker
247d0979a Support for range skips in compaction filter
96fcefbf1 c api: expose option for dynamic level size target
00197cff3 Add C API to set base_backgroud_compactions
5b219eccb deleterange end-to-end test improvements for lite/robustness
aad119176 pass rocksdb oncall to mysql_mtr_filter otherwise tasks get created w…
e33352899 DeleteRange write path end-to-end tests
7784980fc Fix mis-reporting of compaction read bytes to the base level
3c6b49ed6 Fix implicit conversion between int64_t to int
b3b875657 Remove unused assignment in db/db_iter.cc
4f6e89b1d Fix range deletion covering key in same SST file
a2bf265a3 Avoid intentional overflow in GetL0ThresholdSpeedupCompaction
52fd1ff2c disable UBSAN for functions with intentional -ve shift / overflow
1886c435b Fix CompactionJob::Install division by zero
63c30de80 fix options_test ubsan
13e66a8f5 Fix compaction_job.cc division by zero
01eabf737 Fix double-counted deletion stat
7ffb10fc1 DeleteRange compaction statistics
236d4c67e Less linear search in DBIter::Seek() when keys are overwritten a lot
cd7c4143d Improve Write Stalling System
dfb6fe675 Unified InlineSkipList::Insert algorithm with hinting
3068870cc Making persistent cache more resilient to filesystem failures
734e4acaf Eliminate redundant cache lookup with range deletion
182b940e7 Add WriteOptions.no_slowdown
4118e1333 Persistent Cache: Expose stats to user via public API
f2a8f92a1 rocks_lua_compaction_filter: add unused attribute to a variable
4444256ab Remove use of deprecated LZ4 function
548d7fb26 Fix fd leak when using direct IOs
fd43ee09d Range deletion microoptimizations
23a18ca5a Reword support a little bit to more clear and concise
481856ac4 Update support to separate code issues with general questions
a0deec960 Fix deadlock when calling getMergedHistogram
fe349db57 Remove Arena in RangeDelAggregator
e63350e72 Use more efficient hash map for deadlock detection
a13bde39e Skip ldb test in Travis
73843aa63 Direct I/O Reads Handle the last sector correctly.
9d60151b0 Implement PositionedAppend for PosixWritableFile
3f6221521 Lazily initialize RangeDelAggregator's map and pinning manager
41e77b839 cmake: s/STEQUAL/STREQUAL/
c1038d283 Release RocksDB 5.0
635a7bd1a refactor TableCache Get/NewIterator for single exit points
f39452e81 Fix heap use after free ASAN/Valgrind
a4eb7387b Allow plain table to store index on file with bloom filter disabled
36e4762ce Remove Ticker::SEQUENCE_NUMBER
86eb2b9ad Fix src.mk
0765babe1 Remove LATEST_BACKUP file
647eafdc2 Introduce Lua Extension: RocksLuaCompactionFilter
760ef68a6 fix deleterange asan issue
327085b7b fix valgrind
715591bba Ask travis to use JDK 7
972e3ff29 Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default
420bdb42e option_change_migration_test: force full compaction when needed
1543d5d92 Report memory usage by memtable insert hints map.
018bb2ebf DeleteRange support for db_bench
dc51bd716 CMakeLists.txt: FreeBSD has jemalloc as default malloc
48e8baebc Decouple data iterator and range deletion iterator in TableCache
4b0aa3c4c Fix failed compaction_filter_example and add it into make all
53b693f5f ldb support for range delete
661e4c926 DeleteRange unsupported in non-block-based tables
489d14280 DeleteRange interface
eba99c28e Fix min_write_buffer_number_to_merge = 0 bug
2ef92fea5 Remove all instances of relative_url until GitHub pages problem is fixed.
91300d01f Dynamic max_total_wal_size option
ec2f64794 Consider subcompaction boundaries when updating file boundaries for range deletion
800e51553 Fix CSS issues again
b952c898b Parallize persistent_cache_test and transaction_test
3b192f618 Handle full final subcompaction output file with range deletions
6c5795200 Make range deletion inclusive-exclusive
425210cc4 CSS issues are arising on the Github Pages side. Temp fix.
1ea79a78c Optimize sequential insert into memtable - Part 1: Interface
df5eeb85c Optimize sequential insert into memtable - Part 2: Implementation
5ed650857 Fix SstFileWriter destructor
adb665e0b Allowed delayed_write_rate option to be dynamically set.
307a4e80c sst_dump support for range deletion
361010d44 Exporting compaction stats in the form of a map
672300f47 Use relative Urls for stylesheets
b39b2ee12 do not call get() in recovery mode
1ca5f6d13 Fix 2PC Recovery SeqId Miscount
e095d0cbc Rocksdb contruns to new Sandcastle API
14c0380e7 Convenience option to parse an internal key on command line
c90fef88b fix open failure with empty wal
4e20c5da2 Store internal keys in TombstoneMap
a9fb346e4 Fix RocksDB Lite build failure in c_test.cc
d133b08f6 Use correct sequence number when creating memtable
144cdb8f1 16384 as e.g .value for compression_max_dict_bytes
9bd191d2f Fix deadlock between (WriterThread/Compaction/IngestExternalFile)
a9fae0a9d CSS problems again :(
193221e0a Fix Forward Iterator Seek()/SeekToFirst()
e48f3f8b9 remove tabs and duplicate #include in c api
85bd8f518 Minor fix to GFLAGS usage in persistent cache
a7875272d c: support seek_for_prev
0f17f9279 Make the header links a bit more flexible
cf19f559d single quotes in feed
2dc019e09 Fix header links
f1aedda06 More Jekyll 3.3 fixes
c54cdc378 More Jekyll 3.3 updates
2bcaf8246 Update product and feature template for Jekyll 3.3
24bceb096 Java API - Implement GetFromBatch and GetFromBatchAndDB in WBWI
815f54afa Insert range deletion meta-block into block cache
9e7cf3469 DeleteRange user iterator support
5c5d01ae7 Fix wrong comment (Maximum supported block size)
f998c9790 DeleteRange Get support
879f36636 Add C api for RateLimiter
557034f36 Remove all instances of baseurl
437942e48 Add avoid_flush_during_shutdown DB option
2b16d664c Change max_bytes_for_level_multiplier to double
16fb04434 expose IngestExternalFile to c abi
ce22ea99a Fix casts for MSVC
196af035c Introduce FAIL_ON_WARNINGS CMake variable (default ON)
40a2e406f DeleteRange flush support
d5555d95a Fix MSVC compile error in 32 bit compilation
da61f348d Print compression and Fast CRC support info as Header level
f9eb56791 db_bench: --dump_malloc_stats takes no effect
6a4faee5c fix freebsd build include path err and so & jar file name
c90c48d3c Show More DB Stats in info logs
1b295ac8a DBTest.GetThreadStatus: Wait for test results for longer
25f5742f0 Update documentation to point at gcc 4.8
b50a81a2b Add a test for tailing_iterator
04751d534 L0 compression should follow options.compression_per_level if not empty
2946cadc4 Improve RangeDelAggregator documentation
0a9fd05c2 Update Vagrant file (test internal phabricator workflow)
fcd1e0bf6 Make rocksdb work with internal repo
0aab5e55f FreeBSD: malloc_usable_size is in <malloc_np.h> (#1428)
9c0bb7f17 cmake: drop "-march=native" from CXX_FLAGS (#1429)
eeb27e1bb Add handy option to turn on direct I/O in db_bench (#1424)
c6168d13a removed some declarations from c.h which resulted in undefined symbols (#1407)
bc429de49 revert fractional cascading in farward iterator
b9bc7a2aa Use skiplist rep for range tombstone memtable
60a2bbba9 Makefile: generate util/build_version.cc from .in file (#1384)
9ee84067f Disable DBTest.RepeatedWritesToSameKey (#1420)
f41df3045 OptionChangeMigration() to support FIFO compaction
2e8004e60 Changing the legocastle run to use valgrind_test instead of _check
9de2f7521 revert Prev() in MergingIterator to use previous code in non-prefix-seek mode
24495186d DBSSTTest.RateLimitedDelete: not to use real clock
1168cb810 Fix a bug that may cause a deleted row to appear again
99c052a34 Fix integer overflow in GetL0ThresholdSpeedupCompaction (#1378)
f83cd64c0 Fix a bug that mistakenly disable regression_test.sh to update commit (#1415)
0e926b84f Passing DISABLE_JEMALLOC=1 to valgrind_check if run locally
4dfaa6610 Make IsDeadlockDetect() virtual member of Transaction
59a7c0337 Change ioptions to store user_comparator, fix bug
869ae5d78 Support IngestExternalFile (remove AddFile restrictions)
1d9dbef64 Restrict running condition of UniversalCompactionTrivialMoveTest2
4edd39fda Implement deadlock detection
48fd619a4 Minor fixes to RocksJava Native Library initialization (#1287)
48e4e842b Disable auto compactions in memory_test and re-enable the test (#1408)
fb2e41294 column_family_test: disable some tests in LITE
5af651db2 fix data race in compact_files_test
a0ba0aa87 Fix uninitialized variable gcc error for MyRocks
b88f8e87c Support SST files with Global sequence numbers [reland]
08616b493 [db_bench] add filldeterministic (Universal+level compaction)
52c9808c3 not split file in compaciton on level 0
5e0d6b4cc fix db_stress assertion failure
ab5399837 Bump RocksDB version to 4.13 (#1405)
b4d07123c SamePrefixTest.InDomainTest to clear the test directory before testing
aa09d0338 Avoid calling GetDBOptions() inside GetFromBatchAndDB()
6fbe96baf Compaction Support for Range Deletion
257de78d9 Remove "-Xcheck:jni" from Java tests (#1402)
d88dff4ef add seeforprev in history
5027dd17a Fix a minor bug in the ldb tool that was not selecting the specified (#1399)
fea6fdd67 Fix @see in two Java functions (#1396)
b1031d6c1 Remove function local statics that interfere with memory pooling (#1392)
f47054015 Handle WAL deletion when using avoid_flush_during_recovery
e29d3b67c Make max_background_compactions and base_background_compactions dynamic changeable
21e8daced fix assertion failure in Prev()
b9311aa65 Implement WinRandomRW file and improve code reuse (#1388)
a249a0b75 check_format_compatible.sh to use some branch which allows to run with GCC 4.8 (#1393)
040328a30 Remove an assertion for single-delete in MergeHelper::MergeUntil
8cbe3e10c Relax the acceptable bias RateLimiterTest::Rate test be 25%
f26a139d8 Log successful AddFile
5691a1d8a Fix compaction conflict with running compaction
017de666c fixup commit
1b7af5fb1 Redo handling of recycled logs in full purge
27bfe327b Editorial change to README.md
89cc404de A bit of doc restructuring
9e7fda829 Fix arcanist
2e4b5cab0 Add missing RateLimiter class to the Windows build (#1382)
ce4963fdf [doc] Document that Visual Studio 2015+ is now required for Windows builds (#1389)
e48927098 Fix scoped arena iterator (#1387)
f8d8cf53f Fix log_write_bench -bytes_per_sync option. (#1375)
02b3e3985 Make txn->GetState() const
447f17127 new Prev() prefix support using SeekForPrev()
991b585ee More block cache tickers
d6ae6dec6 Add Statistics::getAndResetTickerCount().
aea3ce4c8 Avoid string CONCAT which is not supported in cmake 2.6 (#1383)
2ad68b971 Support running consistency checks in release mode
67501cfc9 Fix -ve std::string::resize
04b02dd12 Testing asset links after config change
8c55bb87c Make Lock Info test multiple column families
d06232897 Revert "Support SST files with Global sequence numbers"
5cd28833a [RocksJava] Adjusted RateLimiter to 3.10.0 (#1368)
37737c3a6 Expose Transaction State Publicly
2c1f95291 Add facility to write only a portion of WriteBatch to WAL
043cb62d6 Fix record_size in log_write_bench, swap args to std::string::assign. (#1373)
4985f60fc env_mirror: fix a few leaks (#1363)
5aded67dd update of c.h (#1371)
912aec13c "Recent Posts" -> "All Posts"
7cbb298db Make sure that when contribtuing we call out creating appropriate directories
a06ad4711 Add top level doc information to CONTRIBUTING.md
3fdd5b971 A little more generic CONTRIBUTING.md
ed4fc31db Add link to CONTRIBUTING.md to main docs README.md
e4922e181 Forgot to truncate one blog post
6d8cd7ede Add CONTRIBUTING.md for rocksdb.org contribution guidance
bd55e5a1e Fix some formatting of compaction blog post
0f60358b0 CRLF -> LF mod (including removing trailing whitespace for those files)
b90e29c90 Truncate posts on the main /blog/ page
0d7acadaf Add author fields to blog posts
01be44181 Add GitHub link to the landing page header
9d6c96138 Fix Mac build
ab01da543 Support SST files with Global sequence numbers
d346ba246 Minor fixes around Windows 64 Java Artifacts (#1366)
e91b4d0cf Add factory method for creating persistent cache that is accessible from public
be1f1092c Expose transaction id, lock state information and transaction wait information
6009c473c Store range tombstones in memtable
3c21c64c7 Use size hint for HashMap in multiGet. Similar to https://github.com/facebook/rocksdb/pull/1344 (#1367)
13f7a01f6 Fixing JNI release build for gcc (#975)
7260662b3 Add Java API for SstFileWriter
26388247a delete unused variable for PrevInterval()
87dfc1d23 Fix conflict between AddFile() and CompactRange()
eb44ed655 Update 2016-09-28-rocksdb-4-11-2-released.markdown
e4437610d Update 2016-09-28-rocksdb-4-11-2-released.markdown
501f05108 Update 2016-09-28-rocksdb-4-11-2-released.markdown
dec9009f8 Update 2016-09-28-rocksdb-4-11-2-released.markdown
4ed69dd0b Create 2016-09-28-rocksdb-4-11-2-released.markdown
21f4bb5a8 cmake support for linux and osx (#1358)
4defe306f fix typo in comments (#1360)
f517d9dd0 Add SeekForPrev() to Iterator
eb3894cf4 Recompute compaction score on SetOptions (#1346)
5c64fb67d Fix AddFile() conflict with compaction output [WaitForAddFile()]
9e9f5a0b9 Fix CompactFilesTest.ObsoleteFiles timeout (#1353)
c2a62a4cb not cut compaction output when compact to level 0
9ed928e7a Split DBOptions into ImmutableDBOptions and MutableDBOptions
4bc8c88e6 Recover same sequence id from WAL (#1350)
0a1bd9c50 add cfh deletion started listener
da5a9a65c Fix mac build
d45eb6c6d Fix typo (#1349)
abc0ae462 Add AddFile() InternalStats for Total files/L0 files/total keys ingested
715256338 forbid merge during recovery
5735b3dc2 Fix compiling under -Werror=missing-field-initializers
654ed9a28 loose the assertion condition of rate_limiter_test
e4d3f5d9b Fix DBImpl::GetWalPreallocateBlockSize Mac build error
7afbb7420 solve the problem of table_factory_to_write_=nullptr (#1342)
d78a4401b DBImpl::GetWalPreallocateBlockSize() should return size_t
42ac9c5f1 Retry getting arcanist token on failure
b666f8544 Consider more factors when determining preallocation size of WAL files
4c3f4496b Add TableBuilderOptions::level and relevant changes (#1335)
3edb9461b Avoid hard-coded sleep in EnvPosixTestWithParam.TwoPools
0a88f38b7 Remove ColumnFamilyData::options()
41a9070f8 Fix java makefile dependencies
8d9bf5c49 Fix DBOptionsTest.GetLatestOptions
40cfa3e02 Fix DBWALTest.RecoveryWithLogDataForSomeCFs with mac
06b4785fe Fix recovery for WALs without data for all CFs
d7242ff4d Fix GetSortedWalFiles when log recycling enabled
17f76fc56 DB::GetOptions() reflect dynamic changed options
215d12826 Fix typo (#903)
a958c2643 Rename jvalue to jval in rocksjni
0a165bd7d Have Facebook link point to RocksDB on FB
3639f3288 Fix bug in UnScSigned-off-by: xh931076284 <931076284@qq.com> (#1336)
8e061f974 Refactor GetMutableOptionsFromStrings
81747f1be Refactor MutableCFOptions
ba65c816b Support POSIX RandomRWFile
1d980a8e3 Create CNAME
2adab1dde Add API links to the header bar
a182b2981 Preserve blog comments in markdown
f54de9230 Adding Dgraph to list of Users (#1291)
9e4aa798c Summary: (#1313)
a10e8a056 Fix C api memtable rep bugs. (#1328)
eb1d4d53c Release RocksDB 4.12
22d88e24d Allow an offset as well as a length to be specified for byte[] operations in RocksJava JNI (#1264)
b06b19136 add C api for set wal_recovery_mode (#1327)
1cca09129 Temporarily revert Prev() prefix support
de28a2553 Update HISTORY.md for thread-local stats
0fcb6dbed Remove extraneous function prototypes from c.h (#1326)
52ee07b02 Move AddFile() tests to external_sst_file_test.cc
66a91e260 Add NoSpace subcode to IOError (#1320)
67036c040 Fix Flaky ColumnFamilyTest.FlushCloseWALFiles
0e2da497c fix typo in option.h's comment (#1321)
6d61358a0 Add real Google Analytics ID
2d9d36ea4 Have "Edit on GitHub" point to master instead of gh-pages
937751898 Update landing page content
1ec75ee76 Add redirects from old blog posts link to new format
607628d34 Support ZSTD with finalized format
ce1be2ce3 Fix build error on Windows (AppVeyor) (#1315)
f7669b40b Fix Windows Build
22696b088 Fix uninitlized CompactionJob::SubcompactionState::current_output_file_size
c1865e0f7 Trigger more tests per diff
a88677d2c Remove ImmutableCFOptions from public API
80c75593e Fix data race in AddFile() with multiple files + custom comparator bug
5051755e3 Fix db_bench memory use after free (detected by clang_analyze)
4fd08f4b8 Ensure Correct Behavior of StatsLevel kExceptDetailedTimers and kExceptTimeForMutex (#1308)
e14fbaae2 Add FAQ based on the front page of the current rocksdb.org
3c2262400 Migrate the RocksDB Worpdress blog over to Jekyll
ee0e2201e Transfer the current Getting Started contents to GitHub Pages
5a0e9a4cf Initial Landing Page
9447a8540 Remove the `doc` directory
32149059f Merge options source_compaction_factor, max_grandparent_overlap_bytes and expanded_compaction_factor into max_compaction_bytes
4590b53a4 add stats to Cache::LookUp()
85bb30825 Expose Utility function StringToMap() (#1306)
8ce1b8440 Fix Travis on Mac
380e651af Fix Mac build failure (#1309)
1613fa949 Thread-specific histogram statistics
6a14d55bd add prefix_seek_mode to db_iter_test
de47e2bd4 Fix ClockCache memory leak
f099af4c7 Fix travis
db74b1a21 fix bug in merge_iterator when data race happens
b18f9c9ea add nullptr check to internal_prefix_transform
4e395e875 Update docs README.md
2482d5fb4 support Prev() in prefix seek mode
7541c7a79 Fix cache_test valgrind_check failure
c8513cde0 Update the download location of Snappy (#1304)
b49b92cf2 Introduce Read amplification bitmap (read amp statistics)
c7004840d store prefix_extractor_name in table
4ad928e17 add comment to SimCache to estimate actual capacity
e9b2af87f Expose ThreadPool under include/rocksdb/threadpool.h
23a057007 Document memtable flush behavior in CancelAllBackgroundWork()
dade61ac2 Mitigate regression bug of options.max_successive_merges hit during DB Recovery
cce702a6e [db_bench] Support single benchmark arguments (Repeat for X times, Warm up for X times), Support CombinedStats (AVG / MEDIAN)
3586901f8 cat tests logs sorted by exit code
b2ce59537 Persist data during user initiated shutdown
4b3438d2d Fix parallel valgrind (valgrind_check)
a081f798b Relax consistency for thread-local ticker stats
b10d65c2a Update and slightly clarify instructions in build_detect_platform (#1301)
f85f99bf6 Fix the Windows build of RocksDB Java. Similar to https://github.com/facebook/rocksdb/issues/1220 (#1284)
7b8109517 Fix a crash when compaction fails to open a file
7c9586837 Thread-specific ticker statistics
ea9e0757f Add initial GitHub pages infra for RocksDB documentation move and update. (#1294)
2a9c97108 [Flaky Test] Disable DBPropertiesTest.GetProperty
d76ddf327 Disable ClockCache db_crashtest
cec2c6436 fix data race in NewIndexIterator() in block_based_table_reader.cc
badbff65b Not insert into block cache if cache is full and not holding handle
4a16c32ec Option to cache index/filter blocks with priority
99c4af716 Make ClockCache available with TSAN build
f57bc1d03 Fix lambda expression for clang/windows
5440675c3 Fix lambda capture expression for windows
6584cec8f Fold function for thread-local data
817eeb29b Add singleDelete to RocksJava (#1275)
ffdf6eee1 Add Status to RocksDBException so that meaningful function result Status from the C++ API isn't lost (#1273)
ecf900386 Fix bug in printing values for block-based table
72f8cc703 LRU cache mid-point insertion
6a17b07ca Add TablePropertiesCollector support in SstFileWriter
78837f5d6 TableBuilder / TableReader support for range deletion
4cc37f59e Introduce ClockCache
ff17a2abf Adding TBB as dependency.
49d88be02 c abi: allow compaction filter ignore snapshot (#1268)
0b63f51fb fixes 1215: execute_process(COMMAND mkdir ${DIR}) fails to create a directory with cmake on Windows (#1219)
3981345be Small nits (#1280)
2a2ebb6f5 Move LRUCache structs to lru_cache.h header
2fc2fd92a Single Delete Mismatch and Fallthrough statistics
3771e3797 WriteBatch support for range deletion
236756f2c Make SyncPoint return immediately when disabled
64a0082c6 Fix DBSSTest::AddExternalSstFileSkipSnapshot valgrind fail
dd7a748cf Fix java build
4fe12baa6 Make db_bench less space for --stats_per_interval
6525ce4ca Compaction stats printing: "batch" => "commit group"
a117891b4 Fixed typo (#1279)
b248e98cf Fix a destruction order issue in ThreadStatusUpdater
deda159b5 Added min/max/avg data block size output to sst_dump
e408e98c8 add Name() to Cache
a297643f2 Fix valgrind memory leak
d11c09d9e Eliminate memcpy from ForwardIterator
d36755502 Added further Java API options for controlling concurrent writes
ebdfe34cc Exposed further Java API options for controlling compaction
d1be59463 Improve documentation of SliceTransform.
6056d6317 Improve comment and bug fix for GetOptionsFromMap functions in convenience.h
76a67cf74 support stackableDB as the baseDB of transactionDB
67c1ae883 Travis build break fix
b693ba68b Minor PinnedIteratorsManager Refactoring
db3dfb164 Fixes for arcanist config (#1271)
87c91bd87 Persistent Read Cache (8) Benchmark tooling
2914de64e add sim_cache stats to Statistics
8b79422b5 [Proof-Of-Concept] RocksDB Blob Storage with a blob log file.
4beffe001 Fix test data race in two FaultInjectionTest tests
821bcb0b3 util/arena.cc: FreeBSD: More portable use of mmap(MAP_ANON) (#1254)
5370f44a8 Increase RocksDB version
56dd03411 read_options.background_purge_on_iterator_cleanup to cover forward iterator and log file closing too.
ccecf3f4f UniversalCompaction should ignore sorted runs being compacted (when compacting for file num)
1b0069ce2 Remove non-gtest from parallelized tests
638c49f24 Change HISTORY.md for release 4.11
6b8e9c68b fix vs generator (#1269)
c38b075e7 Update HISTORY.md
8f399e3fe Update HISTORY.md
98d0b78ea Added check_snapshot option in the DB's AddFile function (#1261)
9fd68b7fb set travis open file descriptor limit
59ddb5059 Fix travis build break
f4d986364 Added SetOptions support to RocksJava (#1243)
7882cb977 Make DBOptionsTest::EnableAutoCompactionAndTriggerStall less falky
44f5cc57a Add time series database (resubmitted)
7c4615cf1 A utility function to help users migrate DB after options change
5bb0a7f73 Update appveyor.yml
86396cc18 Update appveyor.yml
c1db098dc Update appveyor.yml
7da2eaf0d Update appveyor.yml
34723b4c4 Cleanup unused variable pending_fsync_.
7cc0dbd66 cat all logs in sandcastle output
9253767a6 Correct geHistogramData() -> getHistogramData() (#1257)
f35b16f24 db_bench add an option of --base_background_compactions
c3a4bea5d Fix flaky test `ObsoleteFiles`
8234faabf Fix failed test
4990c0d1a Remove deprecated LEVELDB_PLATFORM_POSIX
7323e4c8a Fix clang on macOS
ee027fc19 Ignore write stall triggers when auto-compaction is disabled
e4609a749 Fix Windows build issues (#1253)
2306167d3 Fix clang build failure and refactor unit test
343304e1d Use StopWatch to do statistic job in db_impl_add_file.cc
cdc4eb689 Add a GetComparator() function to the ColumnFamilyHandle base class so that the user's comparator can be retrieved.
712dd27e6 Build break fixes
0155c73de Fix parallel tests `make check -j`
c49ea68c5 Fix to enable running CI jobs locally
726c2f7e5 Build break fix
d51dc96a7 Experiments on column-aware encodings
c116b4780 Persistent Read Cache (part 6) Block Cache Tier Implementation
64046e581 Write a benchmark to emulate time series data
9ae92f50b More granular steps in the Makefile, can help with running all or single Java tests (and with ASAN build - https://github.com/facebook/rocksdb/wiki/JNI-Debugging) (#1237)
7c01d6534 [Fix Java] Remove duplicate cases in LoggerJniCallback::Logv
8796934af Added missing Java ReadOptions settings (#1109)
5e2c79658 Make DBTest.CompressionStatsTest more deterministic
557748ff7 Fix db_stress failure (pass merge_operator even if not used)
811ee2111 Bugfix to ensure that logging can be achieved from threads that are not known to the JVM (#1106)
afad5bd1c Simplify thread-local static initialization
6920cde89 Remove an extra apostrophe
e72ea485e add InDomain regression test
9c8ac144b Avoid duplicate task creation for RocksDB contruns
d4c45428a db_stress shouldn't assert file size 0 if file creation fails
d3bfd3397 Testing out parallel sandcastle changes
7efd9c25c Increse timeout in some tests
50b8d29b9 fixes 1230: Error:string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command (#1231)
5c858ddd2 fix errata in libnuma test (#1244)
e5b5f12b8 Change options memtable_prefix_bloom_huge_page_tlb_size => memtable_huge_page_size and cover huge page to memtable too
0ce258f9b Compaction picker to expand output level files for keys cross files' boundary too.
ac0d93b08 fixes 1217: rocksdbjni javac and javah execute_processes fail on windows (#1218)
1ae46094d Appveyor badge to show master branch
8745f013f [Fix java build] Stop using non standard std::make_unique
e12270dfe fix previous typo
bbd6a5a18 ldb restore subcommand
9498069fc Run error-filtering script on diff-triggered tests
f8061a237 Fix Statistics TickersNameMap miss match with Tickers enum
16e225f70 Fix MergeContext::copied_operands_ strings moving
a4955b39a Run sandcastle tests in /dev/shm
ae0ad719d Fix flaky DBSSTTEST::DeleteObsoleteFilesPendingOutputs
b2a8016df Update db_bench_tool.cc (#1239)
c6654588b Disable two dynamic options tests under lite build
2a6d0cde7 Ignore stale logs while restarting DBs
ee8bf2e41 fixes 1228: rockdbjni loadLibraryFromJarToTemp fails when file is already present (#1232)
f85df120f Re-enable tsan crash white-box test with reduced killing odds
89e4c4882 Update README.md to include appveyor badge
b06ca5f86 ldb load, prefer ifsteam(/dev/stdin) to std::cin (#1207)
4ea0ab3cc Revert "Remove bashism from `make check` (#1225)"
12767b313 fixes 1220: rocksjni build fails on Windows due to variable-size array declaration (#1223)
a9d512a76 Update .gitignore for internal release
d5a51d4de Need to make sure log file synced before flushing memtable of one column family
89f319c2d Fix unit test which breaks lite build
b50632920 Add unit test not on /dev/shm as part of the pre-commit tests
b9a97181a Bump next release version
663afef88 Add EnvLibrados - RocksDB Env of RADOS (#1222)
32604e660 Fix flush not being commit while writing manifest
9ab38c45a Remove %z Format Specifier and Fix Windows Build of sim_cache.cc (#1224)
08ab1d83a Remove bashism from `make check` (#1225)
f9b14be49 Re-enable TSAN crash test but only with black box crash test
68f3eb746 Run release build for CLANG and GCC 4.8.1 in pre-commit tests too
e70020e4f Only cache level 0 indexes and filter when opening table reader
7bedd9440 Build break fix
68a8e6b8f Introduce FullMergeV2 (eliminate memcpy from merge operators)
e70ba4e40 MemTable::PostProcess() can skip updating num_deletes if the delta is 0
2a282e5f5 DBTablePropertiesTest.GetPropertiesOfTablesInRange: Fix Flaky
d9cfaa2b1 Persistent Read Cache (6) Persistent cache tier implentation - File layout
9430333f8 New Statistics to track Compression/Decompression (#1197)
515b11ffa fixes#1210: rocksdb/java/CMakeLists.txt lacks cmake_minimum_required (#1214)
876cb8bfb fixes#1212: rocksdbjni maven build does not escape slashes in groovy script (#1213)
21c55bdb6 DBTest.DynamicLevelCompressionPerLevel: Tune Threshold
4b9525358 Refactor cache.cc
c6a8665b3 Update LANGUAGE-BINDINGS.md
880ee363e ldb backup support
6797e6ffa Avoid updating memtable allocated bytes if write_buffer_size is not set
dda6c72ac Add DestroyColumnFamilyHandle(ColumnFamilyHandle**) to db.h
56222f57d Avoid FileMetaData copy
15b7a4ab8 Fixed output size and removed unneeded loop
6ea41f852 Fix deadlock when trying update options when write stalls
efd013d6d Miscellaneous performance improvements
e6f68faf9 Update Makefile to fix dependency
816ae098e fix test failure
e295da126 Fix Log() doc for default level
8e6b38d89 update DB::AddFile to ingest list of sst files
296545a2c Fix clang analyzer errors
61dbfbb6c Add release build to RocksDB per-diff/post-commit tests
907f24d0e Concurrent memtable inserter to update counters and flush state after all inserts
0f691c4b5 CLI option & Rename() allow overwrite
7c190070b delete unnessary pointer cast in beginInternalTransaction() function
e1b3ee8a7 Cleanup auto-roll logger flush-while-rolling test
cd4178a01 Add a new feature to enforce a sync point only active on a thread
b954847fc Fix release build for MyRocks by using debug-only code only in debug builds
a00bf1b3c Add More Logging to track total_log_size
01f77cb19 Update USER.md to include more services at Facebook.
eb53c05a3 Add comment for GetBackupInfo about returned BackupInfos order
32df9733d Add options.write_buffer_manager: control total memtable size across DB instances
5aaef91d4 group multiple batch of flush into one manifest file (one call to LogAndApply)
a45ee8318 Fix a bug that accesses invalid address in iterator cleanup function
38fae9e65 fix typos in HISTORY.md (#1192)
1a11c934d Disable some persistent cache tests on linux/clang
9b5adea97 Add More Logging to track total_log_size
95d96eeeb remove LockFile
ff45d1b54 if read only backup engine can't find meta dirs, return NotFound() instead of IOError()
cb2476a0c fix rate limiter to avoid starvation
6b7167651 Run env_basic_test on Env::Default
9eb0b5395 Move env_basic_test cleanup to TearDown
1fe3bf829 Re-enable linux on travis
43692793e Fixed Minor Bug on Windows Build and db_bench_tool.cc (#1189)
95c192475 writable file close before reset
197b832af Update USERS.md
bdb1d19a6 Fix UBSan build break caused by variable not initialized
b726bf596 FreeBSD does not have std::to_string (#1190)
faa7eb3b9 Improve regression_test.sh
c4cef07f1 Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
892e9d304 make transaction WriteOptions modifiable
4f2b0946d fix simple typos (#1183)
3b7ed677d ColumnFamilyOptions API [CF + RepairDB part 3/3]
56ac68629 Detect column family from properties [CF + RepairDB part 2/3]
3fc713ed9 delete 2nd level children for default env
343507afb Refactor to use VersionSet [CF + RepairDB part 1/3]
aa432be4b Workarounds for continuous build implementation
8cd9f04fe Test change to verify new commit detection
8a4ee7e90 Trivial change to test cont. build
af6248d8b Fix max_bytes_for_level_base comment
0d7b26123 add tests to env_basic_test.cc
6576fa05a Fix minor typos and PHP source file name used to trigger the builds
c4e19b77e Add a read option to enable background purge when cleaning up iterators
fa813f747 Update DB::AddFile() to ingest the file to the lowest possible level
d6b79e2fd Remove filter_deletes from crash_test
a52e4d7d0 Framework for enabling continuous RocksDB build and tests
f9bd66779 Makefile warning for invalid paths in make_config.mk
88a2776db Update SstFileWriter to use bottommost_compression if avaliable
e87d5df1a TiKV use-case (#1172)
7b79238b6 Deprectate filter_deletes
4939fc389 Bulk load mode shouldn't stop ingest
3a2bccc84 Fixed a crash bug that incorrectly parse deprecated options in options_helper
cf8adc971 Allow arcanist_util to work with both new and old arc versions
30a24f2d3 Add InternalStats and logging for AddFile()
d26a84807 Temporarily disable travis on linux
249e796df Fix Flaky DBCompactionTest.SkipStatsUpdateTest
162c9170d Make sandcastle access secure
8366e10ff Fix clang build
0babce57f Move away from enum char value -1
812dbfb48 Optimize BlockIter::Prev() by caching decoded entries
550bf895e Minor bug fix with log name
886af5910 Fix examples/Makefile jemalloc error
e3b1e3dfa Expose save points in Java WriteBatch and WBWI (#1092)
f5177c761 Remove wasteful instrumentation in FullMerge (stacked on D59577)
97fd2a638 Remove dead Jenkins code and support `arc diff --preview` in RocksDB
7c919decc Reuse TimedFullMerge instead of FullMerge + instrumentation
9a33a723b Remove the comments saying allow_concurrent_memtable_write and enable_write_thread_adaptive_yield are not stable
81f6b33d9 Fix tsan error
bc8af90e8 add option to not flush memtable on open()
8100ec2cd Fix libgcc broken lib path
7360db39e Add a check mode to verify compressed block can be decompressed back
2a79af1c5 Fix Java Break Related to memtable bloom bits to size ratio change
6faddd7c5 Merge db/slice.cc into util/slice.cc
5009b5326 BlockBasedTable::FullFilterKeyMayMatch() Should skip prefix bloom if full key bloom exists
2d05eaeb2 Fix name conflict in delete_shceduler_test and db_sst_test
bde7d1055 Fix clang_analyze path in fbcode_config.sh
ca3db5478 Fetch branches from github for format compatibility test
20699df84 memtable_prefix_bloom_bits -> memtable_prefix_bloom_bits_ratio and deprecate memtable_prefix_bloom_probes
e9c1face6 Minor fix to precommit-check.py
fcc47fa5f New features to precommit check script
56887f6cb Backup Options
a683d4aba URI-based Env selection for db_bench
53a4bd8a6 duplicate line
3e8686961 release 4.9 update version and history
b2973eaae Remove options builder
5b197bff4 Enabled Windows build for volatile tier implementation
281fbdddc Temporarily remove PersistentCacheOptions from persistent_cache_tier.h to fix unity build
2ae15b2d5 Format compatible test should cover forward compatibility up to 4.8.
00a058725 netflix use-case
5091dfc1e use branch names in format compatibility test
edc764e91 Use valgrind built with gcc-4.9-glibc-2.20
8ff59b2b4 Disable PersistentCacheTierTest.VolatileCacheInsertWithEviction test under TSAN temporarily
1ba452226 Fix for GCC 5.4 (#1157)
972c895c3 Previously WARN level logging became FATAL level logging in the Java API (#1089)
a73b26f60 Adding test for contiguous WAL detection
098da8348 Fix CLANG build break caused by the recent Persistent Cache change
54db29b8f Use gvfs links in dependencies.sh
d755c62f9 Persistent Read Cache (5) Volatile cache tier implementation
fda098461 Allow regression_test.sh to specify OPTIONS_FILE. Add header comments.
0fee89684 Fix Windows build
10d46b9c8 Update tp2 clang path to fix clang build
774a6aa29 Java API - Rename geHistogramData -> getHistogramData (#1107)
3070ed902 Persistent Read Cache (4) Interface definitions
e42dc9192 Update paths for fbcode dependencies
5647fa427 stack_trace,cc: The current Stacktrace code does not compile for FreeBSD (#1153)
0d65acec0 threadpool.cc: abort() lives in stdlib.h on FreeBSD (#1155)
19dd5a61c env_chroot.cc: FreeBSD likes stdlib.h for realpaht() and friends (#1154)
5aca977be env_basic_test library for testing new Envs [pluggable Env part 3]
1147e5b05 Adding support for sharing throttler between multiple backup and restores
6e6622abb Create env_basic_test [pluggable Env part 2]
e53287794 Add statistics field to show total size of index and filter blocks in block cache
a791a2cf2 Java API - Bugfix for native linking of Compaction Filter (#1099)
af0c9ac01 Env registry for URI-based Env selection [pluggable Env part 1]
02ec8154e allow updating block cache capacity from C (#1149)
630b732cb fix flaky sim_cache_test
62d548098 Add persistent cache to Windows build system
842958651 Fix race condition in SwitchMemtable
88acd932f Allows db_bench to take an options file
3a276b0cb Add a callback for when memtable is moved to immutable (#1137)
8cf0f86d3 Allow regression test to run db_bench at a remost host
27ad17071 Fix Windows build break
936973d14 Small tweaks to logging to track the number of immutable memtables
21c047ab4 add readahead size option (#1146)
71c7eed91 Assert boundary checks for SetPerfLevel()
5d85fdb2c add missing lock
c40c4cae1 LDBCommand::SelectCommand to use a struct as the parameter
590e2617e fix delete file bug when do checkpoint (#1138)
8dfa980cb Add statically-linked library for tools/benchmarks
f62fbd2c8 Handle overflow case of rate limiter's paramters
57461fba8 In-memory environment read beyond EOF
0e2000017 LDBCommand::InitFromCmdLineArgs() to move from template to function wrapper
472c06e90 Add low and upper bound values for rocksdb::PerfLevel enum
157e0633e MutexLock -> ThreadPoolMutexLock in util/threadpool.cc
23d4cf483 include/rocksdb/sst_file_writer.h should not depend on util/mutable_cf_options.h
345fd73fa Fix flaky DBTestDynamicLevel.DynamicLevelMaxBytesBase2
8fc75de32 Minor fix to disable DynamicLevelMaxBytesBase2
9dd50d990 Fix db_bench
5d660258e add simulator Cache as class SimCache/SimLRUCache(with test)
d379d110e Update CMakeLists.txt for added test
21f847eda Direct IO fix for Mac
99765ed85 Clean up the ComputeCompactionScore() API
def2f7bd0 Expose report_bg_io_stats option in the C API. (#1131)
f89caa127 Direct IO capability for RocksDB
8f1214531 C API: Expose DeleteFileInRange (#1132)
11f329bd4 db/db_impl: restrict WALRecoveryMode when using recycled log files
2b2a898e0 db/log_reader: combine kBadRecord{Len,Checksum} for readability
34df1c94d db/log_reader: treat bad record length or checksum as EOF
7947aba68 db/log_reader: move kBadRecord{Len,Checksum} handling into ReadRecord
847e471db db/log_test: add recycle log test
4e7e41ba7 Disable lite build/testing for persistent read cache
1d725ca51 Deprecate BlockBasedTableOptions.hash_index_allow_collision=false.
0e77246ba backupable_db.cc: lambada to explictly caputre "this" when escaping scope
2073cf377 Eliminate use of 'using namespace std'. Also remove a number of ADL references to std functions.
26adaad43 Split WinEnv into separate classes. (#1128)
bb98ca3c8 Implement GetUniqueId for Mac
1f2dca0ea Add MaxOperator to utilities/merge_operators/
f6e404c20 Added "number of merge operands" to statistics in ssts.
7383b64b3 Fix formatting of HISTORY.md (#1126)
0e665c399 Disable long running GroupCommitTest (#1125)
3c69f77c6 Move IO failure test to separate file
533cda90c Add GetStringFromCompressionType to include/rocksdb/convenience.h
c70a9335d Fix mutex unlock issue between scheduled compaction and ReleaseCompactionFiles()
05c5c39a7 Fix build
a6254f2bd Long outstanding prepare test
2ead11511 Fix TransactionTest.TwoPhaseMultiThreadTest under TSAN
1f0142ce1 Persistent Read Cache (Part 2) Data structure for building persistent read cache index
43afd72be [rocksdb] make more options dynamic
bac3be7c4 Fix build issue. (#1123)
f6aedb62c Fix Transaction memory leak
a08c8c851 Added PersistentCache abstraction
5c06e0814 [ldb] Templatize the Selector
aab91b8d8 Use generic threadpool for Windows environment (#1120)
a40033639 TransactionLogIterator sequence gap fix
fa3536d20 Store SST file compression algorithm as a TableProperty
40123b380 signed vs unsigned comparison fix
49815e384 [ldb] Export LDBCommandRunner
c1af07ce8 Disable backupable_db_test.cc on Windows
e61ba052b Isolate db env and backup Env in unit tests
560358dc9 Fix data race in GetObsoleteFiles()
5c1c90487 ldb option for compression dictionary size
c27061dae [rocksdb] 2PC double recovery bug fix
a657ee9a9 [rocksdb] Recovery path sequence miscount fix
8a66c85e9 [rocksdb] Two Phase Transaction
1b8a2e8fd [rocksdb] Memtable Log Referencing and Prepared Batch Recovery
0460e9dcc Modification of WriteBatch to support two phase commit
f548da33e Follow symlinks in chroot directory
d86f9b9c3 Fix lite build
4b3172343 Add bottommost_compression option
bfb6b1b8a Estimate pending compaction bytes more accurately
258459ed5 Properly destroy ChrootEnv in env_test
fca5aa6fc Initial script for the new regression test
e1951b6f2 Add --index_block_restart_interval option in db_bench
730f7e2e2 Fix win build
a9b3c47c8 Fix includes for clang on OS X
3f16a836a Introduce chroot Env
269f6b2e2 Revert "Modification of WriteBatch to support two phase commit"
04dec2a35 [ldb] Export ldb_cmd*.h
72c73cdc8 Java API - Add missing HEADER_LEVEL logging (#1104)
4d02bfa3a Add support for PauseBackgroundWork and ContinueBackgroundWork to the Java API (#1087)
8f65feafc Have sandcastle run lite_test for every diff
0d590d999 Make max_dict_bytes optional in options string
7ccb8d6ef BlockBasedTable::Get() not to use prefix bloom if read_options.total_order_seek = true
e3c6ba37d OptimizeForSmallDb(): revert some options whose defaults were just changed
967476eae Fix valgrind (DBIteratorTest.ReadAhead)
9790b94c9 Add optimize_filters_for_hits option to db_bench
a4ea345b0 Fixing lite build
24a24f013 Enable configurable readahead for iterators
ff4b3fb5b Fix Iterator::Prev memory pinning bug
cba752d58 sst_dump won't print size for unsupported compression type
6e801b0bd Eliminate memcpy in Iterator::Prev() by pinning blocks for keys spanning multiple blocks
1b166928c Release RocksDB 4.8.0
b8cf9130f Fix#1110, 32-bit build failure on Mac OSX (#1112)
21441c09b Fix calling GetCurrentMutableCFOptions in CompactionJob::ProcessKeyValueCompaction()
4ea6e051e Fix multiple issues with WinMmapFile fo sequential writing (#1108)
f3bb024fd Fix clang build
6e56a114b Modification of WriteBatch to support two phase commit
1d2e4ef74 ldb support new WAL records
a92049e3e Added EventListener::OnTableFileCreationStarted() callback
e8115cea4 Revert "Use async file handle for better parallelism (#1049)" (#1105)
6a14f7a97 Change several option defaults
c6c770a1a Use prefix_same_as_start to avoid iteration in FindNextUserEntryInternal. (#1102)
992a8f83b Not enable jemalloc status printing if USE_CLANG=1
029022b0f Fix crash_test
a06faa632 Skip PresetCompressionDict test for lite
e7899c661 Fix build issue. (#1103)
0f428c561 Fix compression dictionary clang osx error
6d4832a99 Merge pull request #1101 from flyd1005/wip-fix-typo
af70f9ac6 Fix typo in build_tools/fbcode_config.sh
54de13aba Fix compression dictionary clang errors
0850bc514 Fix build on machines without jemalloc
4032145ad Configurable compression in db_bench
843d2e313 Shared dictionary compression using reference block
ad573b902 Temporarily disable CompactFiles in db_stress in its default setting
1c80dfab2 Print memory allocation counters
eb7398085 Fix BackupableDBTest
ac0e54b4c CompactedDB should not be used if there is outstanding WAL files
d719b095d Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData)
1995e34d6 Retrieve file size from proper Env
7c14abf2c Improve BytewiseComparatorImpl::FindShortestSeparator
f3eb0b5b8 Make EventListenerTest.CompactionReasonLevel more deterministic
7b78d623f Shouldn't report default column family's compaction stats as DB compaction stats
995353e46 Fix null-pointer-dereference detected by Infer (https://github.com/facebook/infer)
24110ce90 Correct Statistics FLUSH_WRITE_BYTES
b71c4e613 Alpine Linux Build (#990)
90ffed1f9 Update USERS.md with link to LinkedIn blog post (#1088)
99a3bf8f6 Merge pull request #1068 from daaku/c-purge-old-backups
b54c34742 Use async file handle for better parallelism (#1049)
c146c9be1 rocksdb_create_mem_env to allow C libraries to create mem env (#1066)
6da70c581 expose more options in the c api (#1067)
6f01687aa C rocksdb_create_iterators to expose NewIterators (#1069)
644f978c1 Fix RocksDB Lite build in db_stress
5bd4022fe Add comparator, merge operator, property collectors to SST file properties (again)
7a6045a3c fix typo in HISTORY.md
73a847ef8 Add per-level compression ratio property
ee221d2de Introduce XPRESS compresssion on Windows. (#1081)
874c96ac1 Merge pull request #1083 from flabby/master
6cbffd50d Enable testing CompactFiles in db_stress
b95510ddf Fix DBTest.RateLimitedDelete flakiness
6356b4d51 Fix nullptr dereference in adaptive_table
9385fd72c Delete deprecated backup classes in Java
a2466c885 [db_stress] Make subcompaction random in crash_test
c3c389d54 Fix column label for L0 write sum
ec84bef24 New legocastle output parsing
725184b04 Fix db_block_cache_test in lite build
290883d94 Fix lite build
23089fd28 write_callback_test: clean test directory before running tests
792762c42 Split db_test.cc
40b840f29 Delete deprecated *BackupableDB interface for backups
6affd45d8 Make more tests run in parallel
47833e0ab Merge branch 'master' of github.com:facebook/rocksdb
e5c614e1d Fixing snapshot 0 assertion
1b1adebe8 Fixing snapshot 0 assertion
6d436a3f8 DBTest.HardLimit made more deterministic
994d9bc82 Make parallel valgrind watch-log more readable
9d35ae649 Make DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels more deterministic
cea8ed970 Fix backupable_db_test test cases that can't run by itself
4b6833aec Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too.
3894603fe Allow valgrind_check to run in parallel
c9d668c58 Fix unit tests issues on Windows (#1078)
083cadc7e Minor fix to Java sandcastle job definition
80b74a6c6 Include ldb tool in the windows build (#914)
7c14d11eb Minor fix to java build job
535af525d BlockBasedTable::PrefixMayMatch() to skip index checking if we can't find a filter block.
09be5cad5 Minor fix to sandcastle java job definition
1aeca9733 Release RocksDB 4.7
dfc3de8b7 Split Travis unittests Job
c2c8fe47f Add Java job for sandcastle
19ef3de57 Fix ManualCompactionPartial test flakiness
b345b3662 Add a minimum value for the refill bytes per period value
dff4c48ed BlockBasedTable::PrefixMayMatch: no need to find data block after full bloom checking
0353b853c Propagate sandcastle run error to UI
b885f33a5 Parallelize travis jobs
71303e04e Update db_bench_tool.cc (#1073)
63cf15bb9 Fix option settable tests
e208575b2 using java7 in runtime for hdfs env (#1072)
13e6c8e97 Relax an assertion in Compaction::ShouldStopBefore
ae21d71e9 Fixed a bug in RocksDB Statistics where flush is considered as compaction
8e0e22f76 Fix Windows build by replacing strings.h include
5675d5037 Revert travis commit
91f0f1f5e fix travis
a23c6052c Don't run DBOptionsAllFieldsSettable under valgrind
30d72ee43 PrefixTest.PrefixAndWholeKeyTest should run against a different directory from prefix_test
0e3cc2cf1 Add column family info to TableProperties::ToString()
2448f8037 Make sure that if use_mmap_reads is on use_os_buffer is also on
114a1b879 Fix build errors for windows
052299035 Improve sst_dump help message
0930e5e99 Update comments on include/rocksdb/perf_context.h
3b977bcdc instructing people to use java7 for hdfs (#1063)
1518b733e Change default number of cache shard bit to be 6 and max_file_opening_threads to be 16.
ada88b63f fix wrong assignment of level0_stop_writes_trigger in spatialdb (#1061)
2391ef721 Embed column family name in SST file
ab4c62332 Don't use version in the error message
d02eb8d00 Fix unused variable warning
9278097f8 Merge pull request #1056 from facebook/igorcanadi-patch-1
cc87075d6 No need to limit to 20 files in UpdateAccumulatedStats() if options.max_open_files=-1
8a1a603fd Eliminate std::deque initialization while iterating over merge operands
f38540b12 WriteBatchWithIndex micro optimization
200654067 Merge pull request #1053 from adamretter/benchmark-java-comparator
f2c43a4a2 Stderr info logger
b55e2165b Rocksdb backup can store optional application specific metadata
9b5198752 Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes.
2feafa3db Change some RocksDB default options
a558830f8 Fixed compile warnings in posix_logger.h and coding.h
51c9464df Merge pull request #980 from adamretter/java-arm
925b5d002 Merge pull request #1054 from DCEngines/magic12
63e8f1b55 Formatted lines to adhere to 80 char limit
994b3bd69 Add support for UBsan builds to RocksDB
21700a510 to/from hex refactor
24420947d Replace kHeader by WriteBatchInternal::kHeader in few more places kHeader was moved from write_batch.cc to header file because it is being used wherever the number "12" was being used to check for record size
3bdbe8961 Merge branch 'magic12' of https://github.com/dcengines/rocksdb into magic12
78711524b In all the places where log records are read, there was a check that record.size() should not be less than 12.
e7c64fb11 Imporve sst_file_manager comment
99ffb3d53 Fix perf_context::merge_operator_time_nanos calculation
07bb12d97 Update internal jemalloc and other versions
ad2fdaa82 Correct a typo in a comment
be9816b3d Fix data race issue when sub-compaction is used in CompactionJob
e3802531f Merge pull request #1050 from yuslepukhin/support_db_test2
e7cc49cbd Add support for db_test2 for dev and CI runs
3996770d0 Add comments to perf_context skip counters
4e85b7479 Make WritableFileWrapper not screw up preallocation
ec458dcde Merge pull request #1047 from PraveenSinghRao/wal_filter_ex
583157f71 Avoid overloaded virtual function
b9d4fa890 Options settable tests to use a different special charactor
60e34baef Merge pull request #1044 from PraveenSinghRao/wal_filter_ex
136b8e0ca Merge from master
2dcbb3b4f Addressed review comments
b1fafcaca Revert "Adding pin_l0_filter_and_index_blocks_in_cache feature."
5f8741a69 Revert "Fix failing Java unit test."
43bbb5619 tools/check_format_compatible.sh to use consistent version when testing backward and forward compatibility
d7ae42b0f Fix failing Java unit test.
fbbb8a614 Add test for Snapshot 0
e182f03c1 Add unit tests for RepairDB
7d371863e travis build fixes
fbea4dc66 Merge pull request #1042 from SherlockNoMad/HistFix
780d2b04c Update format compatible checking tool
4f1c74a46 merge from master
f8c218930 Publish log numbers for column family to wal_filter, and provide log number in the record callback
44756260a Reset block cache in failing unit test.
522de4f59 Adding pin_l0_filter_and_index_blocks_in_cache feature.
be2227126 Merge pull request #1041 from yuslepukhin/adjust_for_jemalloc
4ecc03c03 Fix in HistogramWindowingImpl
2ca0994cf Latest versions of Jemalloc library do not require je_init()/je_unint() calls. #ifdef in the source code and make this a default build option.
90aff0c44 Update --max_write_buffer_number for compaction benchmarks
72224104d Forge current file for checkpoint
33d568611 Merge pull request #1040 from bureau14/master
02e62ebbc Fixes warnings and ensure correct int behavior on 32-bit platforms.
9cad56861 Merge pull request #1039 from bureau14/master
3d29f9146 Improve documentation of the allow_os_buffer parameter.
3ff98bd20 Fix no compression test
b9cc42a72 Merge pull request #1038 from SherlockNoMad/HistFix
f76b260ef Fix FB internal CI build failure
774922c68 Merge pull request #1026 from SherlockNoMad/Hist
17b879b91 Merge pull request #1037 from SherlockNoMad/BuildFix
f11b0df12 Fix AppVeyor build error
6b03f93d4 Fix the build break on Ubuntu 15.10 when gcc 5.2.1 is used
697fab820 Updates to RocksDB subcompaction benchmarking script
58379bfb5 remove division from histogramwidowing impl
1a2cc27e0 ColumnFamilyOptions SanitizeOptions is buggy on 32-bit platforms.
e778c34e7 Merge pull request #1035 from bureau14/master
5bd3da1c5 Added quasardb to the USERS.md file
b2ae5950b Index Reader should not be reused after DB restart
0267655da Update change log for 4.6 release
08304c086 Expose RepairDB as ldb command
fd664a27b Fix Build Error
580fede34 Aggregate hot Iterator counters in LocalStatistics (DBIter::Next perf regression)
54f6b9e16 Histogram Concurrency Improvement and Time-Windowing Support
790252805 Add multithreaded transaction test
e8e6cf017 fix: handle_fatal_signal (sig=6) in std::vector<std::string, std::allocator<std::string> >::_M_range_check | c++/4.8.2/bits/stl_vector.h:794 #174
d9620239d Cleanup stale manifests outside of full purge
f71fc77b7 Cache to have an option to fail Cache::Insert() when full
ee8cc3520 Merge pull request #938 from alexander-fenster/master
765597fa7 Update compaction score right after CompactFiles forms a compaction
f0161c37b formatting fix
aa3f02d50 Improve comment in compaction.h and compaction_picker.h
2200295ee optimistic transactions support for reinitialization
badd6b784 Ignore db_test2
200080ed7 Improve snapshot handling for Transaction reinitialization
171c8e80b Update dependencies / Fix Clang
294bdf9ee Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number"
bf1c4089d Use pure if-then check instead of assert in EraseColumnFamilyInfo
a7d4eb2f3 Fix a bug where flush does not happen when a manual compaction is running
68189f7e1 Update benchmarks used to measure subcompaction performance
dfe96c72c Fix WriteLevel0TableForRecovery file delete protection
451678c8c Merge pull request #1025 from SherlockNoMad/BuildFix
58ecd9132 Fix Windows build
501927ffc [backupable db] Remove file size embedded in name workaround
ef204df7e Compaction always needs to be removed from level0_compactions_in_progress_ for universal compaction
e79ad9e18 Add Iterator Property rocksdb.iterator.version_number
19ea40f8b Subcompaction boundary keys should not terminate after an empty level
deb08b822 Add parsing of missing DB options
f8e90e875 Get file attributes in bulk for VerifyBackup and CreateNewBackup
12fd9b186 Change BlockBasedTableOptions.format_version default to 2
4572a2d8c Update current version to 4.6
74b660702 Rename iterator property "rocksdb.iterator.is.key.pinned" => "rocksdb.iterator.is-key-pinned"
6743135ea Fix DB::AddFile() issue when PurgeObsoleteFiles() is called
432f3adf2 Add DB Property "rocksdb.current_version_number"
188bb2e7a Fix formatting identified by `arc lint`
0f2d2fcff Refactored tests to use try-with-resources
f8e02c782 Deprecate org.rocksdb.AbstractNativeReference#dispose() and implement java.lang.AutoCloseable
0f2fdfe23 Fix the javadoc and the formatting of the base-classes for objects with native references
b5b1db167 Recompute compaction score after scheduling manual compaction
5ea9aa3c1 TransactionDB:ReinitializeTransaction
1f5954147 Introduce Iterator::GetProperty() and replace Iterator::IsKeyPinned()
67789419f Merge pull request #1020 from gongsu832/master
69c471bd9 Handle concurrent manifest update and backup creation
3373c81fa Modify build_tools/build_detect_platform to detect and set -march=z10 on Linux s390x.
990509045 Merge branch 'master' of https://github.com/gongsu832/rocksdb
3492889ab Merge pull request #1019 from javacruft/wip-omit-leaf-frame-pointer-archs
7ca731b12 build: Improve -momit-leaf-frame-pointer usage
21f17aaa6 Modified Makefile and build_tools/build_detect_platform to compile on Linux s390x.
8800975fb Make DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels more robust
cd3fe675a Remove stale TODO
69c98f043 Reorder instance variables in backup test for proper destruction order
82f15fb15 Add test to make sure DropColumnFamily doesn't impact existing iterators
38201b359 Fix assert failure when DBImpl::SyncWAL() conflicts with log rolling
2568985ab IOStatsContext::ToString() add option to exclude zero counters
b04691665 Redo SyncPoints for flush while rolling test
291ae4c20 Revert "Revert "Fixed the bug when both whole_key_filtering and prefix_extractor are set.""
eef63ef80 Fixed CompactFiles() spuriously failing or corrupting DB
79ca039eb Relax the check condition of prefix_extractor in CheckOptionsCompatibility
4b1b4b8ae Merge pull request #1004 from yuslepukhin/child_attr
9ea2968d2 Implement ConsistentChildrenAttribute by using default implementation for now as it works.
c7f1a8a46 Fix LITE build thread_local_test
0914f0ca5 Merge pull request #1003 from yuslepukhin/fix_mutexlock_pthread_build
d37d348da This addresses build issues on Windows https://github.com/facebook/rocksdb/issues/1002
d825fc70d Use condition variable in log roller test
6b2a047df Fix SstFileManager uninitialized data member
a3db93c26 Remove the SyncPoint usage in the destructor of PosixEnv
df9ba6df6 Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
3943d1678 Fix race conditions in auto-rolling logger
d733dd572 [build] Fix env_win.cc compiler errors
cf38e56f2 Fix broken appveyor build caused by D53991
351252b49 Merge pull request #998 from fengjian0106/master
133605249 fix ios build error
d08d50295 Fix transaction locking
730a422c3 Improve the documentation of LoadLatestOptions
a7b6f0748 Improve write_with_callback_test to sync WAL
5bcf952a8 Fix WriteImpl empty batch hanging issue
871cc5f98 fix build without gflags
c90d63a23 can_unlock set but not used
44371501f Fixed a segfault when compaction fails
2f084d39b Merge pull request #992 from jofusa/jdennison/options-typo-fix
7bd284c37 Separeate main from bench functionality to allow cusomizations
bd5f842bb fixes typo in options logging
1c868d684 Fix includes for env_test
545a19395 Add J to commit_prereq so comilation/execution happens in parallel
5bb7371ca [build] Evaluate test names only when db_test exists
6a2b4fcb8 Add flag to forcibly disable fallocate
92a9ccf1a Add a new compaction priority that picks file whose overlapping ratio is smallest
3dc3d1c14 Merge pull request #984 from petermattis/pmattis/comparator-iterate-upper-bound
239aaf2fc Use user_comparator when comparing against iterate_upper_bound.
908100399 Fixed a dependency issue of ThreadLocalPtr
337671b68 Add universal compaction benchmarks to run_flash_bench.sh
3a67bffaa Fix an ASAN error in transaction_test.cc
c5af85eca Fix a memory leak of Slice objects from org.rocksdb.WBWIRocksIterator#entry1
e84137c8a Remove unnessecary java.util.List expense in JNI
76e8beeeb Pass by pointer from/to Java from JNI not by object
0e7e6f6e4 Improve Javadoc
18eb56305 Improve the speed and synchronization around the construction of Java/JNI objects
2a04268be Temporarily disable unstable tests in memory_test.cc
08a78b6b4 Merge pull request #979 from facebook/update_licenses
21e95811d Updated all copyright headers to the new format.
59b3ee658 Env function for bulk metadata retrieval
4a8cbf4e3 Allows Get and MultiGet to read directly from SST files.
fe93bf9b5 Transaction::UndoGetForUpdate
2608219cc crash_test: cover concurrent memtable insert in default crash test
a76e9093f Fix LITE db_test build broken by previous commit
b1887c5dd Explictly fail when memtable doesn't support concurrent insert
8ed343877 Add option to run fillseq with WAL enabled in addition to WAL disabled
73a9b0f4b Update version to 4.5
6f71d3b68 Improve perf of Pessimistic Transaction expirations (and optimistic transactions)
8e6172bc5 Add BlockBasedTableOptions::index_block_restart_interval
34a40bf91 Add --allow_concurrent_memtable_write in stress test and run it in crash_test
73bf330c7 Merge pull request #973 from yuslepukhin/moveout_testcode
f7c0f4e3e perf_context.cc and iostats_context.cc use different output macro (fix unity build)
9656eab00 This partially addresses issue https://github.com/facebook/rocksdb/issues/935 testutil.cc and testharness.cc could not be moved out at this time as they are used by 4 benchmarks in release builds.
14a322033 Remove references to files deleted in commit abb405227848581d3e6d2ba40d94dbc0a5513902
8445e5380 Add a mechanism to run all tests in sandcastle
461cec4e8 Merge pull request #972 from adamretter/wb-threads
9ab269ab3 Threaded tests for WriteBatch
bf767c641 Minor fix to makefile
2c1db5ea5 always invalidate sequential-insertion cache for concurrent skiplist adds
c12ff20ab Merge pull request #965 from koldat/jni_for_windows
a09ce4fcd Skip some of the non-critical tests in ./tools/run_flash_bench.sh
284aa613a Eliminate duplicated property constants
94be872ea Merge branch 'master' of github.com:facebook/rocksdb
0c2bd5cb4 Removing data race from expirable transactions
5fcd1ba30 disable kConcurrentSkipList multithreaded test
466c2c1bf Generate tags for *.c files
70c068c97 Merge pull request #960 from koldat/masterFixes2
a62c519bb RollLogFile tries to find non conflicting file until there is no conflict.
57a95a700 Making use of GetSystemTimePreciseAsFileTime dynamic - code review fixes
502d41f15 Making use of GetSystemTimePreciseAsFileTime dynamic to not break compatibility with Windows 7. The issue with rotated logs was fixed other way.
52153930d Adding support for Windows JNI build - fix Java unit test for release build of JNI DLL
e2972803a Adding support for Windows JNI build
9c2cf9479 Fix for --allow_concurrent_memtable_write with batching
ac3fa9a6f Travis CI to disable ROCKSDB_LITE tests
7b943da1b Merge pull request #967 from SherlockNoMad/ValueSize
b5750790e Merge pull request #968 from yuslepukhin/one_shot_buffer
1ad818295 Fix WriteBatchTest.ManyUpdates, WriteBatchTest.LargeKeyValue under clang
ad7ecca72 Add unit tests to verify large key/value
fdd70d149 Skip filters for last L0 file if hit-optimized
aa5e3b7c0 PerfContext::ToString() add option to exclude zero counters
36300fbbe Enable per-request buffer allocation in RandomAccessFile This change impacts only non-buffered I/O on Windows. Currently, there is a buffer per RandomAccessFile instance that is protected by a lock. The reason we maintain the buffer is non-buffered I/O requires an aligned buffer to work. XPerf traces demonstrate that we accumulate a considerable wait time while waiting for that lock. This change enables to set random access buffer size to zero which would indicate a per request allocation. We are expecting that allocation expense would be much less than I/O costs plus wait time due to the fact that the memory heap would tend to re-use page aligned allocations especially with the use of Jemalloc. This change does not affect buffer use as a read_ahead_buffer for compaction purposes.
1d854fa3d Fixed the asan error on column_family_test
37159a644 Add histogram for value size per operation
3b2a1ddd2 Add options.base_background_compactions as a number of compaction threads for low compaction debt
6ee38bb15 Slowdown of writing to the last memtable should not override stopping
d6c838f1e Add SstFileManager (component tracking all SST file in DBs and control the deletion rate)
45768ade4 transaction allocation perf improvements
77926f93e Merge pull request #964 from benoitc/fix/pi2
03a5661a1 fix build for raspberry 2
4b50f1354 Should not skip bloom filter for L0 during the query.
eadd221d3 Merge pull request #959 from koldat/master
d209076fa Merge pull request #961 from wingify/master
26c618004 Add Wingify to USERS.md
4265f81e8 Remove util/auto_roll_logger.cc (it was moved to different directory)
d7f22b6d2 Fixing generated GenerateBuildVersion.vcxproj when one builds on different locale than english. The problem is that date and time CLI utilities generates different format so that REGEX in CMake does not work.
d20915d52 Disable stats about mutex duration by default
0c433cd1e Fix issue in Iterator::Seek when using Block based filter block with prefix_extractor
035857a31 Fix RocksDB lite build
77ef87ccb Update fbcode_config4.8.1.sh to use update_dependencies.sh
955ecf8b4 Fix an ASAN error in compact_files_test
b0afcdeea Fix bug in block based tables with full filter block and prefix_extractor
167bd8856 [directory includes cleanup] Finish removing util->db dependencies
acd7d5869 [directory includes cleanup] Remove util->db dependency for ThreadStatusUtil
46f9cd46a [directory includes cleanup] Move cross-function test points
22ecb752d Add valgrind to pre-commit sandcastle testing
b7ecf3d21 Fix intermittent hang in ColumnFamilyTest.FlushAndDropRaceCondition
38e1d7fea ldb to support --column_family option
da33dfe18 Parameterize DBTest.Randomized
fb9811ee9 Add a perf context level that doesn't measure time for mutex operations
f7ef1a613 Include rest of dependencies in dependencies.sh
3e9209a07 Updated GetProperty documentation
40911e0b3 Run unit tests in parallel to find failing tests
2fbc59a34 Disallow SstFileWriter from creating empty sst files
f53c95f81 Cosmetic fixes and comments for the reader
f1ed17010 Add tests to make sure new DB or ColumnFamily options are settable through string
f57596b0c Improvements to pre-commit
538eec066 Update fbcode_config.sh to use latest versions automatically
8019aa9b5 improve test for manifest write failure
bcd4ccbc3 Revert D7809
b0a15e7fb Mechanism to run CI jobs on local branch via commit_prereq
bb2888738 Cleanup property-related variable names
29289333d Add named constants for remaining properties
2c2b72218 Disable OptionsParserTest.BlockBasedTableOptionsAllFieldsSettable under CLANG
a300d9928 Added sandcastle pre-commit
202be23e4 Add test that verifies all options in BlockBasedTableOptions is settable through GetBlockBasedTableOptionsFromString()
eceb5cb1b Split db_test.cc (part 1: properties)
94918ae84 db_bench: explicitly clear buffer in compress benchmark
fdbff4239 Crash test to make kill decision for every kill point
39c3e94ff Merge pull request #954 from davidbernard/solaris_build
df7c2f3b5 As per google coding standard replace "using" in option_builder.cc and geodb_impl.cc
12809b44b Revert "Change notification email for travis"
34704d5c7 [easy] Fixed a crash in LogAndApply() when CF creation failed
791dbafa9 Merge pull request #953 from sselva/master
594a5ef02 Merge pull request #955 from bcbrock/ppc64-build
f423f05dc Simple changes to support builds for ppc64[le] consistent with X86
3f12e16f2 Make alloca.h optional
eaa563756 Change notification email for travis
d78c6b28c Changes for build on solaris
2e9fae3f2 Add Rakuten Marketing to USERS.md
83e1de92a move internal build to use zstd 0.4.5
aec10f734 Guard falloc.h inclusion to avoid build breaks
f7ebc2f34 Update HISTORY.mc for 4.4.0
addd9545f Merge pull request #947 from yuslepukhin/align_and_yield
ac50fd3a7 Align statistics Use Yield macro to make it a little more portable between platforms.
b54d4dd43 tools/sst_dump_tool_imp.h not to depend on "util/testutil.h"
48a8667c3 Merge pull request #929 from warrenfalk/fix32
d9bca1e14 Reduce iterator deletion overhead
45d794068 Merge pull request #940 from yuslepukhin/fix_windows_build_signed_unsigned
20d7902df Fix compile error. Use constructor style initialization instead of a cast for simplicity.
e16438bb8 fixing build warning
b73fbbaf6 added --no_value option to ldb scan to dump key only
1477dcb37 Merge pull request #937 from petehunt/master
c7cb1076a Add Smyte to USERS.md
df7e3b622 Include <array> in table/plain_table_key_coding.h
235b162be Not scheduling more L1->L2 compaction if L0->L1 is pending with higher priority
9a8e3f73e plain table reader: non-mmap mode to keep two recent buffers
7ece10ecb DeleteFilesInRange: Mark files to be deleted as being compacted before applying change
94d9df248 fix an unused function compiler warning in crc32c in 32-bit mode
2f01e10fa use static_cast in crc32c instead of c-style cast
601f1306a fix shorten-64-to-32 warning in crc32c
f3fb39814 Fix BlockBasedTableTest.NoopTransformSeek failure
55b37efa1 fix a compile error on 32-bit
8c71eb5af Optimize DBIter::Prev() by reducing stack overhead
73c31377b Revert "Fixed the bug when both whole_key_filtering and prefix_extractor are set."
57605d7ef Fixed the bug when both whole_key_filtering and prefix_extractor are set.
6935eb24e Add ColumnFamilyHandle::GetDescriptor()
9760c842c fix valgrind failure in backupable_db_test
b1a3b4c0d Make ldb automagically determine the file type and use the correct dumping function
ba8344736 Merge pull request #923 from petermattis/pmattis/prefix-may-match
da032495d Optimize GetLatestSequenceForKey
260c29762 Fix index seeking in BlockTableReader::PrefixMayMatch.
e541dcc8f Fix issue #921
51adc5457 fix sporadic failure in fault_injection_test
a2422f053 fix potential test SleepingTask race condition
1627c4b1b Merge pull request #918 from mkurdej/fix/assertion-on-no-disk-space
92d0850f1 Fix failing assertion in logger on Windows when the disk is full.
7699439b7 Prevent the user from setting block_restart_interval to less than 1
4041903ec Enhance db_bench write rate limit
399343205 Add Airbnb and Pinterest to USERS.md
d74c9f0a5 DeleteFilesInRange: Clean job context if no files deleted
1dec5b8f5 Merge pull request #916 from warrenfalk/capi_huge_page_option
12fa27b4f Merge pull request #915 from warrenfalk/capi_full_bloom
0fde291ab expose memtable_prefix_bloom_huge_page_tlb_size option to C API
7e81dba5c Support creation of "full" format bloom filter from C API
bae5b0a1d Fix clang build
ac16663bd use -Werror=missing-field-initializers, to closer match MyRocks build
ab5a9a66d Merge pull request #911 from shuzhang1989/fix_envhdfs_virtual_func
eb5a13904 update posix env schedule call
a41f68ac2 fix inconsistency between env_hdfs and env
7238be090 Fix clang build in db_compaction_test
c9e2490bc Fix DynamicBloomTest.concurrent_with_perf to pass TSAN
63ddb783d Delete files in given key range
d8677a8d2 Upgrade internal CLANG version for FB-internal gcc 4.8.1
edf1cd497 Not generating "__attribute__((__unused__))" for padding fields if it is not CLANG
9eb4f9596 Merge pull request #907 from siying/master
22c0ed8a5 Disable Visual Studio Warning C4351
fcafac053 Fix memory leak in ColumnFamilyTest.WriteStall*
b99d4276f Fix java test buid broken by 7d87f02799bd0a8fd36df24fab5baa4968615c86
11672df19 Fix CLANG errors introduced by 7d87f02799bd0a8fd36df24fab5baa4968615c86
7fafd52dc Merge pull request #900 from shuzhang1989/hdfs_env_fix
2b7c810db more foramt
b79ccbd57 indent
7d87f0279 support for concurrent adds to memtable
5b2587b5c DBTest.HardLimit use special memtable
b4aa82366 format
4dfdd1d92 format
298ba27ae Merge pull request #846 from yuslepukhin/enble_c4244_lossofdata
7810aa802 Merge pull request #899 from zhipeng-jia/fix_clang_warning
4c5560d70 Merge pull request #895 from zhipeng-jia/develop
d43da8ae0 DBTest.DelayedWriteRate: fix assert of sign and unsign comparison
3280ae9a2 Fix warning in release
ec2664fef Fix clang compile error under Linux
4fd23fb13 add a factory method for creating hdfs env
9c176ef90 Update liblz4 to r131
15b890226 Change default options.delayed_write_rate
73b175a77 Fix clang warnings regarding unnecessary std::move
b9f77ba12 When slowdown is triggered, reduce the write rate
445d5b8c5 Fix clang build
e089db40f Skip bottom-level filter block caching when hit-optimized
06c05495e Merge pull request #898 from zhipeng-jia/fix_move_warning
aa515823b Fix clang warning
2ba03196d Merge pull request #897 from yuslepukhin/enable_status_move
dbb8260f7 Make Status moveable Status is a class which is frequently returned by value from functions. Making it movable avoids 99% of the copies automatically on return by value.
2bf9b968c Fix lite_build
d005c66fa Report compaction reason in CompactionListener
728f944f0 Fix computation of size of last sub-compaction
8ac7fb837 Merge pull request #863 from zhangyybuaa/fix_hdfs_error
e53e8219a Merge pull request #894 from zhipeng-jia/develop
e0abec158 Sorting std::vector instead of using std::set
33e09c0e1 add call to install superversion and schedule work in enableautocompactions
22c6b50ee Merge pull request #893 from zhipeng-jia/develop
24c7dae13 Fix clang warning regarding implicit conversion
eff309867 Do not use timed_mutex in TransactionDB
97ea8afaa compaction assertion triggering test fix for sequence zeroing assertion trip
521da3abb Fix BlockBasedTableTest.BlockCacheLeak valgrind failure
a48382399 Fix use-after free in db_bench
bf8ffc1d6 Merge pull request #890 from zhipeng-jia/develop
131f7ddf6 fix typo: sr to picking_sr
c37729a6a db_bench: --soft_pending_compaction_bytes_limit should set options.soft_pending_compaction_bytes_limit
7b12ae97d Add signalall after removing item from manual_compaction deque
d72b31774 Slowdown when writing to the last write buffer
6b2a3ac92 Add documentation for unschedFunction
167fb919a ZSTD to use CompressionOptions.level
32ff05e97 Bump version to 4.4
aececc209 Introduce ReadOptions::pin_data (support zero copy for keys)
e6e505a4d Fix examples
aa29cc128 Improve examples/README.md
97265f5f1 Fix minor bugs in delete operator, snprintf, and size_t usage
b68dc0f83 Merge pull request #885 from yuslepukhin/fix_size_t_formatting
b6d19adcf Use port size_t formatting
963660eb5 Merge pull request #883 from zhipeng-jia/master
99ae549d3 Fix typo
636cd3c71 Clean up listener_test (reuse db_test_util)
030215bf0 Running manual compactions in parallel with other automatic or manual compactions in restricted cases
d26a4ea62 Merge pull request #882 from SherlockNoMad/BuildFix
768a61486 Fix appVeyor Build problem
aca403d2b Fix another rebase problems.
a6fbdd64e Fix rebase issues and new code warnings.
3fa68af31 Enable MS compiler warning c4244. Mostly due to the fact that there are differences in sizes of int,long on 64 bit systems vs GNU.
236fe21c9 Enable MS compiler warning c4244. Mostly due to the fact that there are differences in sizes of int,long on 64 bit systems vs GNU.
84f98792d Transaction::SetWriteOptions()
3bfd3d39a Use SST files for Transaction conflict detection
362d819a1 Improving parser
00d6edf6a Ensure the destruction order of PosixEnv and ThreadLocalPtr
64fa43843 Merge pull request #862 from ceph/wip-env
2074ddd62 env: add EnvMirror
a3ba5915c Correct a comment in include/rocksdb/cache.h
f0a8e5a2d Fixed the valgrind error in ColumnFamilyTest::CreateAndDropRace
9e4462906 Change SingleDelete to support conflict checking
c5af8bffb Merge pull request #879 from charsyam/feature/typos
c30b49954 fix typos in comments
56e77f096 Deprecate options.soft_rate_limit and add options.soft_pending_compaction_bytes_limit
d6e1035a1 A new compaction picking priority that optimizes for write amplification for random updates.
de6958b2e Merge pull request #877 from yuslepukhin/fix_unnecessary_type_truncation
0991cee6c Merge pull request #815 from SherlockNoMad/CounterFix
49957f9a9 Prefer integer arithmetics The code had conversion to double then casting to size_t and then casting uint32_t which caused compiler warning (VS15).
0836d265c Merge pull request #876 from warrenfalk/wf_win_master
c6fedf2bf Add compaction_iterator and delete_scheduler tests to Windows build
ac8e56f05 db_bench: in uncompress benchmark, get Snappy size from compressed stream
9c227923c Merge pull request #788 from OpenChannelSSD/to_fb_master2
fa3dbf203 Merge pull request #853 from Vaisman/enable_C4267_warning
ad6aaf4fa Merge pull request #848 from SherlockNoMad/db_bench
56bbecc31 Merge pull request #867 from SherlockNoMad/CacheFix
188170fb4 Updating HISTORY.md
758dbec7f Fix fb-only build for gcc 4.8.1
774b80e99 Resubmit the fix for a race condition in persisting options
afc84731f Include ldb_tools and sst_dump_tools libraries in shared library
e5c5f2381 Support marking snapshots for write-conflict checking - Take 2
ea1192355 Upgrade to ZSTD 0.4.2
b60cb88c7 Update examples/rocksdb_option_file_example.ini
3d8bb2c89 Fix valgrind failure in IncreaseUniversalCompactionNumLevels
7af91d425 Merge pull request #873 from yuslepukhin/make_vs15_build
1d63c3d61 Revert "Support marking snapshots for write-conflict checking"
78de0c922 Fix up VS 15 build. Fix warnings Take advantage of native snprintf on VS 15
ec704aafd Support marking snapshots for write-conflict checking
770dea932 Fix occasional failure of DBTest.DynamicCompactionOptions
ebc2d490d Split histogram per OperationType in db_bench
f307036bd Revert "Fix a race condition in persisting options"
2fa3ed518 Fix a race condition in persisting options
f276c3a82 Fix valgrind failures in 3 tests in db_compaction_test due to new skiplist changes
291088ae4 Fix undeterministic failure of ColumnFamilyTest.DifferentWriteBufferSizes
3c2b995fb Merge branch 'master' of https://github.com/facebook/rocksdb into CacheFix
355fa9436 EstimatedNumKeys Counter Inaccurate
b2863017b Move posix threads into a library
3a98a7ae7 Replace malloc with new for LRU Cache Handle
a9ca9107b Fix db_universal_compaction_test
d3bb572da Build break fix.
b28b7c6dd Added callback notification when a snapshot is created
e8180f990 added public api to schedule flush/compaction, code to prevent race with db::open
19b1201b2 Merge pull request #865 from yuslepukhin/fix_db_table_properties_test
e0de7ef87 Avoid empty ranges vector with subsequent zero element access
a330f0b3b Fix incorrect merge in db/db_compaction_test.cc
bd7a49d44 Make DBCompactionTestWithParam::CompactionTrigger more deterministic
be006d288 fix LinkFile() undefined reference error
4687ced5d fix ToString() not declared error
bcd7bd122 Relax verification condition of DBTest.SuggestCompactRangeTest
f9103d9a3 DBTest.DynamicCompactionOptions: More deterministic and readable
0ad68518b Fix DBCompactionTestWithParam.CompactionTrigger in non-jemalloc build.
459c7fba3 Revert previous behavior of internal_key_skipped_count
481f9edb1 Fix CLANG build
d7421c22f Fixed some typos in the comments of rocksdb options file example
ef8ed3681 Fix DBTest.SuggestCompactRangeTest for disable jemalloc case
db320b1b8 DB to only flush the column family with the largest memtable while option.db_write_buffer_size is hit
4a009f917 Merge pull request #860 from SherlockNoMad/BuildFix
b4efaebff Fix ms version Appveyor build error
d27ea4c9e Initialize options.row_cache
72930485b Fix clang build
6bbfa1874 BackupDB to have a mode to use file size in file name
f3ea00bc8 Merge pull request #856 from ceph/wip-env
4cedd6b03 EnvWrapper: add ReuseWritableFile
33e0c9382 Reduce extra key comparision in DBIter::Next()
9a9d4759b InlineSkipList part 3/3 - new skiplist type that colocates key and node
520172954 InlineSkipList - part 2/3
78812ec6b InlineSkipList - part 1/3
ffb466da4 Merge pull request #855 from yuslepukhin/enable_3rdparty_override
10d257d64 Enable override to 3rd party linkage
41b32c605 Enable C4267 warning
c5b467306 Fix race condition that causes valgrind failures
efb01a055 Merge pull request #850 from yuslepukhin/enable_2015_build
81be49c75 Have a way for compaction filter to ignore snapshots
047bd22aa Build on Visual Studio 2015 Update 1
88e052772 Reduce moving memory in LDB::ScanCommand
890f44f46 Merge pull request #844 from yuslepukhin/enable_C4804_unsafe_bool
9d0b8f19d plain table reader: avoid re-read the same position for index and data in non-mmap mode
89bacb7e7 Enable MS Warning C4804 : unsafe use of type 'bool' in operation
d5239f870 build_tools/fbcode_config4.8.1.sh: upgrade versions of some dependencies
c4ebb66d6 Not to build forward_iterator_bench now
c342549d0 Merge pull request #841 from yuslepukhin/fix_test_rerun_logic
7cb1293b6 Fix log names when scheduling runs and reruns
51fce92e1 "ldb compact" should force bottommost level compaction
f83164120 Merge pull request #837 from yuslepukhin/rerun_concurrency_value
4159ab816 Merge pull request #839 from SherlockNoMad/memtableOption
6170fec25 Fix build broken by previous commit of "option helper refactor"
3a6643c2f Merge pull request #805 from SherlockNoMad/OptionHelperFix
189b3e03d Fix uninitilizeded SpecialEnv::time_elapse_only_sleep_
d5540e18e DBTest.MergeTestTime to only use fake time to be determinstic
bd7be035e Support Memtable Factory Parse in option_helper.cc
94e39e236 Exclude DBTest.FileCreationRandomFailure as a long running test Increase concurrency to 18 Fix exclusion but in the ps script
4189c0f9a Fix Java Makefile
605a24d94 Block forward_iterator_bench under MAC and Windows
2a0510c9f Failed tests must be rerun with concurrency 1
9b8c9be0b Fix forward_iterator allocation of vector.
5cbb7e43e DBTest.MergeTestTime: relax counter upper bound verification
52e04b3d0 Merge pull request #833 from yuslepukhin/fix_win_build_after_lint
314f62194 Remove headers from the cc since they are in the module's header.
472c74006 Add necessary headers after cpplint rearranged includes
9bc9c93bd Move to version 4.3
3381e2c3e Handle multiple calls to DBImpl::PauseBackgroundWork() and DBImpl::ContinueBackgroundWork()
65a042921 Merge pull request #831 from yuslepukhin/remove_forward_iter_bench_win
ca5566d20 Fix clang build
4175472ad Merge pull request #832 from yuslepukhin/fix_forward_iter_outofbounds
cb9459f85 Fix empty vector write in ForwardIterator
a163cc2d5 Lint everything
8f01f2541 Remove forward_iter_bench from Win build.
dac5b248b UniversalCompactionPicker::PickCompaction(): avoid to form compactions if there is no file
d06b63e99 Fix Rocksdb lite build failure in forward_iterator_bench
7824444bf Reuse file iterators in tailing iterator when memtable is flushed
2ae4d7d70 Make sure that CompactFiles does not run two parallel Level 0 compactions
d781da816 Add CheckOptionsCompatibility() API to options_util
2391b459b Merge pull request #824 from yuslepukhin/try_ci_tests_on_daily
2ab3e2df2 Fix a build break so tests can run
247c49a40 Merge branch 'master' into try_ci_tests_on_daily
935d1495c Run tests imporvements Add sequential rerun for any failed tests. Add env_test case. Limit concurrency Allow to specify individual tests Take $Limit into account when displaying number of tests
5ac16300b Fixed valgrind error in options_util_test
6ce42dd07 Don't merge WriteBatch-es if WAL is disabled
56245ddcf Fixed DBCompactionTest.SkipStatsUpdateTest
e11f676e3 Add OptionsUtil::LoadOptionsFromFile() API
e78389b55 Fixed build failure of RocksDBLite test on options_file_test.cc
e114f0abb Enable RocksDB to persist Options file.
7ed2c3e45 Merge pull request #823 from yuslepukhin/fix_off_t_type
7f59e33b1 Make CI build debug/optimized
ae2dfe404 Try running db_test during integration build
720af2269 Merge branch 'fix_off_t_type' of https://github.com/yuslepukhin/rocksdb into fix_off_t_type
5270b33bd Make use of portable `uint64_t` type to make possible file access in 64-bit.
631863c63 track WriteBatch contents
505accda3 remove constexpr from util/random.h for MSVC compat
5421c9728 Make use of portable `uint64_t` type to make possible file access in 64-bit.
b81b43098 Switch to thread-local random for skiplist
75a8bad2a Merge pull request #821 from yuslepukhin/continue_windows_warnings
986230b8c Revert "Fix TSAN build for fbcode"
f3ca28ab0 Correct the comment of GetApproximateMemoryUsageByType
838676c17 Revert "Adding new table properties"
7c86d5049 Enable C4305 'identifier' : truncation from 'type1' to 'type2'
85a2ce9c1 Enable C4702 unreachable code
62aa1b1b7 Enable C4200 warning nonstandard extension used : zero-sized array in struct/union
5b9ce1a32 Merge pull request #820 from yuslepukhin/enable_compiler_warnings
20f57b171 Enable Windows warnings C4307 C4309 C4512 C4701 Enable C4307 'operator' : integral constant overflow Longs and ints on Windows are 32-bit hence the overflow Enable C4309 'conversion' : truncation of constant value Enable C4512 'class' : assignment operator could not be generated Enable C4701 Potentially uninitialized local variable 'name' used
8be568a9c Adding new table properties
2b42000f4 incorrect batch group size computation for write throttling
c745f1d2c Fix TSAN build for fbcode
fe789c5f2 Document SingleDelete
e89e5b253 Merge pull request #818 from yuslepukhin/improve_test_concurrency
ae7940b62 Fix regression failure in PrefixTest.PrefixValid
3277d172b Improve concurrency when running tests PowerShell seems to have a hard time when a flood of async tasks is scheduled at the same time. I speculated that WaitForMultipleObjects() in Windows can only take up to 64 process handles and if you want to handle more than you should write some additional code which can be sub-optimal. I.e to implement Wait-Job -Any. I decided to test that suggestion and introduced a $Concurrency parameter with a default value of 62. So in the new version the script fires up up to $Concurrency value and wait for anything to complete before starting any more processes. This improved matters greatly. Individual tests against ramdrive now run in 8 minutes and all of the 200+ db_tests run in 9 minutes with concurrency values of 8-16. About 48 is required to load a CPU on my box running against HD but that does not improve running times much.
c8e01ef98 Delete test iterators
9d50afc3b Prefix-based iterating only shows keys in prefix
14c6e1a04 Add write_stress to RocksDB Legocastle runs
db3f5e494 Update HISTORY.md
042fb053f Fix clang
2419f435a Merge pull request #816 from SherlockNoMad/GeoDBTestFix
2e4540991 Fix appveyor build failure
183cadfc8 Add OptionsSanityCheckLevel
dba5e0074 Fixed the compile error in RocksDBLite in memory_test.cc
7d7ee2b65 Add Memory Insight support to utilities
3ecbab004 Add GetAggregatedIntProperty(): returns the aggregated value from all CFs
93a966722 Merge branch 'master' of github.com:facebook/rocksdb
c9aef3c41 Add RocksDb/GeoDb Iterator interface
f31442fb5 Merge pull request #803 from SherlockNoMad/SkipFlush
dcc898b02 Merge pull request #812 from yuslepukhin/fix_windows_warnings
df7ed91ef Fix white space at end of line
a0163c068 Do not disable compiler warnings: C4101 'identifier' : unreferenced local variable C4189 'identifier' : local variable is initialized but not referenced C4100 'identifier' : unreferenced formal parameter C4296 'operator' : expression is always false
279c8e0cd Merge pull request #811 from OverlordQ/unused-variable-warning
affd83369 Fix introduced in 2ab7065 was reverted by 18285c1.
db68a2c09 Merge pull request #806 from yuslepukhin/signed_unsigned_warning
ccc8c10c0 Move skip_table_builder_flush to BlockBasedTableOption
eaaf081d1 Do not suppress C4018 'expression' : signed/unsigned mismatch The code compiles cleanly for the most part. Fix db_test. Move debug file to testutil library.
ff4499e29 Update DB::AddFile() to have less restrictions
9ac88c855 Merge branch 'master' of https://github.com/facebook/rocksdb into OptionTestFix
84992d647 Option Helper Refactoring
11c71a365 db_bench: --compaction_pri default should be rocksdb::Options().compaction_pri
335e4ce8c options_test: fix a bug of assertion
66a3a87ab Merge pull request #797 from SherlockNoMad/optionHelper
550af4ee6 Fix Travis Build Error
a6dd0831d Add Option to Skip Flushing in TableBuilder
2872e0c8c Clean and expose CreateLoggerFromOptions
296c3a1f9 "make format" in some recent commits
6388e7f4e Merge pull request #798 from yuslepukhin/readahead_buffermanagement
f4cbb90c4 Merge pull request #799 from yuslepukhin/fix_random_generator_compile
1277a48f1 Fix 80 character limit issue.
ee2c3236d Fix compilation problem on Windows. char is not a valid template parameter for std::uniform_int_distribution according to the standard. Replacing with int should be just fine.
b69b9b624 Support PlainTableOption in option_helper
c97667d9f Fix RocksDB lite build for write_stress
0d720dfc1 Use the correct variable when fetching table properties.
4b66d9534 Write stress test
47414c6cd Move include/posix/io_posix.h to util/io_posix.h
2889df84c Revert "Avoid to reply on ROCKSDB_FALLOCATE_PRESENT in include/posix/io_posix.h"
28c8758a3 Merge pull request #795 from yuslepukhin/fix_mocktable_id
5c8f2ee78 Fix MockTable ID storage On Windows two tests fail that use MockTable: flush_job_test and compaction_job_test with the following message: compaction_job_test_je.exe : Assertion failed: result.size() == 4, file c:\dev\rocksdb\rocksdb\table\mock_table.cc, line 110
72d6e758b Fix WritableFileWriter::Append() return
d0a18c284 Merge pull request #786 from aloukissas/unused_param
c37223c08 Avoid to reply on ROCKSDB_FALLOCATE_PRESENT in include/posix/io_posix.h
6fbc4f9f3 Implement smart buffer management. introduce a new DBOption random_access_max_buffer_size to limit the size of the random access buffer used for unbuffered access. Implement read ahead buffering when enabled. To that effect propagate compaction_readahead_size and the new option to the env options to make it available for the implementation. Add Hint() override so SetupForCompaction() call would call Hint() readahead can now be setup from both Hint() and EnableReadAhead() Add new option random_access_max_buffer_size support db_bench, options_helper to make it string parsable and the unit test.
d6219e4d9 Mac build break caused by include/posix/io_posix.h not declearing errno,
beb69d451 Merge pull request #765 from PraveenSinghRao/wal_filter
ab0f3b964 crash_test to trigger some less frequent crash point more frequently
7beb743cf Merge pull request #778 from Vaisman/master
4ce117c4d Merge branch 'master' into wal_filter
32cdec634 Fail recovery if filter provides more records than original and corresponding unit-test, fix naming conventions
44d4057d7 Avoid some includes in io_posix.h
2adad23a1 Fix unused parameter warnings.
b0980ff74 Fix unused parameter warnings.
bc898c5f8 Fix unused parameter warnings.
138876a62 Merge pull request #746 from ceph/wip-recycle
581f20fd8 Add LITE tests to Legocastle
3d56d868c Merge remote-tracking branch 'upstream/master'
d69111114 include/posix/io_posix.h should have a once declartion
a6962edf8 Merge pull request #783 from yuslepukhin/remove_test_conditional_compilation
3c750b59a No need to #ifdef test only code on windows
8c11c5dee Merge pull request #768 from OpenChannelSSD/to_fb_master2
6e6dd5f6f Split posix storage backend into Env and library
980a82ee2 Fix a bug in GetApproximateSizes
d0d13ebf6 fix bug in db_crashtest.py
01a41af0a Merge remote-tracking branch 'upstream/master'
5678c05d8 Use DEBUG_LEVEL=0 in make release and make clean
ac25fe6b9 Merge pull request #779 from yuslepukhin/optimize_windows_build
e154ee086 Do not build test only code and unit tests in Release builds Test code errors are currently blocking Windows Release builew We do not want spend time building in Release what we can not run We want to eliminate a source of most frequent errors when people check-in test only code which can not be built in Release. This feature will work only if you invoke msbuild against rocksdb.sln Invoking it against ALL_BUILD target will attempt to build everything.
cd3286fae Error while cmake by building from zip-archive
e3d4e1407 DBCompactionTestWithParam.ManualCompaction to verify block cache is not filled in manual compaction
033c6f1ad T7916298, bug fix
7717ad1af Adding artifacts to stress_crash CI job
0bf656b90 Don't spew warnings when flint doesn't exist
6d6776f6b Log more information for the add file with overlapping range failure
7951b9b07 make field order match initialization order
90228bb08 Merge pull request #771 from maximecaron/patch-1
2938c5c13 merge upstream changes
e3b1d23d3 Bump version to 4.2
a7b2bedfb log_{reader,write}: recyclable record format
4e07c99a9 Fix iOS build
0c59691dd Handle multiple batches in single log record - allow app to return a new batch + allow app to return corrupted record status
32c291e3c Merge branch 'master' of github.com:facebook/rocksdb into T7916298
4575de5b9 #7916298: merge tools/db_crashtest2.py into tools/db_crashtest.py
5c727de6a Merge pull request #777 from yuslepukhin/fix_win_build_uint
cfaa33f9a Update transaction iterator documentation
2f680ed09 Make index same type as auto deduced uint32_t
09f853550 uint is a not a datatype on windows.
ec1f8354a Fix the default assignment of DEBUG_LEVEL in Makefile
f18acd887 Fixed the clang compilation failure
4104e9bb6 log_reader: introduce kBadHeader; drop wal mode from ReadPhysicalRecord
9c33f64d1 log_reader: pass in WALRecoveryMode instead of bool report_eof_inconsistency
718805210 db_test_util: add recycle_log_files to set of tested options
3ac13c99d log_reader: pass log_number and optional info_log to ctor
5830c699f log_writer: pass log number and whether recycling is enabled to ctor
666376150 db_impl: recycle log files
d666225a0 db_impl: disable recycle_log_files if WAL archive is enabled
543c12ab0 options: add recycle_log_file_num option
1bcafb62f env: add ReuseWritableFile
e1a09a770 Implementation for GetPropertiesOfTablesInRange
ad471453e Allow GetProperty to report the number of currently running flushes / compactions.
277dea78f Add more kill points
a98fbacfa Moving memtable related files from util to a new directory memtable
8f143e03f Add ClearSnapshot()
f9ba79ecd crash_test to trigger fail points other than file appending more frequently
2f2de338c Run ROCKSDB_LITE tests in travis
680156ca6 crash_test to run with data sync on
e1a5ff857 Allow users to disable some kill points in db_stress
d306a7ea8 Merge pull request #773 from yuslepukhin/update_requirements
bb64d6da4 Disabling TSAN crash test
8c2fe68fd Update 4 is required for building with MS Visual Studio 13
952ad994a Fix db_test under ROCKSDB_LITE
6d730b4ae Block tests under ROCKSDB_LITE
5eee1ef2d Merge pull request #770 from Vaisman/master
63e507c59 Move ldb and sst_dump from utils to tools.
dae49e829 Make DBTest.ReadLatencyHistogramByLevel more robust
92060b215 Fix build error using Visual Studio 12
9f7413502 Error while cmake by building from zip-archive
b81b2ec25 Fix benchmarks under ROCKSDB_LITE
e587dbe03 Move manual_compaction_test.cc from util to db
666fb5df4 Remove DefaultCompactionFilterFactory.
d662b8dab Merge pull request #766 from PraveenSinghRao/lockfix
f55d3009c Make db_test_util compile under ROCKSDB_LITE
29a47cd2b Include the time unit in the comment of perf_context timers
2b925ccb4 Correct the comments in db/internal_stats.h
35ad531be Seperate InternalIterator from Iterator
91c041e57 move debug variable under ifndef
198ed5898 Merge pull request #760 from jwlent55/use-static-library-header-files
cc4d13e0a Put wal_filter under #ifndef ROCKSDB_LITE
385b41600 Merge pull request #764 from dmittendorf/fix-java-static-packaging
7062d0ea6 Make perf_context.db_mutex_lock_nanos and db_condition_wait_nanos only measures DB Mutex
1fe78a407 Fix tests failing in ROCKSDB_LITE
a6efefef7 Fix format specifiers
f7b2a7b40 Fix format specifiers
7f58ff7c3 Remove db_impl_debug from release build
1ddd91cd2 Fixed packaging of java crossbuild jar by forcing all compiled binaries to be output to the java/target directory. The uber crossbuild jar is then assembled within the java/target directory.
eb2417855 merge from master
c64ae05b1 Move TEST_NewInternalIterator to NewInternalIterator
59a0c219b Adding log filter to inspect and filter log records on recovery
0be50ed12 Merge pull request #763 from PraveenSinghRao/lockfix
a1d37602a Fixing mutex to not use unique_lock
9e819d096 Modify the way java static builds are done so that: 1) There is no need to download and install the compression libraries twice just to get access to their header files during the compile phase. 2) Ensure that the compression library headers files used during the compile phase are the same ones used to build the static library that is linked into the library.
f1fdf5205 Clean up dependency: Move db_test_util.* to db directory
237994409 Fixed an incorrect replace of const value in util/options_helper.cc
0bb8ea56b [RocksDB Options File] Add TableOptions section and support BlockBasedTable
4a7970d75 Modify the way java static builds are done so that: 1) There is no need to download and install the compression libraries twice just to get access to their header files during the compile phase. 2) Ensure that the compression library headers files used during the compile phase are the same ones used to build the static library that is linked into the library.
c4366165e Merge pull request #759 from jwlent55/statically-load-compression-libraries
fa4b5b3db Fix for the travis build caused by my previous commit
3d07b815f Passing table properties to compaction callback
64546af83 Adding parser to CI jobs
def74f876 Deferred snapshot creation in transactions
c5f3707d4 DisableIndexing() for Transactions
776bd8d5e Pass column family ID to table property collector
5a7222782 Ensure that the compression libraries are statically linked into dynamic libraries included in the Java jar. Also build the linux libraries using the portable flag to fix a problem with the linux32 build and improve the general portability of the RocksDB dynamic libraries. ==> linux32: util/crc32c.cc:318:39: error: ‘_mm_crc32_u64’ was not declared in this scope
e61d9c148 Make DBTest.AggregatedTableProperties more deterministic
b77eb16ab New Manifest format to allow customized fields in NewFile.
6732a5765 Merge pull request #756 from viveknelamangala/master
a52888ed0 Install snappy headers to standard locations using yum, so that build_tools/build_detect_platform sets -DSNAPPY flag to g++ . Current jars of rocksdb do no have snappy compression avaliable .
831101b5f Make it harder for users to run debug builds in production
000836a88 CompactionFilter::Context to contain column family ID
3a0bf873b Change RocksDB version to 4.1
77318ee14 Enable crash CI jobs
9803e0d81 compaction_filter.h cleanup
51fa7ecec Bytes read/written from cache statistics
f925208ff Create Makefile target unity_test
a065cdb38 bloom hit/miss stats for SST and memtable
40cdf797d Fix compile error on platforms without fallocate()
77e4ad7ce Fix compile failure on Travis
4049bcde3 Added boolean variable to guard fallocate() calls
aadf49fe6 Travis shouldn't fail when clang-format suggests improvements
d80ce7f99 Compaction filter on merge operands
726d9ce1a Disabling unity
026750265 Support for LevelDB SST with .ldb suffix
5855cdb6d Merge pull request #750 from yuslepukhin/fixup_build_options
7bbe10c01 Merge pull request #751 from yuslepukhin/return_noerror
e95b703b7 Mmap reads should not return error if reading past file
25c58a204 Add shared_linked DEBUG flag, remove port from among the include directories.
60b1c0a70 Fix to CI job definition
9babaeed1 Update dump_tool and undump_tool to accept Options
eb5b637fb Fix condition for bottommost level
9eaff629e Make corruption_test more robust
bf19dbff4 Fix valgrind - Initialize done variable
5c7bf56d3 [RocksDB Options] Support more options in RocksDBOptionParser for sanity check.
115427ef6 Add APIs PauseBackgroundWork() and ContinueBackgroundWork()
a39897369 Adding features to CI job description
a47bf325c Merge pull request #748 from yuslepukhin/improve_test_runs
65324a16d Improve test running scripts Introduce proper command line arguments so we can control the script Add appveyor support Add an ability to run all other (non db_tests) test executables in parallel Use .NET HashSet instead of empty valued hashtable to improve the looks TODO: Some of the tests do not use GTests and need to improve log parsing
7e4ee4231 Merge pull request #743 from edsrzf/amalgamation
7a23e4d8c New amalgamation target
e9a6808c4 Merge pull request #745 from yuslepukhin/test_appveyor_baseline
9320ffd67 Improve CI build and fix Windows build breakage Is there a way to enforce CMake additions for internal changes that seem to come w/o a PR?
03b08ba9a Return MergeInProgress when fetching from transactions or WBWI with overwrite_key
c29af48d3 Add max_file_opening_threads to db_bench
da1cf8a9b Add a missing check for deprecated options in options_helper.cc
5a51fa907 Fix accidental object copy in transactions
1e73b11af Better handling of deprecated options in RocksDBOptionsParser
a8b8295d1 Fixed a compile warning in options_test.cc under clang
74b100ac1 RocksDB Options file format and its serialization / deserialization.
75134f756 Merge pull request #741 from yuslepukhin/test_appveyor_baseline
2e7506d82 Improve CI build and build switches Add an optimized build config switch for faster test runs Change compiler options to introduce more opitmizations and be more inline with MS internal switches. Make appveyor build to utilize all the avaiable cores on the VM (parallel) Introduce new appveyor configuration for daily test runs as it would take too long to run db_test after each checkin even in paralell. With some exclusions we make it in 38 minutes. We currently fail to install ramdisk during the build. Add a powershell script to faicilitate paralell run for db_test cases.
1eff1834b Remove non-existing functions. Closes#680
16d1ba700 Clear SyncPoint Trace in DeleteSchedulerTests
30f74fa96 Make CompactionJobStatsTest.UniversalCompactionTest more robust
afe0dc539 SingleDelete support for Transactions
a263002a3 Fixed a tsan warning in db_stress.cc
e4861e7d6 Fixed a compile error in util/arena.h
0fdb4f168 Fixed a compile warning in util/arena.cc when hugetlb is not supported.
94ac8826c Merge pull request #737 from mlin/readonly-syncwal
60fa9cf0b Override DBImplReadOnly::SyncWAL() to return NotSupported. Previously, calling it caused program abort.
7df348b40 Minor fix to CI job definition
dac3f22b7 Fix the test failure
63e0f8679 Fixed a bug which causes rocksdb.flush.write.bytes stat is always zero
7ee445dd6 Fix the compile warning
174e2be5f Merge pull request #735 from jsteemann/fix-potential-leak-in-allocate
aa58958d3 prevent potential memleaks in Arena::Allocate*()
25fd743d7 Fix SingleDelete support in WriteBatchWithIndex
b6aa3f962 Fixed a memory leak issue in DBTest.UnremovableSingleDelete
7b7b5d9f1 [minor] Reuse SleepingBackgroundTask
e01f32cc3 Parameterizing email id
c58bac701 Fix valgrind failure due to memory leaks
a70d08ec0 Fix the bug of using freed memory introduced by recent plain table reader patch
628216fc1 Simplifying valgrind testing
4805fa0ea Remove ldb HexToString method's usage of sscanf
f03b5c987 Add experimental DB::AddFile() to plug sst files into empty DB
3fdb6e523 Fixed old lint errors in db/filename.cc
b349d2278 Fixed old lint errors in db/filename.h
df34aea33 PlainTableReader to support non-mmap mode
d746eaad5 RandomAccessFileReader should not inherit RandomAccessFile
03dd8f3ca Fixing punit job description
d0c31641d Internal stats WAL file synced to match meaning of the stats of the same name
48b4497f7 Merge pull request #730 from yuslepukhin/fix_write_batch_win_const_expr
a6c22e3e4 Disabling parallel test CI job
489a3e95d Re-work to support size_t max constant for 32/64-bit.
ff57c6511 [RocksJava] Fix test failure of InfoLogLevelTest
f1b9f804e Add a mode to always pick the oldest file to compact for each level
5e8f0a66d Use port::constant for std::muneric_limtis<>::max()
2754ec999 Fix Windows constexpr issue and '#ifdef' column_family_test in Release.
dd2e1eeb3 Disabling log running jobs
199744f4c Merge pull request #728 from jsteemann/fix-missing-include-header
0e65693f1 Merge pull request #727 from jsteemann/micro-optimization
4d6eb52d1 Fix to CI jobs.
3bcc072d2 Added more CI jobs
669b892f9 add missing header required for std::function
624ef456d fixed formatting. thanks @4tXJ7f for pointing me at `make format`
bbb18c827 removed unused variable of type Status, fixed indentation
470483335 pass input string to WriteBatch() by const reference
5ec129971 key_ cannot become nullptr, so no check is needed for that
834b12a8d made Size() function const because it does not modify data
1fc16cb2e Fix clang-format on Travis
e244bdf39 Merge pull request #725 from adamretter/fail-failed-java-test
7d937a090 Exit with non-zero status if one or more Java tests fail
1b598213a Check formatting in Travis
c7fba8029 Fix non-deterministic failure in backupable_db_test
014fd55ad Support for SingleDelete()
f35560d00 Merge pull request #723 from jsteemann/fix-typos
f8b770a94 fixed typos
51e1c1125 Do not flag error if file to be deleted does not exist
a5e312a7a Improving condition for bottommost level during compaction
9aca7cd6d DB::Open() to flush info log after printing DB pointer
cecd903ad Fix gflags build in Travis script
16934d495 Fix wrong constants in db_test_util
2e8e7eb39 Fix the verbosity issue in Java makefile
f21c7415a Change the log level of DB start-up log from Warn to Header.
9566342d2 Build gflags from source for Travis
3ebf11ed1 Adding the increment for a counter for a number of WAL syncs
81a61d75d Skipped tests shouldn't be failures [part 2]
1b7ea8ce8 Skipped tests shouldn't be failures
5ba3297d0 Add compaction time to log output
31a27a360 Callback for informing backup downloading added
d93a9f2aa [travis CI] Run ulimit -n 2000
2b683d497 Add DBOption.max_subcompaction to option dump
0e50a3fcc Merge issue with D46773
a7e80379b LogAndApply() should fail if the column family has been dropped
2819a1db3 Minor fix to CI job definition
df22e2fb7 Relax memory order for faster tickers
488607317 Adding Slice::difference_offset() function
5ce63e30e Merge pull request #720 from AMDmi3/fix-constant-overflow
925babc76 Merge pull request #721 from AMDmi3/printf-size_t
f171faa24 Fix printf format for size_t
4b0b0201c Fix `integer overflow in expression' error
f3170b6f6 DBImpl::FindObsoleteFiles() shouldn't release mutex between getting min_pending_output and scanning files
e467bf0de Fix valgrind error
7cb314b9e Skip some tests in ROCKSD_LITE
0bfe0573e Add gflags dependency to Travis script
5de807ac1 Add options.hard_pending_compaction_bytes_limit to stop writes if compaction lagging behind
7143242d1 Fix compaction_job_stats under ROCKSDB_LITE
592f6bf78 Merge pull request #716 from yuslepukhin/refactor_file_reader_writer_win
e2d6011f4 Minor fix to CI job definition
ad0d70ca1 Relax asserts in arena_test
03ddce9a0 Add counters for L0 stall while L0-L1 compaction is taking place
a3fc49bfd Transactions: Release Locks when rolling back to a savepoint
ddc8b4499 Address code review comments both GH and internal Fix compilation issues on GCC/CLANG Address Windows Release test build issues due to Sync
9f3a66a93 Improvements to CI jobs
7db1471cc Minor fix to CI job
c67d20689 Fixed arena_test failure due to malloc_usable_size()
34cedaff6 Initialize variable to avoid warning
aeb461268 Add counters for seek/next/prev
45e9e4f0b Refactor NewTableReader to accept TableReaderOptions
ddb950f83 Fixed bug in compaction iterator
30e82d5c4 Refactor to support file_reader_writer on Windows. Summary. A change https://reviews.facebook.net/differential/diff/224721/ Has attempted to move common functionality out of platform dependent code to a new facility called file_reader_writer. This includes: - perf counters - Buffering - RateLimiting
af7cdbf64 Run full test suite in Travis
c25f6a85b Removed __unused__ attribute
6db0a939d Fix DBCompactionTest failure with parallel L0-L1 compactions
8aa1f1519 Refactored common code of Builder/CompactionJob out into a CompactionIterator
41bce0586 CI job improvements
95ffc5d2b Correct ASSERT_OK() in ReadDroppedColumnFamily
3c37b3ccc Determine boundaries of subcompactions
112664408 Relaxing consistency detection to include errors while inserting to memtable as WAL recovery error.
abc7f5fdb Make DBTest.ReadLatencyHistogramByLevel more robust
ac9bcb55c Set max_open_files based on ulimit
4cbd2f9aa Merge pull request #714 from facebook/travisformac
d0df54d1f Run travis tests on OS X
a55e5a52a Merge pull request #711 from facebook/testtravis
2b676d5bb Upgrade travis to new architecture
c66d53fee Fixed minor issue in CompressionTypeSupported()
f3f2032c4 Release RocksDB 4.0.0
44b6e99e1 update max_write_buffer_number_to_maintain docblock
aa6eed0c1 Transaction stats
25dbc579f Update HISTORY file for transactions
52386a1e7 Minor fix to sandcastle jobs commands
b5b2b75e5 better tuning of arena block size
342ba8089 Make DBTest.OptimizeFiltersForHits more deterministic
e17e92ea1 Relaxed assert in forward iterator
5e94f68f3 TransactionDB Custom Locking API
0ccf2db38 Fixed broken build due to format specifier
6bdc484fd Added Equal method to Comparator interface
7a31960ee Tests for ManifestDumpCommand and ListColumnFamiliesCommand
778cf4449 Adding email notification.
3a0df7f16 Fixed comparison in ForwardIterator when computing hint for GetNextLevelIndex()
91f3c9079 Fix case when forward iterator misses a new update
ff1953c89 Merge pull request #707 from dkorolev/master
7b463e657 Fixed a typo in INSTALL.md
d9f42aa60 Adding a verifyBackup method to BackupEngine
50dc5f0c5 Replace BackupRateLimiter with GenericRateLimiter
20ef64cae Moving jobs to use gcc-4.9
0f1aab6c1 Add SetLockTimeout for Transactions
14456aea5 Fix compile
76f286cc8 Optimize bloom filter cache misses
0e6e5472e Fixed a compile warning in rocksjni/loggerjnicallback.cc
b8a962d4f Adding commands for few more CI jobs.
8a2d59a35 Add Cloudera's blog post to USERS.md
3c9cef1ee Unified maps with Comparator for sorting, other cleanup
3e0a672c5 Bug fix: table readers created by TableCache::Get() doesn't have latency histogram reported
b42cd6bed Remove the need for LATEST_BACKUP in BackupEngine
0f763db20 Merge pull request #705 from yuslepukhin/rate_limiter_fix
20c44fefb t6913679: Use fallocate on LOG FILESS
f14c3363e Make WinEnv::NowMicros return system time Previous change for the function 555ca3e7b7 (diff-bdc04e0404c2db4fd3ac5118a63eaa4a) made use of the QueryPerformanceCounter to return microseconds values that do not repeat as std::chrono::system_clock returned values that made auto_roll_logger_test fail.
aad0572f8 Fixed the build issue of rocksdbjavastaticrelease
5508122ed Fix a perf regression in ForwardIterator
b72200777 Fix listener_test when using ROCKSDB_MALLOC_USABLE_SIZE
40cd91b7f Fixed compile warning in rocksdbjava
90415cfeb Fixed a compile warning in linux32 environment.
9d6503f88 Fix arena_test test break using glibc-2.17
77a28615e Support static Status messages
18db1e469 better db_bench options for transactions
0be260523 Merge pull request #702 from PraveenSinghRao/remove_spurious
8b689546b Add Subcompactions to Universal Compaction Unit Tests
c6d870ffb Merge branch 'arcpatch-D45741'
57b3a8773 Adding sandcastle determinator for RocksDB
3d78eb66b Arena usage to be calculated using malloc_usable_size()
effd9dd1e Fix deadlock in WAL sync
64f07deb8 remove spurious compression definitions
72a9b73c9 Removed unnecessary checks in DBTest.ApproximateMemoryUsage
cb164bfc4 Do not delete iterators for immutable memtables.
7a0dbdf3a Add ZSTD (not final format) compression type
e2db15efd Merge pull request #701 from PraveenSinghRao/usewinapi_notcruntime
e853191c1 Fix DBTest.ApproximateMemoryUsage
7c916a5d3 Merge pull request #699 from OpenChannelSSD/to_fb_master
0886f4f66 Helper functions to support direct IO
7e327980a Remove usage of C runtime API that has file handle limitation
8ef0144e2 Add argument --show_table_properties to db_bench
1fb2abae2 ColumnFamilyOptions serialization / deserialization.
5f4166c90 ReadaheadRandomAccessFile -- userspace readahead
16ebe3a2a Mmap reads should not return error if reading past file
d286b5df9 DBIter to out extra keys with higher sequence numbers when changing direction from forward to backward
3795449c9 Fix DBTest.GetProperty
a7834a129 Merge pull request #698 from yuslepukhin/address_noexcept_windows
9ccf1bd3e Correct the comment for GetProperty() API.
fbe2c05f5 s/NOEXCEPT/ROCKSDB_NOEXCEPT
6924d7582 Address noexcept and const integer lambda capture VS 2013 does not support noexcept. Complains about usage of ineteger constant within lambda requiring explicit capture.
2f8d71ec0 Moving sequence number compaction variables from SubCompactionState to CompactionJob
bab9934d9 Fix build failure caused by bad merge.
4d28a7d8a Add a whitebox test for deleted file iterators.
249fb4f88 Fix use of deleted file iterators with incomplete iterators
53b88784d Add throttling to multi-threaded backups
09d982f9e Fix compact_files_example
6996de87a Expose per-level aggregated table properties via GetProperty()
86d6c3cde Fix Windows build
20d1e547d Common base class for transactions
205083297 Fixing race condition in DBTest.DynamicMemtableOptions
e46bcc08b Remove an extra 's' from cur-size-all-mem-tabless
4ab26c5ad Smarter purging during flush
4c81ac0c5 Fix benchmark report script
b6def58f7 Changed 'num_subcompactions' to the more accurate 'max_subcompactions'
c85296846 db_iter_test: add more test cases for the data race bug
9130873a1 Add options.new_table_reader_for_compaction_inputs
07d2d3416 Add a counter about estimated pending compaction bytes
41a0e2811 Improve defaults for benchmarks
a203b913c Fixed a rare deadlock in DBTest.ThreadStatusFlush
962aa6429 Merge pull request #695 from yuslepukhin/address_windows_build
5bf890762 More indent adjustment.
e2a9f43d6 Adjust indent
6e9a260b0 Merge branch 'address_windows_build' of https://github.com/yuslepukhin/rocksdb into address_windows_build
1cac89c9b Address windows build issues Intro SubCompactionState move functionality =delete copy functionality #ifdef SyncPoint in tests for Windows Release builds
f25f06ddd Address windows build issues Intro SubCompactionState move functionality =delete copy functionality #ifdef SyncPoint in tests for Windows Release builds
027ca5b2c Total SST files size DB Property
b604d2562 Removing unused variables to fix build
1b114eed4 Free file iterators for files which are above the iterate upper bound to Improve memory utilization
3fd70b05b Rate limit deletes issued by DestroyDB
df79eafcb Introduce GetIntProperty("rocksdb.size-all-mem-tables")
888fbdc88 Remove the contstaint that iterator upper bound needs to be within a prefix
137c37667 Removing variables used only in assertions to prevent build error
b47cc5851 Bounding Number of Subcompactions
e58e1b18e Make tailing iterator show new entries in memtable.
9ec957159 DBOptions serialization and deserialization
b2df20a89 Make HashCuckooRep::ApproximateMemoryUsage() return reasonable estimation.
601b1aaca Fixing Failed Assertion in Subcompaction State Diff
f0da6977a [Parallel L0-L1 Compaction Prep]: Giving Subcompactions Their Own State
f32a57209 Simplify querying of merge results
72613657f Measure file read latency histogram per level
b7198c3af reduce db mutex contention for write batch groups
603b6da8b Add options.compaction_measure_io_stats to print write I/O stats in compactions
dc9d5634f Change master to 3.14
b78c8e07d Merge pull request #689 from msb-at-yahoo/add-tools-target
9f0dd2229 Add a 'tools' target.
463720712 Add test case to repro the mispositional iterator in a low-chance data race case
3bd9db420 [Cleanup] Remove RandomRWFile
c3466eab0 Have Transactions use WriteBatch::RollbackToSavePoint
0db807ec2 Transaction error statuses
c2f2cb021 Pessimistic Transactions
c2868cbc5 Use manual_compaction for compaction_job_test
6b2d57039 Fix Windows build by adding snapshot_impl to CMakeLists
e61fafbe7 Fixed clang-build error in util/thread_local.cc
cee1e8a08 Parallelize LoadTableHandlers
4249f159d Removing duplicate code in db_bench/db_stress, fixing typos
a03085b55 Fix linters on non-fb machines
1ae27113c reduce comparisons by skiplist
b47d65b31 Fixed Segmentation Fault in db_stress on OSX.
a1581eca8 Modernize RocksDB linters
a9dcc0a63 Fix clang build
2cf0f4f47 Adding wal_recovery_mode log message
68f934355 Better CompactionJob testing
22dcaaff3 More accurate time measurement for delete_scheduler_test
0a7ea582c Add auto-build manifest for appveyor
0093271ee Merge pull request #685 from flandr/fix-tls-build
ac04a6cfb Fix OSX + Windows build
16ea1c7d1 simple ManagedSnapshot wrapper
257ee895f Fixed memory leaks
254c4fb88 In HISTORY.md Switch unreleased notes to 3.13
40f893f4a Fix delete_scheduler_test valgrind error
6a4aaadcd Avoid type unique_ptr in LogWriterNumber::writer for Windows build break
d7314ba75 Fixing endless loop if seeking to end of key with seq num 0
48e6e9aa8 Add util/delete_scheduler_impl.cc to CMakeLists.txt
c7742452e Add Statistics.getHistogramString() to print more detailed outputs of a histogram
29b028b0e Make DeleteScheduler tests more reliable
fca88f8e1 valgrind_check to exit on test failures
7d364d0d9 Fix build failure
960d936e8 Add function 'GetInfoLogList()'
7ccd1c80a Add two unit tests for SyncWAL()
3ae386eaf Add statistic histogram "rocksdb.sst.read.micros"
8ecb51a7e "make commit-prereq" should clean up rocksjava properly
9aec75fbb Enable DBTest.FlushSchedule under TSAN
bd2fc5f5f Fix TSAN for delete_scheduler_test
8e01bd114 Fix misplaced position for reversing iterator direction while current key is a merge
c46507102 Removing duplicate code
e06cf1a09 [wal changes 3/3] method in DB to sync WAL without blocking writers
5dc3e6881 Update Tests To Enable Subcompactions
c45a57b41 Support delete rate limiting
102ac118b Update JAVA-HISTORY.md for v3.13
3a1d4e6c9 Merge pull request #670 from skunkwerks/fix_osx_shared_library_names
f5d072ae6 Fixed RocksJava test failure of shouldSetTestCappedPrefixExtractor
f39cbcb0a Merge pull request #654 from adamretter/remove-emptyvalue-compactionfilter
f0b5bcc7b add support for capped prefix extractor in java
18ba58a94 Upgrading jemalloc from 3.6.0 to the latest for fbcode+gcc 4.8.1
ce21afd20 Expose the BackupEngine from the Java API
b0d12a135 Merge pull request #569 from adamretter/travis-java-api
241bb2aef Make DBCompactionTest.SkipStatsUpdateTest more stable.
3424eeb1e Polish HISTORY.md
cf3e05304 crash_test cleans up directory before testing if TEST_TMPDIR is set
24daff6d7 Fix a typo and update HISTORY.md for NewCompactOnDeletionCollectorFactory().
14d0bfa42 Add DBOptions::skip_sats_update_on_db_open
e2a3bfe74 First half of whitebox_crash_test to keep crashing the same DB
2e73bd4ff crash_test to put DB under TEST_TMPDIR
1205bdbce crash_test to cover simply cases
d5c0a6da6 Merge branch 'master' of github.com:facebook/rocksdb
2d41403f4 Made change to fix the memory leak
92f7039ee fix memory corruption issue in sst_dump --show_compression_sizes
be8621ffa Fix compile warning in compact_on_deletion_collector in some environment
26894303c Add CompactOnDeletionCollector in utilities/table_properties_collectors.
20b244fcc Fix CompactFiles by adding all necessary files
87df6295d Make SuggestCompactRangeNoTwoLevel0Compactions deterministic
40c64434d Parallelize L0-L1 Compaction: Restructure Compaction Job
47316c2d0 dump_manifest supports DB with more number of levels
bd852bf11 Fixed typos in db_stress
544be638a Fixing fprintf of non string literal
193dc977e Fixing dead code in table_properties_collector_test
05d4265a2 Merge branch 'master' of github.com:facebook/rocksdb
4be6d4416 Compression sizes option for sst_dump_tool
8161bdb5a WriteBatch Save Points
7bfae3a72 tools/db_crashtest2.py should run on the same DB
d06c82e47 Further cleanup of CompactionJob and MergeHelper
e95c59cd2 Count number of corrupt keys during compaction
221a94a5f Another attempt at adding the Java API and tests to the travis build
1bdfcef7b Fix when output level is 0 of universal compaction with trivial move
6a82fba75 Add missing hashCode() implementation
f73c80143 Fixing Java tests.
14f413760 Correct the comment of DB::GetApproximateSizes
8279d4197 Merge pull request #667 from yuslepukhin/fix_now_microsec_win
eb8e3b4c7 Fix shared library names on OSX
555ca3e7b Fix WinEnv::NowMicrosec * std::chrono does not provide enough granularity for microsecs and periodically emits duplicates * the bug is manifested in log rotation logic where we get duplicate log file names and loose previous log content * msvc does not imlement COW on std::strings adjusted the test to use refs in the loops as auto does not retain ref info * adjust auto_log rotation test with Windows specific command to remove a folder. The test previously worked because we have unix utils installed in house but this may not be the case for everyone.
82f148ef9 Fix test DBCompactionTest.PartialCompactionFailure undeterministic failure
6002801e0 Abandon ROCKSDB_WARNING
4922af6f8 fixed DBTest.GetPropertiesOfAllTablesTest and DBTest.GetUserDefinedTablaProperties flakiness
3bf9f9a83 cleaned up PosixMmapFile a little
fe09a6dae [wal changes 2/3] write with sync=true syncs previous unsynced wals to prevent illegal data loss
06aebca59 Report live data size estimate
85ac65536 Tests to avoid to use TMPDIR directly
f09d45690 Merge pull request #664 from yuslepukhin/add_tests_fix_sanity
31b35c902 Add missing tests, fix db_sanity Add heap_test, merge_helper_test Fix uninitialized pointers in db_sanity_test that cause SIGSEV when DB::Open fails in case compression is not linked.
66a3cbc54 Merge pull request #663 from yuslepukhin/fix_windows_build_refactor
ac5e441ad Fix windows build after refactoring Missing and duplicate files in CMake Missing definition of port::Crash
02b635fa3 Fix undeterministic failure of DBTest.GetPropertiesOfAllTablesTest
3dbf4ba22 RangeSync not to sync last 1MB of the file
7219088cd Move general compaction tests from db_test.cc to db_compaction_test.cc
0adecd9f4 Add db_inplace_update_test back to Makefile
064294081 Improved FileExists API
9f1de9518 Revert Makefile
6867fb19c Revert "Add missing db_log_iter_test in the test list"
443c6646b Move remaining universal compaction tests from db_test.cc to db_universal_compaction_test.cc
7462286d3 Move in-place-update related tests from db_test.cc to db_inplace_update_test.cc
03467bdd4 Add missing db_log_iter_test in the test list
331954ab8 Fixed DBTestUniversalManualCompactionOutputPathId test
a75f23eb8 Relax assertions in unit DropWrites to be more permissible
ee80432ff db_bench add an option of --universal_allow_trivial_move
58b4209e0 Trigger non-trivial compaction in fault_injection_test
59eca2cc9 Make memenv_test runnable in ROCKSDB_LITE
aa8ac6445 Skip unsupported tests in ROCKSDB_LITE
ce9712d34 Make mock_env_test runnable in ROCKSDB_LITE
c06d1d839 Make merge_test runnable in ROCKSDB_LITE
144d2910d Block backupable_db_test_lite in ROCKSDB_LITE
0d1d9aeeb Block plain_table_db_test in ROCKSDB_LITE
4853e228e Make table_test runnable in ROCKSDB_LITE
f0fe9126f Fix compile for write_callback_test in ROCKSDB_LITE
cf6a7bebc Block cuckoo table tests in ROCKSDB_LITE
20922c4a5 Make compaction_picker_test runnable in ROCKSDB_LITE
6e9fbeb27 Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env
5ec829bc4 Cleaning up CYGWIN define of fread_unlocked to port
26ca89319 Block document_db_test in ROCKSDB_LITE
35ca59364 Don't let flushes preempt compactions
79373c372 Fix ROCKSDB_WARNING
74c755c55 Added JSON manifest dump option to ldb command
a96fcd09b Deprecate CompactionFilterV2
1d20fa9d0 Fixed and simplified merge_helper
aede5cd8e Merge pull request #656 from qinzuoyan/fb-master
d730c3677 Merge pull request #657 from yuslepukhin/ensure_clean_public_headers
ac2b9367f Fix a typo in variable
415c47323 Merge after rebasing
d1a457181 Ensure Windows build w/o port/port.h in public headers
6c0c8dee7 Fix data loss after DB recovery by not allowing flush/compaction to be scheduled until DB opened
e4af3bfb2 Test for compaction of corrupted keys
91bf1b80e Java facility to use the RemoveEmptyValueCompactionFilter
3d00271e4 The ability to specify a compaction filter via the Java API
62dec0e2b RemoveEmptyValueCompactionFilter - A compaction filter which removes entries which have an empty value
c5bca5319 Fix compile on Mac
487bba434 extend temp str buffer size
247690fe3 Ensure Windows build w/o port/port.h in public headers
84c3577af fix append bug in DumpDBFileSummary()
43e982562 Fix mongo build -take 2
d8263d958 Unbreak mongo build
12c5528a8 Bump to RocksDB 3.13
81d072623 move convenience.h out of utilities
beb19ad0d Fixing delete files in Trivial move of universal compaction
c61396069 Build fix.
2c8de0eca Update --help message in db_bench.
6b2d44b2f Refactoring of writing key/value pairs
e1c99e10c Replace std::priority_queue in MergingIterator with custom heap, take 2
9a6a0bd8c Style fix in compaction_job.cc
e94c510c3 Make ldb_test not depend on compression
ddad40e93 Fixed nullptr deref and added assert
1bc8eb877 make coverage should execute sequentially
801df912a Move UniversalCompaction related db-tests to db_universal_compaction_test.cc
3ca6b2541 Move TailingIterator tests from db_test.cc to db_test_tailing_iterator.cc
ce829c77e Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc
c3f98bb89 Move CompactionFilter tests in db_test.cc to db_compaction_filter_test.cc
0936362a7 Block SyncPoint in util/db_test_util.h in released Windows mode.
05e194158 Merge pull request #639 from cleaton/setMaxTableFileSize
fc2b71d9c Merge pull request #655 from adamretter/java-make-resolve-maven
8a9fca261 Better error handling in BackupEngine
18d5e1bf8 Remove db_impl_readonly dependency on utilities
9d22a9737 Resolve Java test dependencies from local maven repo if present
49640bd82 Allow write_batch_test to run with ROCKSDB_LITE
a9c510951 Deprecate purge_redundant_kvs_while_flush
5aea98ddd Deprecate WriteOptions::timeout_hint_us
ae29495e4 Avoid manipulating const char* arrays
ab137af4b Partial cleanup of CompactionJob
1879d9370 Add ldb_test.py back to `make check`
b10cf4e2e Move DynamicLevel related db-tests to db_dynamic_level_test.cc
e290f5d3c Block reduce_levels_test in ROCKSDB_LITE
04d201fa0 Block spatial_db_test in ROCKSDB_LITE
49f42ad03 Move global static functions in db_test_util to DBTestBase
625467a08 Move reusable part of db_test.cc to util/db_test_util.h
e8e8c9049 fix compile for optimistic_transaction_test under ROCKSDB_LITE
8bca83e5d Add tombstone information in CompactionJobStats
f9728640f "make format" against last 10 commits
76d3cd328 Fix public API dependency on internal codes and dependency on MAX_INT32
5fd11853c Print Fast CRC32 support information in DB LOG
a6e38fd17 Fix a uncleaned counter in PerfContext::Reset()
e41cbd9c2 Merge pull request #646 from yuslepukhin/ms_win_port
4cab5ebec Merge branch 'ms_win_port' of https://github.com/yuslepukhin/rocksdb into ms_win_port
296de4ae6 Address review comments Rule of five: add destructor Add a note to COMMIT.md for 3rd party json.
041b6f95a perf_context: report time spent on reading index and bloom blocks
d08ba9f0c Merge branch 'ms_win_port' of http://vstfbing:8080/tfs/Bing/_git/repo.RocksDB into ms_win_port
805fe84ba Conditional use of third-party libraries Committed by Alexander Zinoviev <alexander.zinoviev@me.com> 7/9/2015 2:42:41 PM
5555cc500 Improve build system
c903ccc4c Merge from github/master
54d124a38 Conditional use of third-party libraries
7189e90c2 Fix a noisy unit test.
1f4d56570 Add db_bench flag to set cache_index_and_filter_blocks
5c7913233 Revert the changes related to Options, as requested to seperate them into a different patch.
d8586ab22 All of these are in the new code added past 3.10 1) Crash in env_win.cc that prevented db_test run to completion and some new tests 2) Fix new corruption tests in DBTest by allowing a shared trunction of files. Note that this is generally needed ONLY for tests. 3) Close database so WAL is closed prior to inducing corruption similar to what we did within Corruption tests.
4bed00a44 Fix function name format according to google style
e2e3d84b2 Added multi WAL log testing to recovery tests.
5219226d3 Merge branch 'ms_win_port' of http://vstfbing:8080/tfs/Bing/_git/repo.RocksDB into ms_win_port
ef4b87f1b Commit both PR and internal code review changes
95f4c2bcb Conditional use of 3rd-party libraries
4f56632b1 Fix occasional failure in compaction_job_test
411c8e3d1 Build fail fix
b7a2369fb Revert "Replace std::priority_queue in MergingIterator with custom heap"
c0b23dd5b Enabling trivial move in universal compaction
d8e3e766f Fixed a bug in test ThreadStatusSingleCompaction
57d216ea6 Remove assert(current_ == CurrentReverse()) in MergingIterator::Prev()
59b50dcef Update HISTORY.md for Listener
4ce5be425 fixed leaking log::Writers
685582a0b Revert two diffs related to DBIter::FindPrevUserKey()
e12b40399 Initialize threads later in constructor
58d7ab3c6 Added tests for ExpandWhileOverlapping()
155ce60da Fix compaction_job_test
b6655a679 Replace std::priority_queue in MergingIterator with custom heap
e25ee32e3 Arena needs mman header for mmap
d2f0912bd Merge the latest changes from github/master
35cd75c37 Introduce InfoLogLevel::HEADER_LEVEL
acee2b08a Fixed endless loop in DBIter::FindPrevUserKey()
218487d8d [wal changes 1/3] fixed unbounded wal growth in some workloads
feb99c31a Merge remote-tracking branch 'origin' into ms_win_port
e70115e71 Fix unity build by removing anonymous namespace
4159f5b87 Prepare 3.12
a69bc91e3 Multithreaded backup and restore in BackupEngineImpl
9dbde7277 Merge remote-tracking branch 'origin' into ms_win_port
03d433ee6 [RocksJava] Fixed test failures
326da912d Add string.h to Histogram as we init the array out of curly braces
ca2fe2c1b Address GCC compilation issues
19e13a595 Fix header inclusion
18285c1e2 Windows Port from Microsoft
c00948d5e [RocksJava] Fix test failure of compactRangeToLevel
05e283196 Allocate LevelFileIteratorState and LevelFileNumIterator from DB iterator's arena
436ed904d Add rpath option to production builds for 4.8.1 toolchain
b0f1927db Increasing timeout for drop writes.
ec70fea4c Fix a comparison in DBIter::FindPrevUserKey()
501591c42 Make column_family_test runnable in ROCKSDB_LITE
91cb82f34 Merge branch 'master' of github.com:facebook/rocksdb
09f5a4b48 set -e in fb_compile_mongo.sh
6199cba99 Fix race in unit test.
0a019d74a Use malloc_usable_size() for accounting block cache size
4cbc4e6f8 Call merge operators with empty values
619167ee6 Fix mac compile
472e64d39 Improve fb_compile_mongo.sh
c9cd404bc Make flush check for shutdown
4fb09c687 Updating SeekToLast with upper bound
dadc42976 Reproducible MongoRocks compile with FB toolchain
62a8fd154 Make stringappend_test runnable in ROCKSDB_LITE
48da7a9ca Improve the comment for BYTES_READ in statistics.
72cab8895 Block redis_test in ROCKSDB_LITE
dec2c9f56 Make table_properties_collector_test runnable in ROCKSDB_LITE
0b1ffe2e1 Remove -Wl,--no-as-needed flag when making shared_lib in OSX and IOS
674b1181c Bottommost level compaction option
782a1590f Implement a table-level row cache
de85e4cad Introduce WAL recovery consistency levels
530534fce Fix trivial move merge
7015fd81c Add read_nanos to IOStatsContext.
7160f5d80 Fix broken gflags link
dda74111a add setMaxTableFilesSize Options unit test
d62b6ed83 add setMaxTableFilesSize to JNI interface
e1d3c7dbe Fixing valgrind error in checkpoint_test
3bdec09cb Remove ldb_tests.py from make check until it is working again.
15325bf55 First version of rocksdb_dump and rocksdb_undump.
04251e1e3 Add wal files to Checkpoint for multiple column families.
18cc5018b Fix memory leaks in PinnedUsageTest
bf03f59c1 Disable CompressLevelCompaction() if Zlib is not supported
df719d496 Make autovector_test runnable in ROCKSDB_LITE
4d6d47688 Block geodb_test in ROCKSDB_LITE
71b438c4a Remove unused target --- compactor_test
eade498bd Block utilities/write_batch_with_index in ROCKSDB_LITE
760e9a94d Fail DB::Open() when the requested compression is not available
69bb210d5 Add Cache.GetPinnedUsageUsage()
4eabbdb7e Skip bottommost level compaction if possible
4b8bb62f0 Don't dump DBOptions for each column family
176f0bedc Merge branch 'master' of github.com:facebook/rocksdb
bb1c74ce1 Fixed a bug of CompactionStats in multi-level universal compaction case
a66b8157d Merge branch 'master' of github.com:facebook/rocksdb
f06be62fd Replace %llu with format macros in ParsedInternalKey::DebugString())
2dc3910b5 Add --benchmark_write_rate_limit option to db_bench
12e030a99 Use CompactRangeOptions for CompactRange
c89369f57 Move dockerbuild.sh to build_tools/
4716ab4d1 Merge pull request #638 from HolodovAlexander/master
25d600569 Clean up InstallSuperVersion
1369f015e Only initialize the ThreadStatusData when necessary.
1a08d0beb Block c_test in ROCKSDB_LITE
40f562e74 Allow GetApproximateSize() to include mem table size if it is skip list memtable
d59d90bb1 db_bench periodically writes QPS to CSV file
46296cc86 Cygwin build not to use -fPIC
bee8d033f Removed two unused macros in iostats_context
5fec96387 Fixed false alarm of size comparison in compaction_job_stats_test
cccd2199a Revert skip bottommost compaction
20f2b5425 Skip bottom most level compaction if no compaction filter
7842920be Slow down writes by bytes written
a84df655f Don't let two L0->L1 compactions run in parallel
d6ce0f7c6 Add largest sequence to FlushJobInfo
ab455ce49 fix clang build
3eddd1abe Add Env::GetThreadID(), which returns the ID of the current thread.
73faa3d41 Handling edge cases for ReFitLevel
bffaf0a8b Merge pull request #631 from mkhq/patch-1
821cff114 Re-generate WriteEntry on WBWIIterator::Entry()
8b7be1808 Updated OS X instructions, replace homebrew/dupes with homebrew/versions
d03f11090 Link all libraries when building shared libraries
75222d130 Revert "Fix compile"
47f1e7212 Merge pull request #630 from rdallman/c-wb-logdata
51440f83e Fix compile
4949ef08d Re-generate WriteEntry on WBWIIterator::Entry()
735df6655 C: add WriteBatch.PutLogData support
e409d3d74 Make "make all" work for CYGWIN
62c3a9579 Add test for iteration+mutation of WBWI
d9b3338eb Add Yahoo's blog post about Sherpa to USERS.md
75d7075a8 Print info message about files need compaction for debuging purpose
406a5682e Fix hang when closing a DB after doing loads with WAL disabled.
d8c8f08c1 GetSnapshot() and ReleaseSnapshot() to move new and free out of DB mutex
643bbbf08 Use nullptr for default compaction_filter_factory
21f2492ac Fix CYGWin release build
f02ce0c65 Fix ASAN errors in c_test
133130a4f Merge pull request #625 from rdallman/c-slice-parts-support
de4d172d0 Merge pull request #622 from rdallman/c-multiget
6df589b44 Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2e764f06e [API Change] Improve EventListener::OnFlushCompleted interface
7322c7401 Revert incorrect commit
31e60e2a7 Unlock mutex in ReFitLevel
7647df8f9 Fixed the tsan failure in util/compaction_job_stats_impl.cc
b2785472c Fix compile
3ce3bb3da Allowing L0 -> L1 trivial move on sorted data
bb808eadd Changed the CompactionJobStats::output_key_prefix type from char[] to string.
0b3172d07 Add EventListener::OnTableFileDeletion()
211a195d4 C: add MultiGet support
5dc174e11 C: add support for WriteBatch SliceParts params
2d0b9e5f0 Fix compile on darwin
3af668ed1 Fix DBTest.MigrateToDynamicLevelMaxBytesBase slowness with valgrind
408cc4b8e Revert "Merge pull request #621 from rdallman/c-slice-parts-support"
78382d4ba Merge pull request #621 from rdallman/c-slice-parts-support
ca8b85ac0 better document max_write_buffer_number_to_maintain
0483dab2a Remove a TODO that has been done
8afafc278 Fix compile warning in db/db_impl
fe5c6321c Allow EventListener::OnCompactionCompleted to return CompactionJobStats.
3083ed212 Fixed heap-use-after-free error in compaction_job_test.cc
8d8d4e45b Fixed ROCKSDB_LITE compile error due to the missing of TableFileCreationInfo
ab946af08 Fix a compile warning in listener_test.cc
fc8382127 Add EventListener::OnTableFileCreated()
898e803fc Add a stats counter for DB_WRITE back which was mistakenly removed.
ac81130fa Fix Bug: CompactRange() doesn't change to correct level caused by using wrong level
ec7a94436 more times in perf_context and iostats_context
4266d4fd9 Allow users to migrate to options.level_compaction_dynamic_level_bytes=true using CompactRange()
d333820ba Removed DBImpl::notifying_events_
495ce6018 Fixed compile warning in compact_files_example.cc
2ecac9f96 add rocksdb::WritableFileWrapper similar to rocksdb::EnvWrapper
a187e66ad Merge pull request #617 from rdallman/wb-merge-sliceparts
16c197627 Fixed db_stress
4c181f08b Fix compile on darwin
bc7a7a400 fix LITE build
832271f6b Fixed a compile warning in db_stress in NDEBUG mode.
dc9d70de6 Optimistic Transactions
d5a0c0e69 Fixed a compile warning in db_stress
ebfdb3c7f Fixed a compile error in ROCKSDB_LITE
9ffc8ba02 Include EventListener in stress test.
a3da59022 Decrease number of jobs in make release
21cd6b7ad C: add support for WriteBatch SliceParts params
a0635ba3f WriteBatch.Merge w/ SliceParts support
c81535103 Support saving history in memtable_list
ec4ff4e99 Rename EventLoggerHelpers EventHelpers
672dda9b3 [API Change] Move listeners from ColumnFamilyOptions to DBOptions
3ab8ffd4d Compaction now conditionally boosts the size of deletion entries.
a81ac2412 Merge pull request #615 from rdallman/master
6d299b70b Fixed a bug in EventLoggerHelpers::LogTableFileCreation
a0580205c Removed an unused private variable in db_impl.h
328ad902a update an import path to fit in with the rest of the kids
9c38ce1d0 C: extra bbto / noop slice transform
8d26799fe Merge pull request #614 from arschles/docker
32198343f fix typo in c_simple_example
6116ccc23 moving dockerfile to root
d90cee9fd adding docker build script and dockerfile
ea6d3a8ac Don't skip last level when calculating compaction stats
5c224d1b7 Fixed two bugs on logging file deletion.
dc81efe41 Change the log-level of DB summary and options from INFO_LEVEL to WARN_LEVEL
687214f87 Ensure ColumnFamilyOptions.num_levels >= 2 when level compaction is used.
2abb59268 Avoid logging under mutex in DBImpl::WriteLevel0TableForRecovery().
309a9d076 Run tests sequentally if J=1
7fee8775a Allow EventLogger to directly log from a JSONWriter.
7a3577519 Don't artificially inflate L0 score
4cb4d546c Set stats_dump_period_sec to 600 by default
e2c1d4b57 [Public API Change] Make DB::GetDbIdentity() be const function.
eaf61ba9f Minor text correction
f16c0b289 Merge pull request #613 from DerekSchenk/DerekSchenk-patch-issue-606
d1a978ae3 Rename JSONWritter to JSONWriter
3e0817541 Add LDFLAGS to Java static library
812c461c9 Dump db stats in WARN level
b588505a7 Update HISTORY.md for GetThreadList() update.
944043d68 Add --wal_bytes_per_sync for db_bench and more IO stats
d5de04d20 Update history for 3.11
08b6b3796 FORCE_GIT_SHA
04feaeebb Fix comparison between signed and usigned integers
4a855c079 Add an option wal_bytes_per_sync to control sync_file_range for WAL files
b0fdda4ff Allow flushes to run in parallel with manual compaction
74f3832d8 Fixed compile errors due to some gcc does not have std::map::emplace
0c8017dba Remove duplicated code
fb5bdbf98 DBTest.DynamicLevelMaxBytesCompactRange: make sure L0 is not empty before running compact range
6fa708512 CompactRange skips levels 1 to base_level -1 for dynamic level base size
84a9c6a53 add comment
eeb44366b C api: human-readable statistics
3f0867c0f Allow GetThreadList to report Flush properties.
a66f643e9 Use a better way to initialize ThreadStatus::kNumOperationProperties.
7413306d9 Take a chance on a random file when choosing compaction
8c52788f0 Use version defined in Makefile in rocksdb_build_git_sha
5aad88129 DBTest.DynamicLevelMaxBytesBase2: remove an unnecesary check
ec43a8b9f Universal Compaction with multiple levels won't allocate up to output size
714fcc067 Make ThreadStatus::InterpretOperationProperties take const uint64_t*
bc68bd5a1 db_bench to support rate limiter
df1f87a88 Fixed compile error in db/column_family.cc
14431e971 Fixed a bug in EventListener::OnCompactionCompleted().
dbd95b753 Add more table properties to EventLogger
b5881762b Reset parent_index and base_index when picking files marked for compaction
711465cce API to fetch from both a WriteBatchWithIndex and the db
3996fff8a Fix clang build - add override
d97813906 SuggestCompactRange() is manual compaction
2fe24e4b5 Don't treat warnings as error when building rocksdbjavastatic
beda81c53 rm -rf in make clean
50eab9cf3 Fix BackupEngine
962f8ba33 Bump to 3.11
37bbd3185 Merge pull request #601 from adamretter/jdb-bench
77a5a543a Allow GetThreadList() to report basic compaction operation properties.
65fe1cfbb Cleanup CompactionJob
df4130ad8 fix crashes in stats and compaction filter for db_ttl_impl
7ea769487 Fix flakiness in column_family_test
a2c4cc756 Don't treat warnings as error when building release
9aa011fa3 Optimize GetRange Function
36a740889 Fix UNLIKELY parenthesis
9bdbaab94 Update USERS.md
2ab7065af build: avoid unused-variable warning
88044340c Add Size-GB column to benchmark reports
d2346c2cf Fix hang with large write batches and column families.
b6b100fe0 Remove iter_refresh_interval_us
197f01b7b Bugfix remove deprecated option use which was removed in previous commit 019ecd19329ee895284e9b040df0ffe4c08b35d8
aa094e8ea Fix conversion from nano-seconds to milli-seconds and seconds
dddceefe5 Fix clang build
d4540654e Optimize GetApproximateSizes() to use lesser CPU cycles.
fd96b5540 Making GetOptions() comment better (#597)
7246ad34d Don't compact bottommost level in SuggestCompactRange
7f47ba0e2 Fix possible SIGSEGV in CompactRange (github issue #596)
aba005c44 Merge pull request #585 from fyrz/RocksJava-RemoveDeprecatedStuff
d6f39c5ae Helper function to time Merges
a087f80e9 Add scripts to run leveldb benchmark
1bb4928da Include bunch of more events into EventLogger
3db81d535 Fix memory leak in cache_test introduced in the previous commit
4961a9622 Fix build
93ab1473d Merge pull request #593 from charsyam/feature/type-1
6ede020dc fix typos
3d1af4ae6 Don't preinstall jemalloc in Travis
242f9b4c2 Fix CLANG build issue introduced in previous commit
794ccfde8 Task 6532943: Rocksdb - SetCapacity() can dynamically change cache capacity if feasible
98a44559d Build for CYGWIN
d01bbb53a Fix CompactRange for universal compaction with num_levels > 1
e003d3864 Abstract out SetMaxPossibleForUserKey() and SetMinPossibleForUserKey
fd7a35731 Enable open source users to use jemalloc (github issue #438)
aa14670b2 Add an assertion in CompactionPicker
2dc421df4 Implement DB::PromoteL0 method
9bf40b64d Print max score in level summary
397b6588b options.paranoid_file_checks to read all rows after writing to a file.
283a04296 Set --seed per test
618d07b06 Making PreShutdown tests more reliable.
0a91bca5d test: avoid vuln-inducing use of temporary directory
6e359419f Add rpath for production builds
78dbd087d Improve benchmark scripts
6a5ffee0c Fix gflags Makefile
108a927f0 Merge pull request #589 from coderplay/patch-1
a58fd7427 Update USERS.md
d85d08c7b One last fix to Makefile
2db96dca1 Fix make install when there is no shared lib
7d136994c Get rid of error output
79c1b021a Fix Makefile
742fa9e31 Fix compile with two gflags
79c21ec0c skip ioctl-using tests when not supported
6059bdf86 Add experimental API MarkForCompaction()
acf8a4141 maint: use ASSERT_TRUE, not ASSERT_EQ(true; same for false
b5400f90f Kill dead code
48b0a045d Speed up reduce_levels_test
00c2afcd3 Fix bug in ExpandWhileOverlapping()
019ecd193 [RocksJava] Remove deprecated methods
98ef21d2f Merge pull request #584 from pshareghi/rocksdb-3.10-falloch
5b7131c75 [RocksJava] Removed deprecated skipLogErrorOnRecovery methods.
566f65271 [RocksJava] Removed deprecated ColumnFamilyDescriptor methods
6997aa0b6 Merge pull request #582 from fyrz/RocksJava-Fix-RateLimiter
582c4b0f7 [RocksJava] Fix RateLimiter Tests in 3.10
6cfb2150d Merge pull request #581 from vladb38/patch-3
d71e8f7ad Update USERS.md
debaf85ef Bug of trivial move of dynamic level
12d7d3d28 Fix and Improve DBTest.DynamicLevelCompressionPerLevel2
a1271c6c6 Fix build break introduced by new SyncPoint interface change
fcb206b66 SyncPoint to allow a callback with an argument and use it to get DBTest.DynamicLevelCompressionPerLevel2 more straight-forward
281db8bb6 Temporarily disable test CompactFilesOnLevelCompaction
e8808b912 Added falloc.h in build_detect_platform
1983fadcb assert(sorted) in vector rep
9da874801 Get benchmark.sh loads to run faster
9b983befa Fix flakiness of WalManagerTest
d41a565a4 Don't do O(N^2) operations in debug mode for vector memtable
08be1803e Fix bad performance in debug mode
0a0501c8d Add Xfunc to makefile
e7ad14926 Fix flakiness in FIFOCompaction test (github issue #573)
abb405227 Kill benchharness
894e9f745 Update Patent Grant.
590fadc40 Fix compile warning on CLANG
47b874398 Make Compaction class easier to use
753dd1fdd Fix valgrind issues in memtable_list_test
697380f3d Repairer documentation improvement.
2f66d7f92 Add LinkedIn back to USERS.md
0feeee643 Fix memtable_list_test
7b9581bc3 Fixed xfunc related compile errors in ROCKSDB_LITE
fabc11569 MemTableList tests
9741dec0e Fix a compile error in ROCKSDB_LITE in db/db_impl.cc
465b25ca9 "make commit-prereq" doesn't really build ROCKSDB_LITE
d2a056241 Fix a compilation error in ROCKSDB_LITE in db/internal_stats.h
316ec80bf fault_injection_test: add a test case to cover log syncing after a log roll
ed229a0de Fixes for readcache-flashcache
91df4e969 Remove use of whole-archive to include jemalloc
84c5bd7eb Add thread-safety documentation to MemTable and related classes
ee9bdd38a Script to check whether RocksDB can read DB generated by previous releases and vice versa
2b019a151 Enabling checksum in repair db as it should have been.
b1bbdd791 Create EnvOptions using sanitized DB Options
edbb08b5d Fix Makefile
199313dc3 build: create .o files specifically for java-related targets
b118238a5 Trivial move to cover multiple input levels
e7adfe690 Fix formatting of USERS.md
4e7543dcf Add USERS.md
58346b9e2 Log writer record format doc.
db6569cd4 Fix the compilation error in flashcache.cc on Mac
cba592001 build: don't use a glob for java/rocksjni/*
c66483c13 Fix github issue #563
de22c7bd1 Integrate Jenkins with Phabricator
f12614070 Fix TSAN build error of D36447
824e64634 Adding another NewFlashcacheAwareEnv function to support pre-opened fd
5e067a7b1 Clean up compression logging
e3ee98b38 run 'make check's rules (and even subtests) in parallel
a45e7581b Avoid naming conflict of EntryType
3be82bc89 Add p99.9 and p99.99 response time to benchmark report, add new summary report
953a885eb A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge
d2a92c13b avoid returning a number-of-active-keys estimate of nearly 2^64
a7ac6cef1 Fix level size overflow for options_.level_compaction_dynamic_level_bytes=true
089509b84 db_test: clean up sync points in test cleaning up
afbafeaea Disallow trivial move if compression level is different
d0695f3e2 Fix crash caused by opening an empty DB in readonly mode
51c8133a7 Fix make unity build compiler warning about "stats" shadowing global variable
df71c6b9e Script to trigger jenkins test
38a01ed1b Update COMMIT.md
76d63b452 Fix one non-determinism of DBTest.DynamicCompactionOptions
b23bbaa82 Universal Compactions with Small Files
2511b7d94 Makefile minor cleanup
1bd70fb54 Add --stats_interval_seconds to db_bench
fd3dbef22 Clean up old log files in background threads
99ec2412e Make the benchmark scripts configurable and add tests
2158e0f83 Fix clang build
d61cb0b9d db_bench can now disable flashcache for background threads
1c47c433b build: always attempt to update util/build_version.cc
e018892bb Formalize the DB properties string definitions.
cfa576402 Make auto_sanity_test always use the db_sanity_test.cc of the newer commit.
e9fddb7a2 Merge pull request #560 from xiaoxichen/patch-1
bcd8a71a2 Fix interger overflow on i386 arch
030859eb5 Dump compression info on startup
3539e0644 Merge pull request #558 from aamihailov/master
a3e4b3248 fix compilation error (same as fix#284)
ff1ff7c62 TSAN: avoid new link failure with -pg
39d508e34 Add a missing section title in HISTORY.md
727684bf9 Fixed a typo in RocksDBSample.java
ca25e86ef build: cause the "check" rule to depend on $(PROGRAMS)
986bdc680 Merge branch 'master' of github.com:facebook/rocksdb
1e57f2bf2 Fix build
2d417e52d Update HISTORY.md for 3.10.0
cbd6a2073 Merge branch 'master' of github.com:facebook/rocksdb
211ca26ae Fixing build issue
2495f9396 Merge pull request #555 from pshareghi/staticLZ4
f06de5f23 Merge pull request #556 from fyrz/RocksJava-Maven-Fix
cd987c383 Fix compile error when NROCKSDB_THREAD_STATUS is not used.
8f104ae5e [RocksJava] Maven build fix
3d1a924ff Adding stats for the merge and filter operation
afc51649e Merge pull request #546 from fyrz/RocksJava-MemEnv
4806cc126 Added static lz4 support for roccksjavastatic
6284eef4c Merge pull request #545 from fyrz/RocksJava-Level-Compression
248c063ba Report elapsed time in micros in ThreadStatus instead of start time.
315abac94 Undeprecate GetLiveFiles()
a057bb2a8 Improve ThreadStatusSingleCompaction
868968c8e Merge branch 'master' of github.com:facebook/rocksdb
689391406 Make SSTDumpTest.GetProperties less noisy
8d8656243 Merge pull request #551 from fyrz/RocksJava-JavaDoc-Fix
f8c505b23 Merge pull request #550 from fyrz/RocksJava-Sample-Fix
46443bfa9 [RocksJava] Add missing JavaDoc annotations
864b7e88f [RocksJava] Java sample bugfix
2b2394cbb [RocksJava] DBBenchmark option for RocksMemEnv
fd8804f97 [RocksJava] Expose MemEnv in RocksJava Summary: In 3.10 the C++ code was extended with a MemEnv implementation. This is now also available in RocksJava.
004b89fba [RocksJava] Add compression per level to API
51da3aab4 Merge pull request #536 from fyrz/RocksJava-32-Bit-adjustment
b7e88cfb5 Merge pull request #543 from fyrz/RocksJava-Logger-Comment
38d286f14 Clean-up WAL directory before running db_test
28bc6de98 rocksdb: print status error message when (ASSERT|EXPECT)_OK fails
9405b5ef8 rocksdb: Remove #include "util/string_util.h" from util/testharness.h
220d0dff7 rocksdb: Remove #include "util/random.h" from util/testharness.h
b088c83e6 Don't delete files when column family is dropped
17ae3fcbc rocksdb: initial util/testharness clean up
39f4271be [RocksJava] Enhanced Logger comment
5615e23d8 [RocksJava] 32-Bit adjustments
836bcc2f0 Merge pull request #532 from fyrz/RocksJava-LevelCompactionDynamicLevelBytes
67d995808 rocksdb: fix make unity
52e0f3353 Clean up compactions_in_progress_
6b626ff24 rocksdb: change db_test::MultiThreadedDBTest as value parameterized test.
9720ea4de A build option to run through all check-in requirements.
0831a3599 Add a DB Property For Number of Deletions in Memtables
f7ed65464 Fix RocksJava test failure due to deprecation of table_cache_remove_scan_count_limit
51301b869 Enable dynamic changing of rate limiter's bytes_per_second
652db51a3 Fix compilation error in rocksjni/write_batch_test.cc
dfccc7b4e Add readwhilemerging benchmark
c345d1ee8 [RocksJava] Integrated changes for D34857
12350115d [RocksJava] Added LevelCompactionDynamicLevelBytes to Options
230e68727 Fix TSAN failue in env_test
155d468c5 Using chrono as a fallback
81345b90f Create an abstract interface for write batches
46214df4a Only run db_test in Travis
c88ff4ca7 Deprecate removeScanCountLimit in NewLRUCache
b4b69e4f7 rocksdb: switch to gtest
413e35273 Merge pull request #540 from dalgaaf/wip-da-fix-elif
969aa806b util/xfunc.h: fix #elif check for NDEBUG
87c7d49d6 util/env_posix.cc: fix #elif check for __MACH__
c86e5d7b9 stack_trace.cc: fix #elif check for OS_MACOSX
98c37fda5 Remove unused parameter in CancelAllBackgroundWork
9fd6edf81 rocksdb: Replace ASSERT* with EXPECT* in functions that does not return void value
d4d42c02e Fixed clang build in env.h
b2b308652 Speed up rocksDB close call.
a7aba2ef6 rocksdb: Add gtest
95344346a rocksdb: Small refactoring before migrating to gtest
bd4963e64 rocksdb: update reference to the latest version of clang dev in fbcode
56337faf3 Fix compaction IO stats to handle large file counts
eafa1bfc3 Merge pull request #529 from fyrz/RocksJava-Logger
ac03c4d51 Merge pull request #522 from fyrz/RocksJava-Optimize-Filters-For-Hits
c6967a1a5 Make RecordIn/RecordOut human readable
c8da67032 Stop printing per-level stall times.
04778a94c [RocksJava] OptimizeFiltersForHits
57f2a00c6 RocksJava - JNI Logger callback
814627af3 [RocksJava] Change log level at runtime in custom logger
a3bd4142f [RocksJava] Logging JNI callback
58878f1c6 Switch to use_existing_db=1 for updaterandom and mergerandom
12134139e Fixed the unit-test issue in PreShutdownCompactionMiddle
fd1b3f385 Fix the issue in PreShutdownMultipleCompaction
56c4a9c76 Fix compile warning in thread_status_util.h on Mac
417367c42 Fix SIGSEGV when not using cache
e25ff039c Prevent slowdowns and stalls in PreShutdown tests
f69071265 Speed up db_bench shutdown
c1b3cde18 Improve the robustness of ThreadStatusSingleCompaction
8c12426c9 Fix the deadlock issue in ThreadStatusSingleCompaction.
b16ead531 DBTest.DynamicLevelCompressionPerLevel should not run without snappy support
a5e60bafc Fix a typo / test failure in ThreadStatusSingleCompaction
cb2c91850 Don't run some tests is snappy is not present
c594b0e89 Allow GetThreadList() to report operation stage.
2623b2cf6 Include chrono
52d8347a9 EventLogger
756532daf Merge pull request #524 from fyrz/RocksJava-Test-Fix
47a2b3a40 Merge pull request #534 from fyrz/RocksJava-Fix-BrokenJacocoReport
9d22a1f13 Allow negative Wnew
2c84303aa Merge pull request #528 from fyrz/RocksJava-NativeLibraryLoader
2dc636f62 [RocksJava] Fix broken jacoco report
f210b0f6c [RocksJava] Fix NativeLibraryLoader
3ebebfccd Prevent xxhash symbols from polluting global namespace
53996149d Removing unnecessary kInlineSize
b411d0603 Prevent stalls in preshutdown tests
1d43bc41f Fixing segmentation fault in db_bench
e9de8b65a Change the way options.compression_per_level is used when options.level_compaction_dynamic_level_bytes=true
2b785d76b Fixed a bug where CompactFiles won't delete obsolete files until flush.
6f5579868 Fixed a compile error in db_bench in mac.
05d92efa7 Add convenience.cc to src.mk
2884b100b db_bench: Better way to randomize repeated read keys in -read_random_exp_range
284be570c Provide a mechanism to inform Rocksdb that it is shutting down
2ddf53b2c Get OptimizeFilterForHits work on Mac
480b28476 Fix make_new_version.sh
89597bb66 Allow GetThreadList() to report the start time of the current operation.
37921b499 db_bench: Add Option -read_random_exp_range to allow read skewness.
485ac0dbd Add rate_limiter to string options
e126e0da5 Single threaded tests -> sync=0 Multi threaded tests -> sync=1 by default unless DB_BENCH_NO_SYNC is defined
dc4532c49 Add --thread_status_per_interval to db_bench
34c75e984 fix-up patch: avoid new link error
ebc647de8 build: fix missing dependency problems
492f6d27e Fix a segfault in fbson under Mac OS X compiler
0d13bbe27 RocksJava] Fix ColumnFamily tests
67533809f [RocksJava] Fixed CompactionTest
1b7b997b8 [RocksJava] Remove MaxValue from Statistics
f862b3812 [RocksJava] Fix cleanup in tests
a01b59259 [RocksJava] DefaultColumnFamily Memory Fix
22c73d15b [RocksJava] Fix ColumnFamily tests
694988b62 Fix a bug in stall time counter. Improve its output format.
b8d23cdcb Revert chrono use
db0373934 options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically.
f29b33c73 Add functionality to pre-fetch blocks specified by a key range to BlockBasedTable implementation.
c4bd03a97 Fix typo in log message
3cf7f353d Instrument memtable seeks
216a9e16f Fix compile
b9ff6b050 Fix a bug in ReadOnlyBackupEngine
afa8156af adding stdlib to fbson
f9c14a42e Fix compile on Mac
a9f0e2b5b Fix compile
e7c434c36 Add columnfamily option optimize_filters_for_hits to optimize for key hits only
ba9d1737a RocksDB on FreeBSD support
4ba119df5 rocksdb: Update path to dev clang in fbcode
8984e5f84 Fix race in sync point.
03b432d4b rocksdb: Fix uninitialized use error
ccef2a766 Merge pull request #518 from fyrz/RocksJava-Native-Library-Loader-Fix
9fcf1a7b0 [RocksJava] RocksJava Testcases
03bbf718c Return fbson
62247ffa3 rocksdb: Add missing override
1e06a4068 Support builds for MongoDB+RocksDB
b74ad6632 Merge pull request #508 from fyrz/RocksJava-Final-Patch
d9f4875e5 Disable pre-fetching of index and filter blocks for sst_dump_tool.
182b4ceac Limit key range to number of keys, not number of writes
a360bb61b Merge pull request #516 from fyrz/RocksJava-Update-Statistics-To-3.10
8c7684474 [RocksJava] Updated TickerTypes and Histogram to 3.10
7f0c77cb3 [RocksJava] Integrated changes from D33417
819e787bb [RocksJava] Final usage correction
5139e678b Upgrade compiler in Travis
4ade89962 Fix compile error on MacOS.
ace3d8506 Revert "Unused managed iterator"
7b8f348e5 Attempt at fixing travis issue
d85993998 Merge pull request #506 from fyrz/RocksJava-Raw-Use
30e93c9b9 Merge pull request #505 from fyrz/RocksJava-Redundant-Modifier
217854dc4 Introduce DISABLE_WARNING_AS_ERROR in Makefile
bd339a979 Unused managed iterator
174a79c99 LevelDb include guard replaced with #pragma once
6fdda8ac4 rocksdb: changes to support 'make analyze' in Jenkins
96d989f70 catch config errors with L0 file count triggers
62f7a1be4 rocksdb: Fixed 'Dead assignment' and 'Dead initialization' scan-build warnings
5636c4a80 Verbose build in travis
a047409ae Fixed a bug in the test case
4f514a53d build: enable more compiler warnings
a2b911b63 inputs: restore "const" attribute removed by D33759
1b4082581 mark as unused some variables with cpp-derived names
c6d54b503 fix erroneous assert: cast kBlockSize (of type unsigned int) to "int"
aa5d8e6d9 table_test.cc: add missing 5th arg in TestArgs initializer
c37937a9c maint: remove extraneous "const" attribute from return type
9283c7afd build: remove always-true assertions
06a766de5 Adding Flush to AutoRollLogger
92416fa7f Fix mac build
96ab15d30 GetOptionsFromString + fixes to block_based_table_options
73711f956 rocksdb: Fix scan-build bug 'Memory leak' in db/db_bench.cc
98870c7b9 rocksdb: Fix scan-build memory warning in table/block_based_table_reader.cc
a42324e37 build: do not relink every single binary just for a timestamp
d45a6a400 Add rocksdb.num-live-versions: number of live versions
11581b741 build: abbreviate AR command, too
b8ac71ba1 Revert "Fbson to Json"
7ce1b2c19 Fbson to Json
7d817268b Managed iterator
b4b8c25a5 build: factor out AM_LINK command (trivial)
dc885c6e9 build: make "make" output readable by default
a37b46ae1 build: fix Makefile inconsistencies (trivial)
55277c328 build: remove unused rules: rocksdb_shell, DBClientProxy_test
3ad6b794c rocksdb: Fix 'Division by zero' scan-build warning
12753130e Remove ThreadStatusMultiCompaction test
f0c36da6e Add thread_status_util_debug.cc back
daebb1f91 build: running "make" with no arguments must not "uninstall"
e60bc99fe Allow GetThreadList to reflect flush activity.
b9a0213cd build: fix unportable Makefile syntax
4e4b85784 rocksdb: Fix scan-build 'Called C++ object pointer is null' and 'Dereference of null pointer' bugs
b3fd16226 build: remove unportable use of sed in favor of $(CXX)'s -MT
e7ea51a8e Introduce job_id for flush and compaction
6a0e737d9 [RocksJava] Raw use of parametrized class
439701270 [RocksJava] Redundant access-modifier in interfaces
2d62e8051 Merge pull request #504 from fyrz/RocksJava-Flush-Correction
eaf39568e [RocksJava] FlushOptions Correction
5d1151deb Added simple monitoring script to monitor overusage of memory in db_bench
5f00af457 DBTest.DestroyDBMetaDatabase: create DB directories if not exists
68af7811e Remember whole key/prefix filtering on/off in SST file
fd5970b45 Merge pull request #503 from weiweisd/master
513ad866b modify double type euqal compare in json_document.cc
933973dc2 Merge pull request #495 from fyrz/RocksJava-CF-Name-Byte-Handling
5e8e453d5 [RocksJava] Integrated changes from D33165
677d02427 [RocksJava] CF Name shall handle bytes correctly
d1cafc089 Merge pull request #1 from facebook/master
6d6305dd7 Perf Context to report DB mutex waiting time
863009b5a Fix deleting obsolete files #2
1851f977c Added RocksDB stats GET_HIT_L0 and GET_HIT_L1
91ac3b206 Print DB pointer when opening a DB
bee4e5124 Merge pull request #492 from fyrz/logger-logv-virtual
cfe8837e4 Switch logv with loglevel to virtual
aaceef363 Fix formatting
ee4aa9a0e Merge pull request #481 from mkevac/backupable
82faa377a added simple example for db restore from backup
d090330c8 fixed c_simple_example and added some comments
965130830 renamed backup to backup_and_restore in c_test for clarity
bbb52b21f Merge pull request #483 from adamretter/restructure-java-build
7e50ed8c2 Added some more wrappers and wrote a test for backup in C
218c3ecea Fix std::cout data race
8f679c290 Merge branch 'master' of github.com:facebook/rocksdb
da9cbce73 Add Header to logging to capture application level information
2a979822b Fix deleting obsolete files
8e83a9d31 Add a missing field for STATE_MUTEX_WAIT to global_state_table
6f1013035 Fix DestroyDB
7de4e99a8 Revert "Fix wal_dir not getting cleaned"
9a52e06a0 Add GetID to ColumnFamilyHandle
181191a1e Add a counter for collecting the wait time on db mutex.
f36d394ae Fix wal_dir not getting cleaned
53ae09c39 db_test: fix a data race in SpecialEnv
fe9f69119 Fix fault_injestion_test
b37f5ffc7 Put db_test back to TESTS in Makefile
108470e96 Fix stack trace on mac
3e53760fc Fix compaction_picker_test
e39f4f6cf Fix data race #3
e63140d52 Get() to use prefix bloom filter when filter is not block based
678503ebc Add utility functions for interpreting ThreadStatus
4d98e2935 rocksdb: Enable scan-build static analysis
756e1f151 Remove unused util/thread_event_info.h
dad98dd4a Changes for supporting cross functional tests for inplace_update
9898f6398 Divide test DBIteratorTest.DBIterator to smaller tests
829363b44 Options::PrepareForBulkLoad() to increase parallelism of flushes
b04408c47 Fix unity build
8d3819369 NewIteratorWithBase() for default column family
2c2d5ab7e Fix compile warning in util/xfunc.h
0b8dec717 Cross functional test infrastructure for RocksDB.
868bfa403 Merge pull request #488 from ekg/master
9900f3821 Merge pull request #484 from fyrz/RocksJava-Release-MD-change
e6eaf938c remove old debugging message (#487)
f33f3955e Moved Java Samples main classes into samples/src/main/java
ad325517f Update test lib versions and maven plugin versions
d6187d07b Maven can now build a standard project layout
157768890 Moved Java Benchmark main classes into benchmark/src/main/java
dd8d5471e Adjustment to NativeLibraryLoader to allow native library to be loaded from either java.library.path or from extracting from the Jar. Means that the test in the build do not need to rely on the Jar, useful when creating similar builds (and executing tests) from Maven
353db6dae Moved Java main classes into src/main/java
98cb501bc Moved Java test classes into src/test/java
7479a62a7 Release.md - Remove version change instrcution
4a4e4279f Update HISTORY-JAVA.md
384cb6619 Merge pull request #480 from fyrz/RocksJava-Deprecate-SkipLogError
ca52a67cf [RocksJava] Deprecate setSkipLogErrorOnRecovery
114d21878 Merge pull request #479 from fyrz/RocksJava-Snapshot-Sequence-Number
cb5c3159f [RocksJava] Snapshot - GetSequenceNumber
ea189b320 Merge pull request #474 from fyrz/RocksJava-GetUpdatesSince
391f85fc8 [RocksJava] Incorporated changes for D32151
68cd93b87 [RocksJava] GetUpdatesSince support
caedd40dd [RocksJava] Adjusted auto pointer
b39006e3d [RocksJava] enable/disable File deletions
9a456fba2 [RocksJava] GetUpdatesSince support
d3a736761 Merge pull request #482 from fyrz/RocksJava-TTL-Fix
939bb3659 [RocksJava] Fix ColumnFamily name alloc in TTL DB
86e2a1eee Allow creating backups from C
db9ed5fdb Unaddressed comment in previous diff. Change only in code comments.
5917de0ba CappedFixTransform: return fixed length prefix, or full key if key is shorter than the fixed length
6c6037f60 Expose Snapshot's SequenceNumber
2fd8f750a Compile MemEnv with standard RocksDB library
173c52a97 Fix build on older compilers -- emplace() is not available
d07fec3bd make DBTest.SharedWriteBuffer to pass MockEnv
4bdf38b16 Disable FlushSchedule when running TSAN
e84299c76 Fix bug recently introduced in MemFile::Lock()
e5aab4c2b Fix data race in HashLinkList
2113ecd3c Merge pull request #472 from fyrz/RocksJava-Cleanup
10af17f3d fault_injection_test: add a unit test to allow parallel compactions and multiple levels
0c4d1053d Fix data race #5
cc0d8be01 [RocksJava] Integrated review comments (D32145)
5257c9c42 Merge pull request #452 from robertabcd/backupable-mem
560ed402b [minor] fprintf to stderr instead of stdout in test
551a41df3 Merge pull request #476 from alabid/alabid/add-to-simple-example
d2a2b058f fault_injection_test: to support file closed after being deleted
f8f040ccc Updated .gitignore to ignore *~ files and example object files
e8bf2310a Remove blob store from the codebase
ea7d0b943 Added WriteBatch block to simple_example.cc
d6c7300cc Fixed a compile warning in clang in db/listener_test.cc
f9758e012 Add compaction listener.
e919ecedf SuperVersion::Unref() to use sequential consistency to decrease ref counting
4c49fedaf Use ustricter consistency in thread local operations
1b43ab58d fault_injection_test: add more logging and makes synchronization slightly stronger
ca2b00277 [RocksJava] Cleanup portal.h & tests
f8dc5c459 [RocksJava] Add missing test to Makefile
b3c133148 [RocksJava] Removed todo comment in portal.h
7ffcc457f [RocksJava] Cleanup portal.h
c4fb83441 Update the comment for the removal of mac-install-gflags.sh
be8f0b12e Rename DBImpl::log_dir_unsynced_ to log_dir_synced_
c1de6c42a fault_injection_test: add a test case to drop random number of unsynced data
d888c9574 Sync WAL Directory and DB Path if different from DB directory
58f34edfc Fix valgrind
f1c886247 Fix data race #1
b08b2fe73 Merge pull request #471 from fyrz/RocksJava-Fix-NativeLibraryLoader
e61f38e5a [RocksJava] Fix native library loader
26b50783d Fix assert in histogramData
42189612c Fix data race #2
f5a839835 Fix archive WAL race conditions
43ec4e68b fault_injection_test: bring back 3 iteration runs
c2e8e8c1c Fix two namings in fault_injection_test.cc
b4c13a868 fault_injection_test: improvements and add new tests
a52dd0024 Fix ASAN failure with backupable DB
910186c27 Return the build with 4.8.1
401d4205e Add thread sanitizer
b068f0a67 Upgrade our compilers
a76d92862 Merge pull request #466 from fyrz/RocksJava-Support-ReadOptions-Iterator
bef7821f0 [RocksJava] ReadOptions support in Iterators
3b494a610 Make options_test runnable on ROCKSDB_LITE
912c52e82 Merge pull request #465 from fyrz/RocksJava-BlockBasedTable-FormatVersion
cd4c07197 Update HISTORY.md for GetThreadStatus() support on compaction.
46a7048dc Reduce false alarm in ThreadStatusMultipleCompaction test
aed028698 Merge pull request #462 from fyrz/RocksJava-JNI-allocation-correction
e5df90f5d Fix comment (minor)
dd53428f8 Incorporated review comments
908258a4f [RocksJava] BlockBasedTableConfig 3.10
2efe22849 [RocksJava] Incorporated changes for D31809
4e48753b7 Sync manifest file when initializing it
e204a5a16 [RocksJava] ColumnFamily name JNI correction
96264784d [RocksJava] ColumnFamily name JNI correction
ae82849bc Fix build failure
423dee841 Abort db_bench if Get() returns error
206237d12 DBImpl::CheckConsistency() shouldn't create path name with double "/"
5e98e5324 Merge pull request #458 from fyrz/RocksJava-TTLDB-Support
4ffe0be41 [RocksJava] Integrated changes for D31449
e82856754 [RocksJava] Integrated changes from D31449
859c54a03 [RocksJava] TTL-Support
5ff8aec4d [RocksJava] TTL Support
ca47da9e6 [RocksJava] TTL-Support
1190ebe5a Merge pull request #461 from fyrz/RocksJava-DirectSlice-Fix
ea25ff715 [RocksJava] Integrated proposed simplificiation
d68e83c35 [RocksJava] DirectSlice String termination fix
0ddf5f73e memenv: normalize file path
4d9d5955a Merge pull request #464 from fyrz/RocksJava-Various-Fixes
ceaea2b72 Adding prefix_extractor string config parameter
3d628f8f2 Update format_version comment
155bec4cf fallocate also tests FALLOC_FL_KEEP_SIZE
c75c02e7a [RocksJava] WriteBatchWithIndexTest fix
c787fb50b [RocksJava] JavaDoc errors in Java8
b229f970d Remove Compaction::ReleaseInputs().
f2ddb8b45 Fix for bug where GeoDB accesses key after next modification of iterator
d10f1de2b Ported LevelDB's fault_injection_test
2bb059007 Change db_stress to work with format_version == 2
9ab5adfc5 New BlockBasedTable version -- better compressed block format
2355931c6 Merge pull request #450 from adamretter/writebatch-with-index
3d246c89c Abstract duplicate code on key and value slice objects into generic methods
2d0dd8db3 Implement WBWIRocksIterator for WriteBatchWithIndex in the Java API
de678b288 Abstractions for common iterator behaviour
e01acb3a0 Test for WriteBatchWithIndex#newIterator()
56f24941a Simplify the Java API by permitting WriteBatchWithIndex to be provided straight to RocksDB#write
95d5f9848 Test for RocksDB#write(WriteBatchWithIndex)
ef5b34dee Implement WriteBatchWithIndex in the Java API
c6e554561 Abstractions for common write batch behaviour
be905491b Test for WriteBatchWithIndex#newIteratorWithBase(org.rocksdb.RocksIterator)
2241e3f4d Extract the interface for a RocksIterator
a8cfa7ace Extract the interface for a WriteBatch
45e43b81d Adds support for db->DefaultColumnFamily() to the Java API
516a04267 Add LZ4 compression to sanity test
2ccc54301 Merge pull request #460 from neutronsharc/master
2a7bd0ea4 Remove duplicated method declarations in C header.
bb128bfec More accurate message for compaction applied to a different version
96b8240bc Support footer versions bigger than 1
53f615df6 Fix clang build
02b30202c Merge pull request #455 from Andersbakken/stdlib_fix
2159484dd Remove two unnecessary blank lines in db/db_test.cc
e7dd88c57 Merge pull request #441 from fyrz/RocksJava-ColumnFamilyDescriptor-Alignment
d2c018fd5 Make ThreadStatusMultipleCompaction more robust.
bf9aa4dfc Improve GetThreadStatus to avoid false alarm in some case.
c91cdd59c Allow GetThreadList() to indicate a thread is doing Compaction.
402c1152a Fix c_simple_example
a9ea65d65 Build with clang 3.5 on Linux.
23ad5f401 [RocksJava] Incorporated changes for D30525
0aab1005f [RocksJava] ColumnFamilyDescriptor alignment with listColumnFamilies
15d2abbec Fix build issues
abb9b95ff Move compression functions from port/ to util/
9132e52ea DB Stats Dump to print total stall time
93b35c299 Merge pull request #453 from fyrz/SimpleCExampleSigSegv
628a67b00 Reduce memory footprint in backupable db.
ef3901642 Fixed memory issue in c_simple_example
b89d58dfa :%s/build_config/make_config
242b9769c Memtablerep Benchmark
73ee4feba Add comments about properties supported by DB::GetProperty() and DB::GetIntProperty()
2dca48f55 Merge pull request #451 from StanislavGlebik/document_db_improvement
4b57d9a82 Fixed negative numbers comparison in DocumentDB
9ef59a09a VersionSet::AddLiveFiles() to assert current version is included.
4d16a9a63 VersionBuilder to optimize for applying a later edit deleting files added by previous edits
7731d51c8 Simplify column family concurrency
07aa4e0e3 Fix compaction summary log for trivial move
9d5bd411b benchmark.sh won't run through all tests properly if one specifies wal_dir to be different than db directory.
62ad0a9b1 Deprecating skip_log_error_on_recovery
fa0b126c0 Fix corruption_test -- if status is not OK, return status -- during recovery
d7b4bb62a Fail DB::Open() on WAL corruption
9619081d9 Merge pull request #449 from robertabcd/improve-backupable
49376bfe8 Fix errors when using -Wshorten-64-to-32.
a8c5564a9 Do not issue extra GetFileSize() calls when loading BackupMeta.
caa1fd0e0 Improve performance when loading BackupMeta.
e9ca35815 Fix CLANG build for db_bench
bf287b76e Add structures for exposing thread events and operations.
a801c1fb0 db_bench --num_hot_column_families to be default off
2067058a6 Dump routine to BlockBasedTableReader (valgrind)
ddc81440d db_bench to add an option as number of hot column families to add to
a944afd35 Fixed a compile error in db/db_impl.cc on ROCKSDB_LITE
7ea7bdf04 Dump routine to BlockBasedTableReader
ae508df90 Clean up compile for c_simple_example
b62300961 Fix compile of compact_file_example
ded26605f Merge pull request #444 from adamretter/java-api-fix
98490bccf Fix the build on Mac OS X
4d9972974 Merge pull request #443 from behanna/master
5045c4394 add support for nested BlockBasedTableOptions in config string
d232cb156 Fix the build with -DNDEBUG.
45bab305f Move GetThreadList() feature under Env.
4fd26f287 Only execute flush from compaction if max_background_flushes = 0
0acc73881 Speed up FindObsoleteFiles()
d8c4ce6b5 Merge pull request #442 from alabid/alabid/fix-example-typo
949bd71fd fix really trivial typo
f8999fcf3 Fix a SIGSEGV in BackgroundFlush
ade4034a9 MultiGet for DBWithTTL
fdb6be4e2 Rewritten system for scheduling background work
a3001b1d3 Remove -mtune=native because it's redundant
e27c84522 Merge pull request #437 from fyrz/RocksJava-SliceTests-Fixes
1fed1282a [RocksJava] Incorporated changes D30081
5b9ceef01 [RocksJava] JavaDoc correction
5fbba60b6 [RocksJava] Incorporated changes D30081
b0230d7e0 [RocksJava] Incorporate additions for D30081
b015ed0ca [RocksJava] Slice / DirectSlice improvements
4d422db01 Merge pull request #430 from adamretter/increase-parallelism
04c4e4969 Merge pull request #411 from fyrz/RocksJava-RangeCompaction
62d19b7b5 Merge pull request #427 from haneefmubarak/c-examples
28424d734 style fixes in c example
7198ed5a2 Handle errors during pthread calls
91c58752f error detection and memory leaks in c example
25f70a5ab Avoid unnecessary unlock and lock mutex when notifying events.
7661e5a76 Move the file copy out of the mutex.
17e84f215 Rudimentary test cases for setIncreaseParallelism
eda0dcdd9 Exposed IncreasedParallelism option to Java API as setIncreasedParallelism
efc94ceb2 [RocksJava] Incorporated changes for D29283
69188ff44 [RocksJava] CompactRange support
48adce77c [RocksJava] CompactRange support
153f4f071 RocksDB: Allow Level-Style Compaction to Place Files in Different Paths
06eed650a Optimize default compile to compilation platform by default
cef6f8439 Added 'dump_live_files' command to ldb tool.
7ab1526c0 Add an assert and avoid std::sort(autovector) to investigate an ASAN issue
74b3fb6d9 Fix Mac compile errors on util/cache_test.cc
d7a486668 Improve scalability of DB::GetSnapshot()
ee95cae9a Modifed the LRU cache eviction code so that it doesn't evict blocks which have exteranl references
0ab0242f3 VersionBuilder to use unordered set and map to store added and deleted files
e93f044d9 add range scan test to benchmark script
cb82d7b08 Fix#434
046ba7d47 Fix calculation of max_total_wal_size in db_options_.max_total_wal_size == 0 case
1b7fbb9e8 Update HISTORY.md for release 3.9
635c61fd3 Fix problem with create_if_missing option when wal_dir is used
2871bc7bc Merge pull request #422 from fyrz/RocksJava-Quality-Improvements
8c5781666 Add -fno-exceptions flag to ROCKSDB_LITE.
1f04066ca Add DBProperty to return number of snapshots and time for oldest snapshot
6436ba6b0 Provide mechanism to restart tests from previous error
d84b2bade Replace exception by abort() in dummy HdfsEnv implementation.
9260e1ad7 Bump version to 3.9
8f4e1c1c9 Remove the compability check on log2 OS_ANDROID as it's already blocked by ROCKSDB_LITE
c4a7423c1 Replace runtime_error exception by abort() in thread_local
a94d54aa4 Remove the use of exception in WriteBatch::Handler
a5d4fc0a2 Fix compile warning in db_stress
1a8f4821a Replace exception by assertion in autovector
97c194088 Fix compile warning in db_stress.cc on Mac
5f719d720 Replace exception by setting valid_ = false in DBIter::MergeValuesNewToOld()
c0dee851c Improve formatting, add missing newlines
815f638cd Fix java build
32a0a0384 Add Moved(GB) to Compaction IO stats
a14b7873e Enforce write buffer memory limit across column families
3e684aa68 Integrated changes from D29571
37d73d597 Fix linters
a15169f2e Fixed a Lint problem
b7f9e644c [RocksJava] Quality improvements
e002a6122 [RocksJava] Comparator tests for CF
335e6ad5c [RocksJava] Remove obsolete dbFolder cleanup
b036804ac RocksJava - FindBugs issues
9a632b4a9 Merge pull request #429 from fyrz/RocksJava-MacOSX-strip-fix
b42667506 [RocksJava] MacOSX strip support
e463cb0bc Merge pull request #424 from eile/master
91d898163 Tweak Makefile for building on BG/Q
c6f31a289 minor memory leak in C example
703ef66a8 Merge pull request #426 from fyrz/RocksJava-Restore-PrecisionFix
ac4ed1e30 fix examples/makefile for C example
d7f5ccb0c add c example to makefile and fix "make clean"
9c34d5e36 fix type in C simple example
0a9a7e753 added C version of simple_example
bcf908689 Block Universal and FIFO compactions in ROCKSDB_LITE
67cb7ca75 [RocksJava] Fixed MacOS build of RocksJava
b8136a7d2 Merge pull request #398 from fyrz/RocksJava-CreateCheckPoint
533592a27 Merge pull request #401 from fyrz/RocksJava-Sigsegv-MergeOperatorName
73d72ed5c Block ReadOnlyDB in ROCKSDB_LITE
e47f0fa9e Merge pull request #425 from adamretter/macosx-clean-fix
ff0cb90d1 Do not delete Java Fatal Error Log, developers may still want these for reference
2a792cd30 There will also be a librocksdbjni-osx.jnilib.dSYM folder on MacOSX builds to be deleted
beb74c14c Fix travis-build error
a486352e0 Merge pull request #423 from zerebubuth/c_iterate_upper_bound
26109d487 Store upper bound `Slice` with the same lifetime as the `ReadOptions` so that we can provide a pointer to it.
a97314219 Fix compile error in ROCKSDB_LITE
9d5019327 Replace log2 by implementing Log2 in options_builder
805bac6d2 Add test for upper bounds on iterators using C interface.
f193deea3 [RocksJava] Addressed comments in D28971
94f70a86b [RocksJava] Incoroporated changes for D29013
a280af2a5 [RocksJava] Sigsegv fix for MergerOperatorByName
fcc2dfd9f [RocksJava] Support for stored snapshots
274ba6270 Block internal_stats in ROCKSDB_LITE
4f2e8bab5 Merge pull request #421 from fyrz/RocksJava-PrecisionFix
c4765dc10 [RocksJava] Fix precision problem in rocksjni
14788e181 Merge pull request #420 from rdallman/add-wal
88dd8d889 c api: add max wal total to opts
7e608e2fe Block plain_table_index.cc in ROCKSDB_LITE
13de000f0 Add rocksdb::ToString() to address cases where std::to_string is not available.
90ee85f8e Improve listener_test to avoid possible false alarm
2946e37a0 remove unreliable test in db/cuckoo_table_db_test.cc
9c7ca65d2 free builders in VersionSet::DumpManifest
7530c75ab Merge pull request #413 from saghmrossi/master
d699d7034 Make RocksDB compile without gflags
325722149 Fixes valgrind error in GetSnapshotLink. Free checkpoint now.
ada3d7873 Merge pull request #415 from fyrz/RocksJava-Makefile
569853ed1 Fix leak when create_missing_column_families=true on ThreadStatus
c4b65f70f [RocksJava] Makefile correction
141018016 Make arena use hugepage if possible
3a40c427b Fix db_bench on CLANG mode
9222a2d02 Fixed iOS build caused by GetThreadList feature.
aa31fc506 Improve listener_test by ensuring flushes are completed before assert.
7ec71f101 Provide default implementation of LinkFile, don't break the build
cd278584c Clean up StringSplit
d84069995 Fix mac compile
4f882924d Merge pull request #404 from fyrz/RocksJava-Backup-Restore-3.8
4b63fcbff Add enable_thread_tracking to DBOptions
bafce6197 first rdb commit
9e285d423 Added CompatibleOptions for compatibility with LevelDB Options
353307758 Add IOS_CROSS_COMPILE to macro guard for GetThreadList feature.
eecdebe65 Fixed the destruction order of static variables in ThreadStatusImpl.
004f416b7 Moved checkpoint to utilities
beabc6879 Fixed ~ThreadStatusImpl().
faa8d32be [RocksJava] Integrated changes from D29019.
3d78c7a8c [RocksJava] Lint adjustments
d7529b2de [RocksJava] Cleanup Backupable implementations
fa703efb2 [RocksJava] Improved BackupableDBTest
24fdc4741 [RocksJava] Backupable/Restorable DB update 3.8.0
9972f969e [RocksJava] BackupableDBOptions alginment + 3.8
fbc42a093 Fixed -Werror=unused-but-set-variable in thread_status_impl
a564be715 Fix asan error in thread_status_impl.cc
7165d1886 Fix clang compile error
d0c5f28a5 Introduce GetThreadList API
1fd1aecb3 Merge pull request #409 from fyrz/RocksJava-Make-cleanup
91c8dcefc [RocksJava] Strip library in publish
e7fcaa4d9 [RocksJava] JavaDoc is executed too often
2cd1794e4 [RocksJava] Make cleanup - Clean Target
be005e17b fix clang compilation
5e69f19c4 Merge pull request #405 from fyrz/RocksJava-Convenient-Options
55a344872 Merge pull request #408 from fyrz/Missing-include
9e9a83baf Missing header in build on CentOS
91ccc8ebe [RocksJava] Integrated changes in D29025
5249d0db5 [RocksJava] Convenience methods for Options
8d3f8f969 remove all remaining references to cfd->options()
1e4a45aac remove cfd->options() in DBImpl::NotifyOnFlushCompleted
517c28994 Options helper supports k, m, g, and t unit suffixes
c46c2be8d Merge pull request #397 from fyrz/RocksJava-GetIntProperty
8efd4bb42 [RocksJava] Improved comments in RocksDB class
5529c1ad1 [RocksJava] GetIntProperty in RocksDB
db59eeb61 Merge pull request #406 from fyrz/Build-Fix
e97f014b9 [RocksJava] JavaDoc corrections - Java8
98e59f981 Fixed a bug which could hide non-ok status in CompactionJob::Run()
ec24bd4e6 Merge pull request #402 from adamretter/bugfix-native-library-loader
d3c4a0f4a Improve the comment in InfoLogLevelTest.java
a77e97c53 Merge pull request #396 from fyrz/RocksJava-LogLevel
585c759cf Make sure to use the correct Java classloader for loading the RocksDB Native Library
c3915abba Minor tidyup and use Java 7 for file copying
a122a42bb Merge pull request #399 from fyrz/RocksJava-Version-to-3.8
b8d5e3f08 [RocksJava] MVN Build reads version from version.h
23295b74b Clean job_context
0ce38fe98 Fix signed/unsigned compile
e7960c03a Don't parallelize the build in travis
84af2ff8d Clean job context in DeleteFile
8a1bcc39c [RocksJava] Bump version to 3.8 in rocksjni.pom
5c04acda0 Explicitly clean JobContext
4947a0674 [RocksJava] Incorporated review comments D28947
07cd3c42a [RocksJava] LogLevel support in Options
26dc5da96 Fix compaction_job_test
5f583d2a9 Merge pull request #394 from lalinsky/cuckoo-c
353303a76 Merge pull request #380 from fyrz/RocksJava-Junit-Framework
3f9c95a51 [RocksJava] Minor lint correction
e46450da6 [RocksJava] Rebased + integrated CF tests
cd82beb0c [RocksJava] Merged in latest changes.
b6abab8b7 [RocksJava] Merged & rebased to HEAD
74057d6d2 [RocksJava] Improved tests within RocksJava
628e39e97 [RocksJava] Integrated review comments from D28209
a4b28c1ae [RocksJava] Extended Testcases
36f3a0bb8 [RocksJava] Integrated review comments from adamretter in D28209
b09268695 [RocksJava] Extended testcases
9bec23c41 [RocksJava] Test-framework integration
f617135d5 [RocksJava] Testcase improvements
1fe7a4c62 [RocksJava] Test-framework integration
04ca7481d Fix build
6c1b040cc Provide openable snapshots
9be338cf9 CompactionJobTest
c9fd03ec5 Update docs for NewAdaptiveTableFactory
e6c3cc657 Add very basic tests to make sure the C cuckoo table options compile and run
7fe247080 Update HISTORY.md for RocksJava
c44a29278 Add cuckoo table options to the C interface
136b8583b Merge pull request #395 from lalinsky/fix-env-test
94fa542f8 Update HISTROY.md for 3.8 release
a177742a9 Make db_stress built for ROCKSDB_LITE
746cfaac5 Relax the block count check on deallocation in env_test
f822129b3 Add a unit test for behavior when merge operator and compaction filter co-exist.
4161de92a Fix SIGSEGV
373c665ed Fix broken test in 31b02d.
772bc97f1 No CompactFiles in ROCKSDB_LITE
1d1a64f58 Move NeedsCompaction() from VersionStorageInfo to CompactionPicker
cd0980150 Add concurrency to compacting SpatialDB
3c92e5233 Fix include
25f273027 Fix iOS compile with -Wshorten-64-to-32
fa50abb72 Fix bug of reading from empty DB.
31b02dc21 Improve Backup Engine.
1033db29f Merge pull request #390 from fyrz/RocksJava-Cleanup
9a03da773 Merge pull request #375 from fyrz/RocksJava-ColumnFamilyOptions-Extension-3.6
d50c68e3a [RocksJava] JavaDoc cleanup warnings with Java8
079d942ea [RocksJava] Code-cleanup + Java7 warnings removed
9a255b95f [RocksJava] Sample and Default value
9d2ba2136 [RocksJava] Incorporated review comments
fa9cfc65f [RocksJava] Integrated Review comments from yhchiang in D28023
75010d208 [RocksJava] ColumnFamily custom Options API extension
0345c2156 [RocksJava] Extend Options with ColumnFamilyOptions implementation ColumnFamilyOptions implementation with tests [RocksJava] Extended ColumnFamilyTest
975949522 Fixed clang compile error in version_builder_test
581141935 Fixed GetEstimatedActiveKeys
1f621e6ab Fix additional -Wshorten-64-to-32 errros
767777c2b Turn on -Wshorten-64-to-32 and fix all the errors
113796c49 Fix NewFileNumber()
625e162c6 Merge pull request #393 from fyrz/RocksJava-Flush
fc6fcbab9 [RocksJava] Flush functionality
8e5547f64 [RocksJava] Makefile restructured
26f0a78b0 Merge pull request #363 from adamretter/write_batch-iterate
35c8c814e Make ForwardIterator::status() more efficient
d88568c68 Move -Wnon-virtual-dtor to c++ flags
c7ee9c3ab Fix -Wnon-virtual-dtor errors
746252197 Merge pull request #384 from msb-at-yahoo/compaction-filter-2-empty-changed-values
dd726a59e Bump Version Number to 3.8
4a3bd2bad Optimize usage of Status in CompactionJob
bcdb9671c Fix build
d904fbbb0 Addresed comments from code review https://reviews.facebook.net/D27567
eeb9cf6c4 Test for WriteBatchHandler
8fb4751d5 Iterator support for Write Batches
00211f9c5 Fix SIGSEGV in db_stresS
a4a2bfd6b Merge pull request #391 from Liuchang0812/fixmake
856059094 Merge pull request #389 from Liuchang0812/master
01a770637 remove unused target
dc3410463 Merge branch 'fix-example'
e7620536c fix make static_lib error
543df158c Expose sst_dump functionality as library call.
e3d3567b5 Get rid of mutex in CompactionJob's state
344edbb04 Fixed the shadowing in db/compaction.cc and include/rocksdb/db.h
b8b390342 Fixed compile error in db/db_impl.cc
b622ba5d6 Fixed compile error in db/flush_job.cc
642ac9d8a Fixed compile error in db/compaction.cc and db/compaction_picker.cc
68effa034 Fix -Wshadow for tools
844786189 Fixed -WShadow errors in db/db_test.cc and include/rocksdb/metadata.h
28c82ff1b CompactFiles, EventListener and GetDatabaseMetaData
5c9309053 Turn on -Wshadow
31342c400 Fix implicit compare
a0f887c9e Fix compile
53af5d877 Redesign pending_outputs_
ec101cd49 Correctly test both compaction styles in CompactionDeletionTriggerReopen
8d87467bb Make PartialCompactionFailure Test more robust again.
64d302d30 make DropWritesFlush deterministic
cd5c0925a Merge pull request #387 from fyrz/RocksJava-WShadow-Fix
c4bf07c24 [RocksJava] -WShadow improvements
e526b7140 Make PartialCompactionFailure Test more robust.
0c2be0de3 Turn on -Wshadow for travis
5fd33d26f Turn off -Wshadow
9f20395cd Turn -Wshadow back on
c02338a69 update HISOTRY.md for new release
367a3f9cb Improve DBTest.GroupCommitTest: artificially slowdown log writing to trigger group commit
b52b144c7 Merge pull request #386 from EugenePig/java8
ac95ae1b5 Make sure WAL is synced for DB::Write() if write batch is empty
59d549798 suppress JDK8 errors for #385
ea18b944a Add db_bench option --report_file_operations
2ea1219eb Fix RecordIn and RecordDrop stats
e4211d10c Apply InfoLogLevel to the logs in util/env_hdfs.cc
76f6c7c7c CompactionFilterV2: eliminate an often unnecessary allocation.
29a9161f3 Note dynamic options in options.h
fd24ae9d0 SetOptions() to return status and also add it to StackableDB
b1267750f fix the asan check
83bf09144 Bump verison number to 3.7
da5daa061 Replace some ASSERT_TRUE() asserts in DBTest.DynamicMemtableOptions and DynamicCompactionOptions with more specific ones
b0cda4a11 DBTest.DynamicMemtableOptions to use single background compaction
8810850dd Apply InfoLogLevel to the logs in db/compaction_job.cc
71783f652 Merge pull request #377 from fyrz/RocksJava-KeyMayExist
614bbcbe2 Merge pull request #374 from fyrz/RocksJava-DBOptions-Extension-3.6
d8e119663 Apply InfoLogLevel to the logs in db/version_set.cc
2a019f1d0 Apply InfoLogLevel to the logs in db/wal_manager.cc
469d474ba Apply InfoLogLevel to the logs in db/db_impl.cc
ac6afaf9e Enforce naming convention of getters in version_set.h
09899f0b5 DB::Open() to automatically increase thread pool size if it is smaller than max number of parallel compactions or flushes
636e57b52 Fix coverage script
30ca3752b Revamp our build tools
051c67ff5 Merge pull request #378 from baotiao/master
d0e7e49ae Merge pull request #379 from fyrz/RocksJavaBuildFix
94e31ac22 [RocksJava] Extend Options with DBOptions implementation [RocksJava] Included DBOptionsTest and refactored OptionsTest
b060d3006 [RocksJava] Build fix after options refactoring
9fd65e566 add make clean in examples makefile
8e79ce68c Revert "Fix lint errors and coding style of ldb related codes."
45a612f99 Revert "Fix incorrect fixing of lint errors in ldb_cmd.cc"
27129c739 [RocksJava] KeyMayExist w/o ColumnFamilies
86905e3cb Move VersionBuilder logic to a separate .cc file
74eb4fbe9 CompactionJob
8ddddd62d Fix incorrect fixing of lint errors in ldb_cmd.cc
46c14c666 Fix#258. benchmarkharness -- make bm_min_usec uint
72cb7cf20 Add fsync / corrupt simulation to env_mem
0e526eb9d introduce TestMemEnv and use it in db_test
8db24f4b3 exclude mock test file from MOCK_SOURCES
5594d446f unfriend DBImpl and InternalStats from VersionStorageInfo
82e3ae540 fix c_test
bc9f36fd5 Fix lint errors and coding style of ldb related codes.
c645250ee CompactionStats to support larger value of RecordIn and RecordDrop
f7e6c856a Fix BaseReferencedVersionBuilder's destructor order
c76dcb44d fix
b452dede5 fix
29d83cc33 temporarily remove -Wshadow
c1a924b9f Move convenience.h to /include
7e01d1202 Add support for in place update for db_stress
9f7fc3ac4 Turn on -Wshadow
98849a35f Apply InfoLogLevel to the logs in table/block_based_table_reader.cc
4d2ba38b6 Make VersionBuilder unit testable
2b1f23dca Apply InfoLogLevel to the logs in db/db_iter.cc
ccaf1aa7c Merge pull request #372 from fyrz/RocksJava-CF-Merge-Hardening
a29118ffc Merge pull request #355 from fyrz/RocksJava-Options-Refactoring-3.6
85b04ca76 [RocksJava] Review comments - reformatted MergeTest
df7abb4e8 [RocksJava] Integrated code review comments
171be0ed5 Merge with ColumnFamilies & Hardening CFHandle
39464a990 [RocksJava] Options Refactoring 3.6
0f7f3b860 Check InfoLogLevel earlier in Log functions.
73605d917 Apply InfoLogLevel to the logs in util/db_info_dumper.cc
fda592d90 Merge pull request #356 from fyrz/RocksJava-TableOptions-3.6
c73d13bb8 [RocksJava] Integrate review comments from yhchiang
b011e201f Integrated review comments by ankgup87
2c1bd8846 BlockBasedTableConfig & PlainTableConfig enhancements
e770d6175 Merge pull request #371 from dlezama/master
41af0f56b Fix build break because of unsigned/signed mismatch
c5db7f260 Fix CompactionPickerTest.Level1Trigger2
37e9b6370 Apply InfoLogLevel to the logs in utilities/ttl/db_ttl_impl.h
217cc217d Apply InfoLogLevel to the logs in table/meta_blocks.cc
635905481 WalManager
fd95745a5 Fix compile error in table/plain_table_index.cc
c3dd0f75d comparator_db_test to cover more irregular comparators
6afafa369 Apply InfoLogLevel to the logs in utilities/merge_operators/uint64add.cc
e7ad69b9f Apply InfoLogLevel to the logs in table/plain_table_index.cc
bbd9c5345 Apply InfoLogLevel to the logs in table/block_based_table_builder.cc
065766b8d DynamicCompactionOptions: relax the check bound a little
5c82a8837 Add a test in compaction_picker_test to test the max score
86de2007b Add ComparatorDBTest to test non-default comparators
17be187ff dummy var to suppress compiler warning/error
c2999f54b Revert "tmp"
76d1c28e8 Make CompactionPicker more easily tested
01e6f8509 Apply InfoLogLevel to the logs in db/transaction_log_impl.h
082e49ba8 Apply InfoLogLevel to the logs in db/repair.cc
c4b468000 Apply InfoLogLevel to the logs in db/flush_job.cc
34d436b7d Apply InfoLogLevel to the logs in db/column_family.cc
cda9943f9 Apply InfoLogLevel to the logs in db/compaction_picker.cc
7b3a618f9 Apply InfoLogLevel to the logs in db/db_filesnapshot.cc
2d4fe048f remove dead code
9ab013236 tmp
76d54530d minor - remove default value for ChangeFilterOptions() and ChangeCompactionOptions()
44f0ff31c use fallocate(FALLOC_FL_PUNCH_HOLE) to release unused blocks at the end of file
97451f837 add an env var ROCKSDB_TESTS_FROM to control where to start from a list of tests
e130e88bc DBTest: options clean up - part 4
34f3c5a20 DBTest: options clean up - part 3
cdc7230e4 DBTest: options clean up - part 2
5a921b895 DBTest: options clean up - part 1
c9c935923 Move the check to the beginning of the loop in VersionEdit::EncodeTo()
2110e43a5 Remove an unnecessary include file in version_edit.cc
412b7f85b Include atomic in mock_table.h
c08285334 Include all the mocks
abac3d647 TableMock + framework for mock classes
fb3f8ffe5 Improve the robustness of PartialCompactionFailure test again.
60fa7d136 Improve the robustnesss of PartialCompactionFailure test.
3772a3d09 Fix the bug where compaction does not fail when RocksDB can't create a new file.
c49dedbe0 Merge pull request #366 from fyrz/RocksJava-Backup-Restore-Improvements
a39e931e5 FlushProcess
efa2fb33b make LevelFileNumIterator and LevelFileIteratorState anonymous
f7c973069 [RocksJava] Integrated review comments
7e12ae5a2 [RocksJava] - BackupInfos & Restore-/BackupableDB enhancements
eb357af58 unfriend ForwardIterator from VersionSet
f981e0813 unfriend ColumnFamilyData from VersionSet
834c67d77 rename FileLevel to LevelFilesBrief / unfriend CompactedDBImpl
a28b3c438 unfriend UniversalCompactionPicker,LevelCompactionPicker and FIFOCompactionPicker from VersionSet
5187d896b unfriend Compaction and CompactionPicker from VersionSet
75d7e2c37 Merge pull request #369 from fyrz/Small-Fix
db52419cf Merge pull request #290 from vladb38/master
45e756f04 [RocksJava] Minor correction to the previous pull request merge
f94f1a97d Merge pull request #368 from fyrz/RocksJava-Hardening-RocksIterator
b08c39e14 [RocksJava] RocksIterator: Assert for valid RocksDB instance & documentation
b680033e6 Include atomic in env_test
56ef2caaa [RocksJava] - Hardening RocksIterator
c1c68bce4 remove atomic_pointer.h references
7c303f0e7 Include atomic
179c23021 Merge pull request #351 from fyrz/RocksJava_Snapshot_Support
48842ab31 Deprecate AtomicPointer
f37048ad1 Merge pull request #367 from fyrz/FixBrokenJavaBuild
679a9671f RocksJava Fix after MutableCFOptions change.
714c63c58 db_stress for dynamic options
f1841985e dynamic inplace_update options
a04929aa4 fixed conflict in java/Makefile
a1bae76c8 Integrated changes due to review bei ankgup87
b8ce52648 [RocksJava] Support Snapshots
bc3bc4bc2 Merge pull request #357 from fyrz/JavaTest-Fix
42f0eaceb Merge pull request #354 from fyrz/RocksJava-memtables-3.6
965d9d50b Fix timing
001ce64dc Use chrono for timing
240ed0cd7 Fix uninitialized parameter caused by D24513
724fba2b3 Improve the log in Universal Compaction to include more debug information.
720c1c056 fix erro during merge
b794194ad Remove java build from travis
122f98e0b dynamic max_mem_compact_level
1fee591e7 comments for DynamicCompactionOptions test
574028679 dynamic max_sequential_skip_in_iterations
bd4fbaee3 Fixed cross platform build after introducing Java-7 dependencies
1eb545721 Fix incorrectly merged Java - Makefile
9aa9668a8 [RocksJava] Memtables update to 3.6
4b1786e95 Fix SIGSEGV when declaring Arena after ScopedArenaIterator
05b2e60dd Merge pull request #362 from adamretter/travis-jcheck
9383922cc Added java tests to travis build
90f156402 Fix CompactBetweenSnapshots
2717422e2 Merge pull request #361 from adamretter/fix-yosemite-build
3b5fe3a1f Correct the log message in VersionEdit
c584d2b53 Fix for building RocksDB Java on Mac OS X Yosemite
2a8e5203d db_bench: --batch_size used for write benchmarks too
d755e53b8 Printing number of keys in DB Stats
839c376bd fix table_test
0fd985f42 Avoid reloading filter on Get() if cache_index_and_filter_blocks == false
e11a5e776 Improve the comment of util/thread_local.h
6398e6a6a Fix DeleteFile() + enable deleting files oldest files in level 0
0d3145198 Merge pull request #231 from adamretter/master
a6fb7f312 Fix code review comments raised in https://reviews.facebook.net/D22779
c63494fb6 Tests for ComparatorOptions, Comparator and DirectComparator, and by proxy we also exercise Slice and DirectSlice
5e2527411 Fix code style problems identified by lint
25641bfc9 Fix to memory dealocation when creating a slice from a byte buffer
fc12cb83f Add locking to comparator jni callback methods which must be thread-safe
d6fe8dacc Feature - Implement Java API for Comparator and Slice. Allows use of either byte[] or DirectByteBuffer for accessing underlying data.
700f6ec3f Ignore IntelliJ idea project files and ignore java/out folder
5bfb7f5d0 db_bench: seekrandom can specify --seek_nexts to read specific keys after seek.
ff8f74c20 remove checking lower bound of level size
2dd9bfe3a Sanitize block-based table index type and check prefix_extractor
dbcfe27d6 Merge branch 'master' of https://github.com/facebook/rocksdb
6c6691864 Speed up DB::Open() and Version creation by limiting the number of FileMetaData initialization.
5db9e7664 Fix Mac compile error: C++11 forbids default arguments for lambda expressions
f4363fb81 Fix DynamicMemtableOptions test
8f01bf80c Merge pull request #353 from fyrz/FixBloomfilterRocksJava
ee80fb4b4 Total memtables size counter
c12f571d3 Fix mac compile, second try
d2e60f5ce Fix mac compile
274dc81c9 fix build failure
d6c8dba72 Log MutableCFOptions in SetOptions
4d5708aa5 dynamic soft_rate_limit and hard_rate_limit
065a67c4f dynamic disable_auto_compactions
dc50a1a59 make max_write_buffer_number dynamic
6a150c011 ldb: support --fix_prefix_len
bafbc23ba Filters getting disposed by System.gc before EOL
ca250d71a Move logging out of mutex
5cc9adf5b WriteBatchWithIndex's Iterator bug of SeekToFirst() and SeekToLast()
7658bcc1e Merge pull request #352 from fyrz/OptionsTestMergeProblem
1b97934a2 Options correction
2ef3ed86f Integrated feedback from ankgup87
a40ce219b Adding merge functions to RocksDBJava
cc6c883f5 Stop stopping writes on bg_error_
4942f6efb Merge pull request #350 from fyrz/reenable_accidentally_disabled_test
ee28f431d With the last commit a Test was accidentally disabled. This commit solves this.
b5dd7eed6 Merge pull request #319 from fyrz/column_families_java
18004d2f2 [RocksJava] Column family support
5908d08a0 Merge pull request #336 from fyrz/32BitRocksJavaBug
9e5f2a952 Merge pull request #344 from fyrz/java-doc-enhancements
4f5a68725 32-Bit RocksJava resolution for jlong overflows
16d2ebdbc Minor adjustment to prevent two warnings
70294c911 JavaDoc improvements on RocksJava
833357402 WriteBatchWithIndex supports an iterator that merge its change with a base iterator.
4f65fbd19 WriteBatchWithIndex's iterator to support SeekToFirst(), SeekToLast() and Prev()
f441b273a WriteBatchWithIndex to support an option to overwrite rows when operating the same key
3ead857a0 Fixed Mac compile error in util/options_test.cc
5a7618634 Fixed compile error on Mac: default arguments for lambda expressions
cd0d581ff convert Options from string
f18b4a484 minor update to benchmark script
b7d3d6ebc db_bench: set thread pool size according to max_background_flushes
c5f54a8f3 Merge pull request #339 from fyrz/bloomFilterSupportNewFormat
ced612957 Improved JavaDoc
5e43155b3 RocksJava should support not only BlockBased Bloomfilter
1d525891b Update HISTORY for 3.6
88edfd90a SkipListRep::LookaheadIterator
6a443309d Merge pull request #342 from fyrz/java_makefile_fix
4f272408c RocksJava Makefile includes incorrect paths to version.h
f78b832e5 Log RocksDB version
25f6a852e add db_test for changing memtable size
daab6dc51 Merge pull request #318 from criccomini/master
63eade401 Fix error introduced by merge
ba882972f Merge pull request #340 from nbougalis/nullderef
6bcff9dc2 Merge pull request #341 from fyrz/arc-lint
d6169954b Removed code which prevents `arc lint` from working properly.
b87db0715 Avoid dereferencing a null field
1a1b95347 Merge pull request #270 from tdfischer/check-with-unity
e107b6b2a Merge pull request #337 from fyrz/cross-platform-jdb_bench
70e401a2e Merge pull request #338 from fyrz/rm_obs_code_write_batch_with_index_test
1e5a52815 update release readme
d44871e80 fix java doc directory in git ignore
2a4d6e796 merge master to resolve merge conflicts
6b2c1d962 make publish jni jars depend on release jni jars
1c7c76476 Replaced obsolete comparator with builtin variant.
22c64be43 Cross platform fix for Java benchmark shell script.
81828592b Merge pull request #335 from fyrz/version-script-cross-platform
69d4c5123 Cross-platform fix version.sh
fcd13a77a Merge pull request #334 from fyrz/JavaDoc-Cleanup
bfb0246f5 Merge pull request #331 from fyrz/findbug_issues
05204bb11 Lint changes
da8ff9ff8 Fixed Findbugs issues
a5757ff3c Listing of changes
df3373fbf [Java] Fix compile error on DbBenchmark.java
4eb5a40f7 [Java] Fixed link error on library loading on Mac.
56dfd363f Fix a check in database shutdown or Column family drop during flush.
0e516a75d Fix lint errors in java/rocksjni/options.cc
8ea232b9e Add number of records dropped in compaction summary
f4086a88b perf_context.get_from_output_files_time is set for MultiGet() and ReadOnly DB too.
e869fc6a8 remove proper javadoc directory
c1273533b Merge pull request #333 from nbougalis/cleanups
99744e0c4 bump version to 3.6
45d526e22 singular javadoc directory
378f321da merge master to resolve merge conflicts
a1d3f0d2b don't fail if javadocs diretory doesn't exist
c832f1644 add not about updating pom version and rename pom to be unversioned
a213971d8 Don't return (or dereference) dangling pointer
2a1add673 use proper major/minor/micro version rather than hard coding 3.5.0
2d72f7807 update release docs in java
8322cf000 use javadoc instead of javadocs
079a612b5 Fix build_tools/version.sh
2e8012498 add javadoc and sources targets for sonatype
df08a2d03 add single rocksdbjni pom
deefcf476 make fat jar unclassified to satisfy sonatype
fd2545c80 add maven publication target and instructions
1e9af10ef Merge pull request #330 from fyrz/setFilterPolicyForBBTConfig
017354177 FilterTest
d410b39d5 BlockBasedTableConfig Filter policy support RocksJava
0908ddcea Don't keep managing two rocksdb version
d0916f452 add major minor micro version to java jars
d6987216c Merge pull request #327 from dalgaaf/wip-da-SCA-20141001
25888ae00 Merge pull request #329 from fyrz/master
89833e5a8 Fixed signed-unsigned comparison warning in db_test.cc
fcac705f9 Fixed compile warning on Mac caused by unused variables.
b3343fdea resolution for java build problem introduced by 5ec53f3edf62bec1b690ce12fb21a6c52203f3c8
187b29938 ForwardIterator: update prev_key_ only if prefix hasn't changed
5ec53f3ed make compaction related options changeable
d122e7bcf Update INSTALL.md
9d6f38086 backupable_db_test.cc: pass const string param by reference
8ff0b4095 document_db_test.cc: pass const string param by reference
177caca42 ttl/ttl_test.cc: pass const string param by reference
4a171882d db/version_set.cc: remove unnecessary checks
bf3bfd044 util/cache_test.cc: use static_cast over C-Style cast
86e29f033 document_db.cc: remove unused variable
28a6e3158 table/block_based_table_builder.cc: remove unused variable
091153493 db/db_test.cc: remove unused variable
6b6cedbb1 table/format.cc: reduce scope of some variables
55652043c table/cuckoo_table_reader.cc: pass func parameter by reference
5abd8add7 db/deletefile_test.cc: remove unused variable
d6483af87 db/db_test.cc: reduce scope of some variables
44cca0cd8 db/db_iter.cc: remove unused variable
986dad025 Merge pull request #324 from dalgaaf/wip-da-SCA-20140930
3a0d498a3 rebase master
8ee75dca2 db/memtable.cc: remove unused variable merge_result
0fd8bbca5 db/db_impl.cc: reduce scope of prefix_initialized
676ff7b1f compaction_picker.cc: remove check for >=0 for unsigned
e55aea551 document_db.cc: fix assert
d517c8364 in_table_factory.cc: use correct format specifier
b14037556 ttl/ttl_test.cc: prefer prefix ++operator for non-primitive types
43c789c8f spatialdb/spatial_db.cc: use !empty() instead of 'size() > 0'
0de452ee9 document_db.cc: pass const parameter by reference
4cc8643ba util/ldb_cmd.cc: prefer prefix ++operator for non-primitive types
af8c2b2d9 util/signal_test.cc: suppress intentional null pointer deref
33580fa39 db/db_impl.cc: fix object handling, remove double lines
873f1356a db_ttl_impl.h: pass func parameter by reference
855845714 ldb_cmd_execute_result.h: perform init in initialization list
063471bf7 table/table_test.cc: pass func parameter by reference
93548ce8f table/cuckoo_table_reader.cc: pass func parameter by ref
b8b7117e9 db/version_set.cc: use !empty() instead of 'size() > 0'
8ce050b51 table/bloom_block.*: pass func parameter by reference
53910ddb1 db_test.cc: pass parameter by reference
68ca53416 corruption_test.cc: pass parameter by reference
726ac5bca shrink vagrant commands to single line
a2f98ef61 fix tabs in Makefile
7506198da cuckoo_table_db_test.cc: add flush after delete
1f963305a Print MB per second compaction throughput separately for reads and writes
ffe3d490d Add an instruction about SSE in INSTALL.md
0b923f0f9 add centos 5.6 build instead of ubuntu.
ee1f3ccb0 Package generation for Ubuntu and CentOS
f0f795549 Fixing comile errors on OS X
99fb613e5 remove 2 space linter
b2d64a486 Fix linters, second try
747523d24 Print per column family metrics in db_bench
56ebd4087 Fix arc lint (should fix#238)
637f89179 Merge pull request #321 from eonnen/master
827e31c74 Make test use a compatible type in the size checks.
fd5d80d55 CompactedDB: log using the correct info_log
2faf49d5f use GetContext to replace callback function pointer
6a64ea617 add note about java 7
983d2de2d Add AUTHORS file. Fix#203
c4519c777 fix mis-named jar in JNI loader
abd70c5e1 Merge pull request #316 from fyrz/ReverseBytewiseComparator
2dc6f62bb handle kDelete type in cuckoo builder
8b8011a68 Changed name of ReverseBytewiseComparator based on review comment
b8e26615a since we're not sharing folders with the vm, copy built .so files and jars back to host system.
389edb6b1 universal compaction picker: use double for potential overflow
4e735bb7f Rsync files to VM rather than sync folders, since sync folders was causing clock skew and confusig make.
82a8f43cc Document RELEASE.mdgit status
9db13987b Update RocksDB's Java bindings to support multiple native RocksDB builds in the same Jar file. Cross build RocksDB for linux32 and linux64 using Vagrant. Build a cross-platform fat jar that contains osx, linux32, and linux64 RocksDB static builds.
534048426 Built-in comparator(s) in RocksJava
d439451fa delay initialization of cuckoo table iterator
94997eab5 reduce memory usage of cuckoo table builder
c6275956e improve memory efficiency of cuckoo reader
581442d44 option to choose module when calculating CuckooTable hash
fbd2dafc9 CompactedDBImpl::MultiGet() for better CuckooTable performance
3c6800610 CompactedDBImpl
f7375f39f Fix double deletes
21ddcf6e4 Remove allow_thread_local
fb4a492cc Merge pull request #311 from ankgup87/master
611e286b9 Merge branch 'master' of https://github.com/facebook/rocksdb
0103b4498 Merge branch 'master' of ssh://github.com/ankgup87/rocksdb
1dfb7bb98 Add block based table config options
cdaf44f9a Enlarge log size cap when printing file summary
7cc1ed7f0 Merge pull request #309 from naveenatceg/staticbuild
ba6d660f6 Resolving merge conflict
51eeaf65e Addressing review comments
fd7d3fe60 Addressing review comments (adding a env variable to override temp directory)
cf7ace886 Addressing review comments
0a29ce539 re-enable BlockBasedTable::SetupForCompaction()
55af37075 Remove TODO for checking index checksums
3d74f0997 Fix compile
53b003995 Fix release compile
d0de413f4 WriteBatchWithIndex to allow different Comparators for different column families
57a32f147 change target_file_size_base to uint64_t
5e6aee432 dont create backup_input if compaction filter v2 is not used
49b5f94c5 Merge pull request #306 from Liuchang0812/fix_cast
787cb4db2 remove cast, replace %llu with % PRIu64
a7574d4fa Update logging.cc
7e0dcb953 Update logging.cc
57fa3cc5b Merge pull request #304 from Liuchang0812/fix-check
cd44522a9 Merge pull request #305 from Liuchang0812/fix-logging
6a031b6a8 remove unused variable
4436f17bd fixed#303: replace %ld with % PRId64
7a1bd057f Merge pull request #302 from ankgup87/master
423e52cd4 Merge branch 'master' of https://github.com/facebook/rocksdb
bfeef94d3 Add rate limiter
ed9a2df8c fix unity build
32f2532a0 Print compression_size_percent as a signed int
976caca09 Skip AllocateTest if fallocate() is not supported in the file system
3b897cddd Enable no-fbcode RocksDB build
f44594743 RocksDB: Format uint64 using PRIu64 in db_impl.cc
e17bc65c7 Merge pull request #299 from ankgup87/master
b93797abc Fix build
adae3ca1f [Java] Fix JNI link error caused by the removal of options.db_stats_log_interval
90b8c07b4 Fix unit tests errors
51af7c326 CuckooTable: add one option to allow identity function for the first hash function
035043559 Fixed a signed-unsigned comparison in spatial_db.cc -- issue #293
2fb1fea30 Fix syncronization issues
6f0964e37 Only run make unity on travis instead of make check
9ed1b49a2 Build unity build on make check
ff7689561 Remove some unnecessary constructors
feadb9df5 fix cuckoo table builder test
3c232e164 Fix mac compile
54cada92b Run make format on PR #249
27b22f13a Merge pull request #249 from tdfischer/decompression-refactoring
fb6456b00 Replace naked calls to operator new and delete (Fixes#222)
5600c8f6e cuckoo table: return estimated size - 1
a062e1f2c SetOptions() for memtable related options
e4eca6a1e Options conversion function for convenience
a7c209452 Merge pull request #292 from saghmrossi/master
4d0523452 Merge branch 'master' of github.com:saghmrossi/rocksdb
60a4aa175 Test use_mmap_reads
94e43a1df [Java] Fixed 32-bit overflowing issue when converting jlong to size_t
f9eaaa66e added include for inttypes.h to fix nonworking printf statements
f090575e4 Replaced "built on on earlier work" by "built on earlier work" in README.md
faad439ac Fix#284
49aacd8d2 Fix make install
acb9348ff [Java] Include WriteBatch into RocksDBSample.java, fix how DbBenchmark.java handles WriteBatch.
4a27a2f19 Don't sync manifest when disableDataSync = true
9b8480d93 Merge pull request #287 from yinqiwen/rate-limiter-crash-fix
28be16b1d fix rate limiter crash #286
04ce1b25f Fix#284
add22e351 standardize scripts to run RocksDB benchmarks
dee91c259 WriteThread
540a257f2 Fix WAL synced
24f034bf4 Merge pull request #282 from Chilledheart/develop
49fe329e5 Fix build issue under macosx
ebb5c65e6 Add make install
0352a9fa9 add_wrapped_bloom_test
9c0e66ce9 Don't run background jobs (flush, compactions) when bg_error_ is set
a9639bda8 Fix valgrind test
d1f24dc7e Relax FlushSchedule test
3d9e6f775 Push model for flushing memtables
059e584dd [unit test] CompactRange should fail if we don't have space
dd641b211 fix RocksDB java build
53404d9fb add_qps_info_in cache bench
a52cecb56 Fix Mac compile
092f97e21 Fix comments and typos
6cc12860f Added a few statistics for BackupableDB
0a42295a2 Fix SimpleWriteTimeoutTest
06d986252 Always pass MergeContext as pointer, not reference
d343c3fe4 Improve db recovery
6bb7e3ef2 Merger test
88841bd00 Explicitly cast char to signed char in Hash()
52311463e MemTableOptions
1d284db21 Addressing review comments
55114e7f4 Some updates for SpatialDB
171d4ff4a remove TailingIterator reference in db_impl.h
9b0f7ffa1 rename version_set options_ to db_options_ to avoid confusion
2d57828d0 Check stop level trigger-0 before slowdown level-0 trigger
659d2d50c move compaction_filter to immutable_options
048560a64 reduce references to cfd->options() in DBImpl
011241bb9 DB::Flush() Do not wait for background threads when there is nothing in mem table
a2bb7c3c3 Push- instead of pull-model for managing Write stalls
0af157f9b Implement full filter for block based table.
9360cc690 Fix valgrind issue
02d5bff39 Merge pull request #277 from wankai/master
88a2f44f9 fix comments
7c16e3922 Merge pull request #276 from wankai/master
823773837 replace hard-coded number with named variable
db8ca520b Merge pull request #273 from nbougalis/static-analysis
b7b031f42 Merge pull request #274 from wankai/master
4c2b1f097 Merge remote-tracking branch 'upstream/master'
a5d286307 typo improvement
9f8aa0939 Don't leak data returned by opendir
d1cfb71ec Remove unused member(s)
bfee319fb sizeof(int*) where sizeof(int) was intended
d40c1f742 Add missing break statement
2e97c3898 Avoid off-by-one error when using readlink
40ddc3d6c add cache bench
9f1c80b55 Drop column family from write thread
8de151bb9 Add db_bench with lots of column families to regression tests
c9e419ccb rename options_ to db_options_ in DBImpl to avoid confusion
5cd0576ff Fix compaction bug in Cuckoo Table Builder. Use kvs_.size() instead of num_entries in FileSize() method.
0fbb3facc fixed memory leak in unit test DBIteratorBoundTest
adcd2532c fix asan check
4092b7a0b Merge pull request #272 from project-zerus/patch-1
bb6ae0f80 fix more compile warnings
6d3144118 Merge pull request #271 from nbougalis/cleanups
0cd0ec4fe Plug memory leak during index creation
4329d74e0 Fix swapped variable names to accurately reflect usage
45a5e3ede Remove path with arena==nullptr from NewInternalIterator
5665e5e28 introduce ImmutableOptions
e0b99d4f5 created a new ReadOptions parameter 'iterate_upper_bound'
51ea88900 Fix travis builds
a4816269f Relax backupable rate limiting test
f7f973d35 Merge pull request #269 from huahang/patch-2
ef5b38472 fix a few compile warnings
2fd3806c8 Merge pull request #263 from wankai/master
1785114a6 delete unused Comparator
1b1d9619f update HISTORY.md
703c3eacd comments about the BlockBasedTableOptions migration in Options
4b5ad8865 Merge pull request #260 from wankai/master
19cc588b7 change to filter_block std::unique_ptr support RAII
9b976e34f Merge pull request #259 from wankai/master
5d25a4693 Merge remote-tracking branch 'upstream/master'
dff2b1a8f typo improvement
343e98a7d Reverting import change
ddb8039e3 RocksDB static build Make file changes to download and build the dependencies .Load the shared library when RocksDB is initialized
REVERT: 1fdd726a8 Hotfix RocksDB 3.5
REVERT: d67500a59 Add `make install` to Makefile in 3.5.fb.
REVERT: 4cb631aa0 update HISTORY.md
REVERT: cfd0946be comments about the BlockBasedTableOptions migration in Options
git-subtree-dir: src/rocksdb2
git-subtree-split: aead4041720f40e305e7f9d432875749d91d5a2d
Previously if you mistyped the "submit_multisigned" command as
"submit_multisign", the returned message was "Internal error". Not
very helpful. It turns out this was caused by a small amount of
code in RPCCall.cpp. Removing that code improves two situations:
1. It improves the situation with a mistyped command. Now the
command returns "Unknown method" and provides the string of
the mistyped command.
2. The "transaction_entry", if properly entered in its command
line form, would fire an assert. That assert is now removed.
In the process, it was discovered that the command line form of
the "transaction_entry" command has not worked correctly for at
least a year. Therefore support for that the command line form
of "transaction_entry" is added along with appropriate unit
tests.
It is common for a validator operator to connect their validator
only to other nodes under their control, using clustering. Relaying of
untrusted validations and proposals must be unreliable to prevent
denial of service attacks. But currently, they are unreliable even
within a cluster.
With this change, a cluster member's decisison to relay (or originate)
a validation or proposal is honored by other cluster members. This
ensures that validators in a cluster will get reliable relaying to
hubs outside the cluster, even if other members of the cluster do not
have that validator on their UNL.
* Changes to Travis seem to be making these builds slower, and are
resulting in spurious timeouts. Remove the builds that fail more
often than not until we complete our migration to another build
system.
* Use CMake for unity builds, instead of the slow build.
* Can be exercised from the command line with json2
* Rewrite Env::do_rpc to call the same code as
rpc from the command line. This puts rpc
handling logic in one place.
* If any of the destructor, copy assignment or copy constructor
are user-declared, both copy members should be user-declared,
otherwise the compiler-generation of them is deprecated.
* If any of the destructor, copy assignment or copy constructor
are user-declared, both copy members should be user-declared,
otherwise the compiler-generation of them is deprecated.
* Remove composite helper functions
* Add set difference and Bitset/uint256 operators
* Convert tests to use new feature bitset set difference operator
In order to automatically run unit tests with newly created
amendments, prefer to start with jtx::supported_features() and
then subtract unwanted features.
These changes identified a few bugs that were hiding in
amendments. One of those bugs, in FlowCross, is not yet fixed.
By uncommenting the test in CrossingLimits_test.cpp you can see
failures relating to that bug. Since FlowCross is not yet
enabled on the network we can fix the bug at our convenience.
Both Tickets and SHAMapV2 have been around for a while and don't
look like they will be enabled on the network soon. So they are
removed from the supportedAmendments list. This prevents Env
from automatically testing with Tickets or SHAMapV2 enabled,
although testing with those features can still be explicitly
specified.
Drive-by cleanups:
o supportedAmendments() returns a const reference rather than
a fresh vector on each call.
o supportedAmendments() implementation moved from Amendments.cpp
to Feature.cpp. Amendments.cpp deleted.
o supportedAmendments() declared in Feature.h. All other
declarations deleted.
o preEnabledAmendments() removed, since it was empty and only
used in one place. It will be easy to re-add when it is needed.
o jtx::all_features_except() renamed to
jtx::supported_features_except(), which is more descriptive.
o jtx::all_amendments() renamed to jxt::supported_amendments()
o jtx::with_features() renamed to with_only_features()
o Env_test.cpp adjusted since featureTickets is no longer
automatically enabled for unit tests.
- Separate `Scheduler` from `BasicNetwork`.
- Add an event/collector framework for monitoring invariants and calculating statistics.
- Allow distinct network and trust connections between Peers.
- Add a simple routing strategy to support broadcasting arbitrary messages.
- Add a common directed graph (`Digraph`) class for representing network and trust topologies.
- Add a `PeerGroup` class for simpler specification of the trust and network topologies.
- Add a `LedgerOracle` class to ensure distinct ledger histories and simplify branch checking.
- Add a `Submitter` to send transactions in at fixed or random intervals to fixed or random peers.
Co-authored-by: Joseph McGee
We need to ensure that pointers and/or references to existing
elements will not be invalidated during the course of element
insertion and removal.
Containers like std::vector do not offer this guarantee, and
cannot be used as the underlying container for the stack.
By choosing to explicitly specify std::deque as the underlying
cotnainer, we avoid:
- the unlikely possibility of the C++ standards committee
changing the default template parameter to a container;
- the more likely possibility of an accidental change by
a programmer, without fully considering the consequences.
* Stores recent history of "good" ledgers. Uses the maximum as the
expected ledger size. When a large value drops off, use a 90%
backoff to go down to to the new maximum.
* If consensus is unhealthy, wipe the history in addition to the current
clamping.
* Include .md doc files in xcode and VS projects
In support of dynamic validator list, this changeset:
1. Adds a new `validator_list_expires` field to `server_info` that
indicates when the current validator list will become stale.
2. Adds a new admin only `validator_lists` RPC that returns the
current list of known validators and the most recent published validator
lists.
3. Adds a new admin only `validator_sites` RPC that returns the list of
configured validator publisher sites and when they were most recently
queried.
Manifests of validators newly added to a published validator list are
not reliably propagated to network nodes.
This solves the problem by allowing a published validator list to
include the manifest.
RIPD-1559
Switches the default behavior of Consensus to use roundCloseTime instead of
effCloseTime. effCloseTime is still used when accepting the consensus ledger to
ensure the consensus close time comes after the parent ledger close time. This
change eliminates an edge case in which peers could reach agreement on the close
time, but end up generating ledgers with different close times.
Introduce "fix1523" which corrects a minor technical flaw with
the original implementation of the escrow feature.
When creating an escrow, the entry would only be tracked in the
owner directory of the sender; as a result, an escrow recipient
would not be able to detect incoming escrows without monitoring
the ledger in real-time for transactions of interest or without
the sender communicating this information out of band.
With the fix in place, escrows where the recipient differs from
the sender will be listed in the recipient's owner directory as
well.
* If the transaction can't be queued, recover to the open ledger once,
and drop it on the next attempt.
* New result codes for transactions that can not queue.
* Add minimum queue size.
* Remove the obsolete and incorrect SF_RETRY flag.
* fix#2215
Removes unnecessary common link libs related to openssl, which we link
to explicitly using finder vars now. THis allows the generated XCode
project to link without error. Tested with CMake 3.9.0 and XCode 8.3.3.
Add coverage for a few invariant checks. Handle exception in invariant
checking code so that an check that throws an exception will still
properly return tef/tecINVARIANT_FAILED.
if openssl is configured with compression support AND you link
to the static lib, you are going to need to link to zlib.
Fundamentally the CMake finder should take care of this (as
described in https://gitlab.kitware.com/cmake/cmake/issues/16885),
but we work around it here by adding zlib explicitly. Update
beast error test for OpenSSL 1.1.0
When two ledgers have the same number of validations, the code
will now use the ledger hash itself to break the tie rather than
the highest node ID supporting each validation.
* Use fixed size UNL if the total listed validators are below
threshold.
* Set quorum to provide Byzantine fault tolerance until a
threshold of total validators is exceeded, at which time
quorum is 80%.
* Ensure that a quorum of 0 cannot be configured.
Sets a maximum TransferRate value of 100%. Squashes any
requested TransferRate over the limit to the max value.
This change requires an amendment ("fix1201") in rippled.
Adds test functionality for enabling an amendment mid-test.
Additionally, adds tests utilizing varying transfer rates both
with and without the amendment enabled.
This commit introduces the "SortedDirectories" amendment, which
addresses two distinct issues:
First, it corrects a technical flaw that could, in some edge cases,
prevent an empty intermediate page from being deleted.
Second, it sorts directory entries within a page (other than order
book page entries, which remain strictly FIFO). This makes insert
operations deterministic, instead of pseudo-random and reliant on
temporal ordering.
Lastly, it removes the ability to perform a "soft delete" where
the page number of the item to delete need not be known if the
item is in the first 20 pages, and enforces a maximum limit to
the number of pages that a directory can span.
The two active users of DeadlineTimer, NetworkOPs and Application,
now use asio::steady_timers rather than DeadlineTimer.
DeadlineTimer is removed since it is no longer used.
To assure that all in-flight closures on timers are done before
Stoppables call stopped(), the JobCounter is made more generic.
It's now a ClosureCounter. The ClosureCounter is currently used
to count closures in flight for the JobQueue, NetworkOPs, and the
Application.
If the JobQueue is used during shutdown then those Jobs may access
Stoppables after they have already stopped. This violates the
preconditions of Stoppables and may lead to undefined behavior.
The solution taken here is to reference count all Jobs in the
JobQueue. At stop time all Jobs already in the JobQueue are
allowed to run to completion, but no further Jobs are allowed
into the JobQueue.
If a Job is rejected from the JobQueue (because we are stopping),
then JobQueue::addJob() returns false, so the caller can make any
necessary adjustments.
Add more test coverage for ServerHandlerImp.cpp. Ensure limit parameter
is propagated from parsed object to in-memory config.
Release Notes
-------------
This fixes a bug whereby the limit parameter on a port configuration was
ignored.
Check and modify amendment blocked status with each new ledger (provided
by @wilsonianb). Honor blocked status in certain RPC commands and when
deciding whether to propose/validate.
Fixes: RIPD-1479
Fixes: RIPD-1447
Release Notes
-------------
This resolves an issue whereby an amendment blocked server would still
serve some RPC requests that are unreliable in blocked state and would
continue to publish validations.
A bug in gdb is crashing when demangling some symbols. Gdb needs to be updated
to at least gdb 8.
Only run the debug build under gdb (other builds can time out).
Fixes: RIPD-1417
Fix incorrect error case messages. Fix crash in NetworkOps instance when
exiting with remaining RPC subscriptions. Add code to remove URL
subscription when requested.
Enable all supported amendments in Env by default. Rename `features()`
to `with_features()` and add `all_features_except()` to support feature
subsets in Env. Refactor internal feature handling based on a bitset.
Introduces a generic Validations class for storing and querying current and
recent validations. Aditionally migrates the validation related timing
constants from LedgerTiming to the new Validations code.
Introduces RCLValidations as the version of Validations adapted for use in the
RCL. This adds support for flushing/writing validations to the sqlite log and
also manages concurrent access to the Validations data.
RCLValidations::flush() no longer uses the JobQueue for its database
write at shutdown. It performs the write directly without
changing threads.
3bcd986 Set version to 79
4f42f8c Remove spurious fallthrough guidance
fa1ac16 Set version to 78
0cb9b63 Fix warning in root ca declaration
f78c73a Tidy up file_posix unused variable
72ce9ef Tidy up dstream for existing Boost versions
efe8e58 Add Boost.Locale utf8 benchmark comparison
f7c745e Remove string_view_body
7a79efa Tidy up FieldsReader doc
e51aefd Header file tidying
69898f4 Fix warning in zlib
53723c0 Add message keep_alive, chunked, content_length members
d7af73b Fix spurious uninitialized warning
ca42cc0 Tidy up invalid characters in test vector
48d3e60 Use make_unique_noinit
a1ff804 span, string, vector bodies are public
feab6a0 Documentation work
116c0b0 Add span_body
7fb901d Tidy up includes and javadocs
f0f58be Add span
7a8982b Add vector_body
524f73a Tidy up basic_string_body
a8ad67b Set version to 77
d555859 file_posix works without large file support
1bc30cb Set version to 76
9a1e7a8 Disable SSE4.2 optimizations
09af312 Fix parse illegal characters in obs-fold
7dd684c Add file_body_win32:
1bbc71c serializing file_body is not const
9a4b55e BodyReader, BodyWriter use two-phase init (API Change):
dc400ce Serializer members are not const
1a33c37 Rename to serializer::keep_alive (API Change):
55935c5 Add serializer::chunked
63ace37 Add serializer::get
2c4047b BodyReader may construct from a non-const message
0a0a225 Use Boost.Config
6f83d07 Always go through write_some:
0e23066 Set version to 75
28f3ece Doc tidying
3495331 Using SSE4.2 intrinsics in basic_parser if available
bc1f0ac file_body tests
4e03d7e Add serializer::limit
85e3ee8 Shrink serializer buffers using buffers_ref
78bcdb1 Tidy up BEAST_NO_BIG_VARIANTS
3ea6cf2 Construct buffer_prefix_view in-place
69f9f7a Use file_body for valid requests, string_body otherwise.
6f88f01 Set version to 74
93fed8e remove redundant flush() from example:
e0f56da Fix Beast include directories for cmake targets
5ff9e0a Add file_posix
1bb5705 Remove common/file_body.hpp
5c89d87 Add file_body
67a55c8 Add file_win32
647d3b0 Add file_stdio and File concept
89c416c Set version to 73
0efc32f Fixes for gcc-4.8
c8910ab Initialize local variable in basic_parser
8a28193 Adjust benchmarks
81e51d8 Verify certificates in SSL clients
a43f6d4 Jamroot tweak
8c85ee8 Put more... links on overload reference pages
ff1104e Documentation tidy
826ff0e serializer::next replaces serializer::get (API Change):
8d67775 Refactor header and message constructors:
6c79f19 Add basic_parser tests
25127d9 basic_parser optimizations:
9d082fd Set version to 72
c88e2b9 Various improvements to http_server_fast.cpp:
20b0fdb Documentation tidying
afd1fa7 Add websocket-server-async example
954b597 Add http-server-threaded example
df8f253 Refactor file_body for best practices
11c1037 Newly constructed responses have a 200 OK result
a648817 Refine Body::size specification
40aad37 Tidy up set payload in http-server-fast
52cefbc Set version to 71
8c51c77 Tidy up Jamroot /permissive-
5efecea Update README.md
5a47acd Tidy up http_sync_port error check
a2af2b5 Concept check in basic_dynamic_body
8b80a6f Fix buffer overflow handling for string_body and mutable_body
ec3b4fd Return std::size_t from Body::writer::put (API Change)
effbb37 Check trailers in test
f5368cf Call prepare_payload in HTTP example
a3e5e01 Fix spurious on_chunk invocation
96d94eb Add options for building examples and tests. Move zlib test sources to test/zlib
e0efdc7 Allow close, ping, and write to happen concurrently
9c1a419 Refactor websocket composed ops
d5659a4 Fine tune websocket asserts
b8e8943 std::pair "last" -> "first" in http_message.qbk
c691bf4 Fix can/cannot thinko in FAQ.
6dd006b Documentation revision
6d2e315 Fix extra ; warning
78a065b Set version to 70
00c7e9d Fix HEAD response in file_service
67d70d2 Fix BEAST_FALLTHROUGH in config
4f33655 Add parser::on_header to set a callback
9c16b21 Add basic_parser header and body limits:
b64d1f7 Rename to message::base (API Change):
436c66a Serialize in one step when possible
3e1061b Set version to 69
f709273 Add /permissive- to msvc toolchain
0dae464 Use BEAST_FALLTHROUGH to silence warnings
a70d386 basic_parser optimizations
4269f35 Set version to 68
544327f Link statically on cmake MSVC
e213ffe Add const_body, mutable_body to examples
0568763 Optimize field lookups
8fc3001 Use string_ref in older Boost versions
8af77da bad_target replaces bad_path (API Change):
325dd62 Adjust buffer size in fast server
be59785 Doc erratum
d9b44f3 Small speed up in fields comparisons
3e6ce38 Use Boost master on Appveyor
09f3d64 Split common tests to a new project
adfd22a Remove BodyReader::is_deferred (API Change):
582d28d Change BodyReader, BodyWriter requirements (API Change):
8982e14 Set version to 67
daa58a2 Group common example headers
afd8f1a Rename to http-server-fast
07cb9f7 control_callback replaces ping_callback (API Change):
91e83ed Use boost::string_view
067db87 Merge stream_base to stream and tidy
d61241a Add http-server-small example
eb08e92 Fix doc example link
7fb75d0 Set version to 66
df86723 Respect debug flag for marked output
c08565a Squelch spurious warning on gcc
188ef7c Documentation work
1c62d3a Add http-server example
3f54582 basic_fields optimizations
a8b05b8 Add header aliases
b94eac3 Tidy up message piecewise ctors
9c48b52 Handle bad_alloc in parser
1b57c54 Fix costly potential value-init in parser
1edc41e Make consuming_buffers smaller
72ac918 Add serializer request/response aliases
18f7606 string_param optimizations
c675252 Set version to 65
c398cdd Enable msvc warnings in Jamfile
380cceb Fix unused variable warnings
4172e7e Enable unused variable warning on msvc cmake
f04d227 Fix integer warnings
ca975b3 Fix narrowing in deflate_stream
2fab796 Fix narrowing in inflate_stream
cff87f6 Fix narrowing in ostream
1956886 Fix narrowing in static_ostream
69cdc4b Fix integer types in deflate_stream::bi_reverse
3e3dd21 Enable narrowing warning on msvc cmake
e11a294 Set version to 64
a00e070 Remove make_serializer (API Change):
8449697 Add link_directories to cmake
7b2b066 Doc tidying
158d3e8 async_write requires a non-const message:
d13328b Better User-Agent in examples
ebcb2c0 Exemplars are compiled code
b9054d3 Simplify websocket write_op
c2571fe Simplify ssl teardown composed op
d8ad3d1 Simplify buffered_read_stream composed op
f68dc34 Set version to 63
a99f7ef Control running with valgrind explicitly
4eb7af4 Tidy up Jamfiles
49b42a5 Tidy up CMakeLists.txt
dadb54f Only run the tests under ubasan
d1c7696 Move benchmarks to a separate project
a4aada8 Only build and run tests in variant=coverage
f835b9a Don't use cached Boost
a0edd82 Put num_jobs back up on Travis
a8d5823 Use std::to_string instead of lexical_cast
45d8b81 Set version to 62
09e07ce Put slow tests back for coverage builds
295b1d7 Doc tidy
f58425c Squelch harmless not_connected errors
9b537f7 Add http::is_fields trait
d43701b message::prepare_payload replaces message::prepare (API Change):
42ba289 Refine FieldsReader concept (API Change)
bde90a1 Narrow the use of Fields parameters:
5f47526 parser requires basic_fields (API Change):
60f58e4 Avoid explicit operator bool for error
352f8de Clear the error faster
34befd8 Tidy up namespaces in examples
9e0b4b5 Doc fixes and tidy
c003a2a Tidy up test build scripts and projects
b929130 Add server-framework tests
03d4301 Increase detail::static_ostream coverage
80d7cbc Remove libssl-dev from a Travis matrix item
4c15db4 Set version to 61
1dfbd0b Don't run slow tests on certain targets
6bb1109 Use one job less on CI
6f58342 Tidy up resolver calls
530b044 Add multi_port to server-framework
bfef5d1 Tidy up http-crawl example
e2f2f33 Reorganize SSL examples
adc301b Fix shadowing warnings
c60185e Add server-framework SSL HTTP and WebSocket ports
7912fb8 Refactor WebSocket, HTTP examples:
cd4b9e0 Flush the output stream in the example
d046b20 Tidy up names in error categories
9d4a422 status-codes is unsigned (API Change)
e3599b0 header::version is unsigned (API Change)
a26b043 Add message::header_part()
fc8d2e9 Tidy up some integer conversion warnings
c91732e Reorganize SSL examples
9907b31 Documentation work
4b2e78e Use generic_cateogry for errno
38c46cd Remove Spirit dependency
c111d6f Set version to 60
d78dc12 Documentation work
141a524 New server-framework, full featured server example:
3f7ffd9 Fix response message type in async websocket accept
13f3750 String comparisons are public interfaces
4e4bcf8 Set version to 59
5015cdb Remove obsolete doc/README.md
71c3f0a Fix base64 alphabet
aa2b843 Change Body::size signature (API Change):
80a599a Documentation work
9c19449 Integrated Beast interface.
3f8097b Set version to 58
4581777 Better message formal parameter names
5879cd8 Fix parsing chunk size with leading zeroes
56bd228 Remove redundant code
534ca63 Use static string in basic_fields::reader
a7b3810 basic_parser::put doc
1e4413f basic_fields::set optimization:
9b244c1 Fix basic_fields insert ordering
4f854d0 Avoid std::string in websocket
dc8f146 Renamed to basic_fields::set (API Change):
660c465 Specification for http read
981285b Documentation tidy
983d676 Reorganize examples:
a2a5c57 Qualify size_t in message template
d86769c Fix unaligned reads in utf8-checker
8ba182c Set version to 57
42e2791 Update doc/ for docca
1ee0afd Merge commit '101d7dbfb9725674cb9ce5a4196f19aa1d4bb801' as 'doc/docca'
101d7db Squashed 'doc/docca/' content from commit c50b3ba5
900c04e Documentation work
8eee932 Fix warning in basic_parser.cpp
437a616 Fix message.hpp javadocs
18c68ce Set version to 56
b058e90 Convert buffer in range loops
cbc9212 Add Beast INTERFACE CMake target
2914b59 More basic_parser tests
ed5c317 Reset error codes
ba14251 Test error code handling
e45e50b Tidy up README.md build instructions
16efb9b Try harder to find Boost (cmake)
e281d91 HTTP/1 is the default version
916fe4a Call on_chunk when the extension is empty
9855598 Add string_view_body
19d4520 Tidy up
6e59f9b Add provisional IANA header field names
84722f2 Revert "Add a Beast CMake interface target:"
fde6929 Set version to 55
01f6cc4 Documentation work
a7a388c read_size replaces read_size_helper:
ed8f0bb Tidy up type_traits
c2f6268 Avoid a parser allocation using non-flat buffer:
906db45 Add a Beast CMake interface target:
47f2541 Don't allocate memory to handle obs-fold
6a8912a Set version to 54
296ef3b Documentation work
e3e9b61 Fix incorrect use of [[fallthrough]]
3c44398 Retain ownership when reading using a message
a71bb2b basic_fields refactor (API Change):
d8d3562 Add string_param
83b2558 basic_fields members and coverage
c4f5fa5 consuming_buffers members and coverage
e10507c multi_buffer members and coverage
0e6bd3f flat_buffer coverage
7351d6e static_buffer coverage
18a8ef5 Set version to 53
452df59 Remove extraneous doc file
3ef0359 Fix read_size_helper usage:
b64e6d3 Fix basic_parser::maybe_flatten (#462)
76402f7 Set version to 52
b0ceb2a Add drain_buffer class
4c6735a flat_buffer is an AllocatorAwareContainer
9d5d4d5 Documentation work
d4ec693 finish(error_code&) is a BodyReader requirement (API Change)
7b24cad opcode is private (API Change):
068c2ac Documentation work
a1ff89b Disable std::future snippet for libstdc++ bug
b5ef664 read_frame returns `bool` fin (API Change):
7911847 Remove opcode from read, async_read (API Change):
c72d70f ping_callback is a member of stream (API Change):
720a309 write_buffer_size is a member of stream (API Change):
7ff0178 read_message_max is a member of stream (API Change):
cd40964 read_buffer_size is a member of stream (API Change):
a58e5e1 binary, text are members of stream (API Change):
ad35846 auto_fragment is a member of stream (API Change):
ccee139 Documentation work
13c64e3 Set version to 51
cafc8e2 Fix infinite loop in basic_parser
dc4b69d Add construct, destroy to handler_alloc
58c2739 multi_buffer implementation change (API Change):
dd7f5c0 DynamicBuffer benchmarks
1c4811b Use BOOST_STRINGIZE
31051ac Use BOOST_FALLTHROUGH
8f2430b Documentation work
eb35b92 Fix file_body::get() not setting the more flag correctly
566244a Tidy up file_body
53cbeea Tune up static_buffer (API Change):
20c59b7 Fix operator<< for header
a2c1117 Set version to 50
6045b74 http read_some, async_read_some don't return bytes (API Change):
4df6885 Fix chunk header parsing
36bf32b Fix test::pipe read_size
bf69ce1 Fix chunk delimiter parsing
0c6b6b1 Add missing handler_alloc nested types
a06b8f9 Tidy up message javadoc
3bd8260 Remove obsolete serializer allocator (API Change)
001c979 Remove message free functions (API Change)
745876b Remove message connection settings (API Change)
bcf2c33 Body documentation work
1e303b0 Fields concept work
9d0464a Tidy up basic_fields, header, and concepts
3ba81b5 Use field in basic_parser
b5f6cc1 Use field in basic_fields and call sites
cfd6d14 Documentation reference tidy
2adc80a Protect basic_fields special members (API Change)
d55b079 Fix basic_fields allocator awareness
d8febda Documentation work
485a6e5 Refactor prepare (API Change)
9fd3071 Derive header from Fields (API Change)
8ad26b8 Use allocator more in basic_fields
0071039 Add verb to on_request for parsers (API Change)
74f6cbb Add field enumeration
be0d74f Documentation fixes
054ac40 Remove header_parser (API Change)
a007eba parser is constructible from other body types
d89809f Documentation work (buffer_body)
ac5bc4f Set version to 49
af47128 Documentation work
a1848f0 Add HEAD request example
ddfbfbf Use <iosfwd> instead of <ostream>
e67c0ab Refactor header status, reason, and target (API Change):
60f044a Tidy up empty_body writer error
7d267f4 Canonicalize string_view parameter types
ac175cb Refactor method and verb (API Change):
e18efed Documentation work
d77e423 Set version to 48
d3a5a05 Documentation work
acf18fb Tidy up traits
6cb188e Remove detail::sync_ostream
d6092bc Documentation work
4707b21 Rename to parser (API Change):
3cb385d Consolidate parsers to parser.hpp
290bdf1 Documentation work
7cb442c Make buffer_prefix_view public
ef0b121 Rename to buffer_cat_view (API Change)
b9df187 Tidy up chunk decorator (API Change):
458fa7e Set version to 47
fc83a03 Documentation work
1ee3013 Fix leak in basic_flat_buffer
55fbf76 Fix undefined behavior in pausation
fe75a7c Refactor HTTP serialization and parsing (API Change):
50cba32 buffer_size overload for basic_multi_buffer::const_buffers_type
d977bf2 Disable operator<< for buffer_body
5db707a Refactor treatment of status code and obsolete reason (API Change):
9a585a8 Refactor treatment of request-method (API Change):
3ae76d0 Set version to 46
6004712 Documentation work
34ea0b3 Refactor serialization algorithms:
407b046 Rename to make_serializer
c29451a Refactor type_traits (API Change):
8578419 Refactor HTTP serialization (API Change):
f8612aa Remove HTTP header aliases (API Change):
b0054e3 Add test::pipe
dfba72b Set version to 45
6ba3697 Disable reverse_iterator buffer_view test
266ebac buffer_view skips empty buffer sequences
96b9892 Documentation work
c23f1e2 Fix header::reason
9796106 Better test::enable_yield_to
9a8bcb7 Fix message doc image
7a5e87e Workaround for boost::asio::basic_streambuf type check
663c275 Set version to 44
f205976 Make buffers_adapter meet requirements
8e39c60 Tidy up is_dynamic_buffer traits test
0088f7c Add buffers_adapter regression test
8a23de1 Fix README websocket example
949504a Fix async return values in docs
cd9f41b Use BOOST_STATIC_ASSERT
1b616fa Tidy up and make get_lowest_layer public
612e616 Require Boost 1.58 or later
1b1daa7 Tidy up read_size_helper and dynamic buffers
bf0145d Use BOOST_THROW_EXCEPTION
e762818 Add GitHub issue template
dab679c Set version to 43
386b817 Reformat README.md QR code
50e5123 Additional constructors for consuming_buffers
f7289b9 Add write limit to test::string_ostream
3aa87e0 Tidy up buffer_prefix overloads and test
bee583c Fix strict aliasing warnings in buffers_view
6b54d3a Require Boost 1.64.0
76f1084 Set version to 42
0bdb148 Make buffers_view a public interface
338fc81 Add formal review notes
784f965 Fix javadoc typo
823aee2 Set version to v41
88adbdd Remove handler helpers, tidy up hook invocations (API Change)
4974af2 Rename prepare_buffer(s) to buffer_prefix (API Change)
ebd459b Tidy up websocket::close_code enum and constructors
c3fd6f9 Tidy up formal parameter names
210cd70 Remove coveralls integration
d811962 Concept revision and documentation (API Change):
bdae92a Replace asynchronous helper macros with template aliases (API Change)
df95a09 Move prepare_buffers to prepare_buffer.hpp (API Change)
787de21 Remove placeholders (API Change)
c59b544 Trim Appveyor matrix rows
b7184f3 Return http::error::end_of_stream on HTTP read eof (API Change)
f2d8255 Set version to 40
40b9194 Tidy up .travis.yml:
9b240c7 Fix basic_streambuf movable trait
76a2617 Consolidate get_lowest_layer in type_traits.hpp
6d00321 Add to_static_string:
f888136 Set version to 39
47c82b5 Better travis deps
4ed7865 Squelch openssl spurious leak and memory errors
b6bc26f Fixed braced-init error with older gcc
59b2f8f ostream workaround for gcc 4.8.4
8363d86 Increase ostream test coverage
5631936 Tidy up HTTP reason_string (API Change):
2bf5150 Harmonize concepts and identifiers with net-ts (API Change):
728e9d8 Tidy up basic_parser javadocs
1c9067b Use beast::string_view alias
771c5ca Doc fixes and tidying
e2b5c31 Rename to buffered_read_stream (API Change):
a753f1c Rename to static_buffer, static_buffer_n (API Change):
24b6686 Rename to flat_buffer, basic_flat_buffer (API Change):
69259ef Rename to multi_buffer, basic_multi_buffer (API Change):
bef9ae1 New buffers() replaces to_string() (API Change):
a7ef4f5 New ostream() returns dynamic buffer output stream (API Change):
87fd60c Fix eof error on ssl::stream shutdown
606fc9d Add websocket async echo ssl server test:
ff5e659 Refactor http::header contents (API Change):
dd02097 Set version to 1.0.0-b38
5596e97 Prevent basic_fields operator[] assignment
c2b32dc Remove websocket::keep_alive option (API Change):
32dbfb2 Refactor WebSocket error codes (API Change):
dd6b500 WebSocket doc work
0b4d87c More flat_streambuf tests
aacefb4 Add test_allocator to extras/test
931a5fb Simplify get_lowest_layer test
ba4228a Use static_string for WebSocket handshakes:
6df3ff3 Refactor base64:
19b124d Refactor static_string:
30e8d16 Set version to 1.0.0-b37
b141020 Fix narrowing warning in table constants
d554b81 Add -funsigned-char to asan build target
bcc6ad8 Add ub sanitizer blacklist
e1f08e9 Fix flat_streambuf:
7d08f59 Fix typo in documentation example
21ef97d Rename to http::dynamic_body, consolidate header:
45a2d73 Rename project to http-bench
c86fee9 Move everything in basic_fields.hpp to fields.hpp (API Change)
a14a5d6 Rename to detail::is_invocable
540d037 Rename to websocket::detail::pausation
84e1739 Document websocket::stream thread safety
dc274af Add is_upgrade() free function:
2c17d04 Refactor websocket decorators (API Change):
235fe68 Provide websocket::stream accept() overloads (API Change):
a715825 CMake hide command lines in .vcxproj Output windows
32024d8 Set version to 1.0.0-b36
f48b95f Update README.md
d8db5f1 Set version to 1.0.0-b35
dd2a514 Tidy up doc declarations
2c50aba Fix README.md CMake instructions
4ffdce2 Update .gitignore for VS2017
403011f Remove http::empty_body (API Change)
f47b661 New HTTP interfaces (API Change):
f6835b8 Rename to BEAST_DOXYGEN
7e37723 Add flat_streambuf:
5b68faa Doc XSL support for list and table markdown
3de46de Make websocket::close_code a proper enum:
0128743 Tidy up MSVC CMake configuration
ccd188e Add appveyor build script
git-subtree-dir: src/beast
git-subtree-split: 3bcd9865f80f12ba5faad35c564918f85b02e271
If you work at a digital asset exchange or wallet provider
and are trying to integrate with Ripple, please contact us
at support@ripple.com. We can help guide your integration.
Invariant checks run after a transaction has been processed and
ensure that the end result of that transaction did not violate
any protocol rules.
New checks include:
* XRP balance checks for negative balances
* Offer balance checks for negative balances
* Zero balance checks when handling escrow
A brain wallet is a standard wallet that is generated not from a
random seed but by hashing a user-supplied passphrase. Typically,
human-selected passphrases can contain insufficient entropy.
When generating a wallet from a passphrase, we include a warning
to this effect. The warning would be incorrectly displayed even
if the wallet was being generated from a seed.
The job queue can automatically tune the number of threads that
it creates based on the number of processors or processor cores
that are available.
The existing tuning was very conservative, limiting the maximum
number of threads to only 6.
Adjust the new algorithm to allow a larger number of threads and
allow server administrators to override the value in the config
file.
Using a PaymentSandbox for taker cross can cause transaction breaking changes. A
PaymentSandbox uses a deferred credits table, which can cause tiny differences
in computations.
If an offer transaction touched multiple ledger entries associated with the same
book, that offer transaction would be published multiple times to anyone subscribed
to that book stream.
Fixes#2095.
Make sure statically-configured bootcache entries have at least
a reasonable minimum priority. This provides additional protection
against Sybil attacks.
Show the bootcache in the ouput of the print command.
- Add Consensus::Result, which represents the result of the
establish state and includes the consensus transaction set, final
proposed position and disputes.
- Add Consensus::Mode to track how we are participating in
consensus and ensures the onAccept callback can distinguish when
we entered the round with consensus versus when we recovered from
a wrong ledger during a round.
- Rename Consensus::Phase to Consensus::State and eliminate the
processing phase. Instead, accept is a terminal phase which
notifies RCLConsensus via onAccept callbacks. Even if clients
dispatch accepting to another thread, all future calls except to
startRound will not change the state of consensus.
- Move validate_ status from Consensus to RCLConsensus, since
generic implementation does not directly reference whether a node
is validating or not.
- Eliminate gotTxSetInternal and handle externally received
TxSets distinct from locally generated positions.
- Change ConsensusProposal::changePosition to always update the
internal close time and position even if we have bowed out. This
enforces the invariant that our proposal's position always
matches our transaction set.
Replace Taker.cpp with calls to the payment flow() code.
This change required a number of tweaks in the payment flow code.
These tweaks are conditionalized on whether or not offer crossing
is taking place. The flag is explicitly passed as a parameter to
the flow code.
For testing, a class was added that identifies differences in the
contents of two PaymentSandboxes. That code may be reusable in
the future.
None of the Taker offer crossing code is removed. Both versions
of the code are co-resident to support an amendment cut-over.
The code that identifies differences between Taker and Flow offer
crossing is enabled by a feature. That makes it easy to enable
or disable difference logging by changing the config file. This
approach models what was done with the payment flow code. The
differencing code should never be enabled on a production server.
Extensive offer crossing unit tests are added to examine and
verify the behavior of corner cases. The tests are currently
configured to run against both Taker and Flow offer crossing.
This gives us confidence that most cases run identically and
some of the (few) differences in behavior are documented.
6d5547a Set version to 1.0.0-b34
6fab138 Fix and tidy up CMake build scripts:
ccefa54 Set version to 1.0.0-b33
32afe41 Set internal state correctly when writing frames:
fe3e20b Add write_frames unit test
578dcd0 Add decorator unit test
aaa3733 Use fwrite return value in file_body
df66165 Require Visual Studio 2015 Update 3 or later
b8e5a21 Set version to 1.0.0-b32
ffb1758 Update CMake scripts for finding packages:
b893749 Remove http Writer suspend and resume feature (API Change):
27864fb Add io_service completion invariants tests
eba05a7 Set version to 1.0.0-b31
484bcef Fix badge markdown in README.md
5663bea Add missing dynabuf_readstream member
0d7a551 Tidy up build settings
0fd4030 Move the handler, don't copy it
git-subtree-dir: src/beast
git-subtree-split: 6d5547a32c50ec95832c4779311502555ab0ee1f
Add new functionality to enforce one or more sanity checks (invariants)
on transactions. Add tests for each new invariant check. Allow
for easily adding additional invariant checks in the future.
Also Resolves
-------------
- RIPD-1426
- RIPD-1427
- RIPD-1428
- RIPD-1429
- RIPD-1430
- RIPD-1431
- RIPD-1432
Release Notes
-------------
Creates a new ammendment named "EnforceInvariants" which must be
enabled in order for these new checks to run on each transaction.
A full ledger on the production Ripple network could
exceed the default maximum reply size for the
HTTPClient code. Remove the reply size maximum for
responses that include a Content-Length header.
The ledger_request RPC call, under some conditions, did not
actually check that the entire ledger was present in the
database, making it unsuitable for use in cases where the
database was believed to be incorrect or incomplete.
With this change, the full ledger will be checked for
integrity unless it has already recently been checked
(according to the InboundLedgers cache).
The 'type' field allows the rpc client to specify what type of ledger
entries to retrieve. The available types are:
"account"
"amendments"
"directory"
"fee"
"hashes"
"offer"
"signer_list"
"state"
"ticket"
Consensus::checkLCL can change state_ but it was being called inside
timerEntry after a switch on the current state_. In rare cases, this might
end up calling stateEstablish even when the state_ was open.
A new JobCounter class is introduced. The JobCounter keeps
a reference count of Jobs in flight to the JobQueue. When
NetworkOPs needs to stop, in addition to other work, it calls
JobCounter::join(), which waits until all Jobs in flight
have been destroyed before returning. This ensures that all
NetworkOPs Jobs are completed before NetworkOPs declares
itself stopped().
Also, once a JobCounter is join()ed, it refuses to produce
more counted Jobs for the JobQueue. So, once all old Jobs
in flight are done, then NetworkOPs will add no additional
Jobs to the JobQueue.
Other classes besides NetworkOPs should also be able to use
JobCounter. NetworkOPs is a first test case.
Also unneeded #includes were removed from files touched for
other reasons.
`env.fund` requires two transactions: `pay` and `set account`. If there is a
`trust` transaction in the same set of txs, the txs may be reordered so
`pay` -> `trust` -> `set account` so the wrong `no ripple` flag would be used
on the trust line.
Adding a `close` between `env.fund` and `env.trust` resolves this problem.
This is a substantial refactor of the consensus code and also introduces
a basic consensus simulation and testing framework. The new generic/templated
version is in src/ripple/consensus and documents the current type requirements.
The version adapted for the RCL is in src/ripple/app/consensus. The testing
framework is in src/test/csf.
Minor behavioral changes/fixes include:
* Adjust close time offset even when not validating.
* Remove spurious proposing_ = false call at end of handleLCL.
* Remove unused functionality provided by checkLastValidation.
* Separate open and converge time
* Don't send a bow out if we're not proposing
* Prevent consensus stopping if NetworkOPs switches to disconnect mode while
consensus accepts a ledger
* Prevent a corner case in which Consensus::gotTxSet or Consensus::peerProposal
has the potential to update internal state while an dispatched accept job is
running.
* Distinguish external and internal calls to startNewRound. Only external
calls can reset the proposing_ state of consensus
The DatabaseImp has threads that asynchronously call JobQueue to
perform database reads. Formerly these threads had the same
lifespan as Database, which was until the end-of-life of
ApplicationImp. During shutdown these threads could call JobQueue
after JobQueue had already stopped. Or, even worse, occasionally
call JobQueue after JobQueue's destructor had run.
To avoid these shutdown conditions, Database is made a Stoppable,
with JobQueue as its parent. When Database stops, it shuts down
its asynchronous read threads. This prevents Database from
accessing JobQueue after JobQueue has stopped, but allows
Database to perform stores for the remainder of shutdown.
During development it was noted that the Database::close()
method was never called. So that method is removed from Database
and all derived classes.
Stoppable is also adjusted so it can be constructed using either
a char const* or a std::string.
For those files touched for other reasons, unneeded #includes
are removed.
Calling OverlayImpl::list_[].second->stop() may cause list_ to be
modified (OverlayImpl::remove() may be called on this same thread).
So iterating directly over OverlayImpl::list_ to call
OverlayImpl::list_[].second->stop() could give undefined behavior.
On MacOS that undefined behavior exhibited as a hang.
Therefore we copy all of the weak/shared ptrs out of
OverlayImpl::list_ before we start calling stop() on them. That
guarantees OverlayImpl::remove() won't be called until
OverlayImpl::stop() completes.
The DatabaseImp holds threads that access DatabaseRotateImp. But
the DatabaseRotateImp's destructor runs before the DatabaseImp
destructor. The DatabaseRotateImp now assures that the
DatabaseImp threads are stopped before the DatabaseRotateImp
destructor completes.
RootStoppable was using two separate flags to identify that it
was stopping. LoadManager was being notified when one flag was
set, but checking the other flag (not yet set) to see if we were
stopping. There is no strong motivation for two flags. The
timing window is closed by removing one flag and moving around
a chunk of code.
Modify doStart Application method to specify whether or not to start the
DeadlineTimers. Specify inactive timers for jtx::Env Applications and
active timers for standard Applications.
* RPC `ledger` command returns all queue entries in "queue_data"
when requesting open ledger, and including boolean "queue: true".
* Includes queue state. e.g.: fee_level, retries, last_result, tx.
* Respects "expand" and "binary" parameters for the txs.
* Remove some unused code.
* Sanity check on newly created strands
* Better loop detection
* Better tests (test every combination of path element pairs)
* Disallow any root issuer (even for xrp)
* Disallow compount element typs in path
* Issue was not reset when currency was XRP
* Add amendment
Add envconfig test helper for manipulating Env config via
callables. Create new common modifiers for non-admin config,
validator config and one for using different server port values.
* Clean up and refactor
* Resume parents of nodes read asynchronously
* Resume at tip of new stack if exhausted prior stack
* No need to restart at root
Escrow replaces the existing SusPay implementation with improved
code that also adds hashlock support to escrow payments, making
RCL ILP enabled.
The new functionality is under the `Escrow` amendment, which
supersedes and replaces the `SusPay` amendment.
This commit also deprecates the `CryptoConditions` amendment
which is replaced by the `CryptoConditionSuite` amendment which,
once enabled, will allow use of cryptoconditions others than
hashlocks.
Make LEDGER_MIN_CONSENSUS slightly smaller and not a multiple of
LEDGER_GRANULARITY to avoid fluctuations in the heartbeat timer needlessly
delaying consensus.
* `CMAKE_C_COMPILER` and `CMAKE_CXX_COMPILER` must be defined
before `project`. However, it will clear `CMAKE_BUILD_TYPE`.
Use `CACHE` variables and reorder some code to work around
these constraints.
* Also correct a couple of copy paste errors.
All uses of beast::Thread were previously removed from the code
base, so beast::Thread is removed. One piece of beast::Thread
needed to be preserved: the ability to set the current thread's
name. So there's now a beast::CurrentThreadName that allows the
current thread's name to be set and returned.
Thread naming is also cleaned up a bit. ThreadName.h and .cpp
are removed since beast::CurrentThreadName does a better job.
ThreadEntry is also removed, but its terminateHandler() is
preserved in TerminateHandler.cpp. The revised terminateHandler()
uses beast::CurrentThreadName to recover the name of the running
thread.
Finally, the NO_LOG_UNHANDLED_EXCEPTIONS #define is removed since
it was discovered that the MacOS debugger preserves the stack
of the original throw even if the terminateHandler() rethrows.
The deferred credits table can compute a balance that's different from the
ledger balance.
Syntax:
A number written with no decimal means that number exactly. I.e. "12". A number
written with a decimal means that number has a non-zero digit at the lowest
order digit. I.e. "12.XX" means a number like "12.00000000000005"
Consider the following payment:
alice (USD) -> USD/XRP -> (XRP) Bob
Alice initially has 12.XX USD in her account.
The strand is used to debit alice the following amounts:
1) Debit alice 5
2) Debit alice 0.XX
3) Debit alice 3.XX
The next time the strand is explored, alice has a USD/XRP offer on the books,
and her account is credited:
1) Credit alice 20
When the beginning of the strand is reached, consider what happens when alice is
a limiting step. Calculate how much we can get out the step. According to the
deferred credit table this is:
12.XX - (5 + 0.XX + 3.XX)
This is also limited by alice's balance, which is large thanks to the credit she
received in the book step.
Now that the step has calculated how much we can get out, throw out the
sandbox (the one with the credit), and re-execute. However, the following error
occurs. We asked for 12.XX - (5 + 0.XX + 3.XX). However, the ledger has
calculated that alice has:
((12.XX - 5) - 0.XX) - 3.XX
That's a problem, because that number is smaller. Notice that there are two
precision losing operations in the deferred credits table:
1) The 5 + 0.XX step
2) The 12.XX - (total of debits). (Notice total of debits is < 10)
However, there is only one precision losing operation in the ledger calculation:
1) (Subtotal of 12.XX-5) - 0.XX
That means the calculation for the ledger results in a number that's smaller
than the deferred credits. Flow detects this as a re-execution error.
Allow manifest revoking validator keys to be stored in a separate
[validator_key_revocation] config field, so the validator can run
again with new keys and token.
Validator lists from configured remote sites are fetched at a regular
interval. Fetched lists are expected to be in JSON format and contain the
following fields:
* "manifest": Base64-encoded serialization of a manifest containing the
validator publisher's master and signing public keys.
* "blob": Base64-encoded JSON string containing a "sequence",
"expiration" and "validators" field. "expiration" contains the Ripple
timestamp (seconds since January 1st, 2000 (00:00 UTC)) for when the
list expires. "validators" contains an array of objects with a
"validation_public_key" field.
* "signature": Hex-encoded signature of the blob using the publisher's
signing key.
* "version": 1
* "refreshInterval" (optional)
Instead of specifying a static list of trusted validators in the config
or validators file, the configuration can now include trusted validator
list publisher keys.
The trusted validator list and quorum are now reset each consensus
round using the latest validator lists and the list of recent
validations seen. The minimum validation quorum is now only
configurable via the command line.
Squashed 'src/beast/' changes from 9f10b11..1b9a714
1b9a714 Set version to 1.0.0-b30
faed9e5 Allow concurrent websocket async ping and writes:
31cda06 Fix race when write suspends
48dd38e Fix race in close frames during reads
e2d1bb0 Fix race in pings during reads
36143be Set version to 1.0.0-b29
f0399b6 Fix doc link typo
787b7c2 Check ostream modifier correctly
4fa0bf6 Fix Writer return value documentation
6406da0 Document type-pun in buffer_cat
66cdb37 Fix illegal HTTP characters accepted as hex zero
e64ca2f Fix Body requirements doc
6dfd9f9 Fix compilation error in non-template class
fa7fea8 Fix race in writes during reads:
git-subtree-dir: src/beast
git-subtree-split: 1b9a71483347b7027b2fb7fe27ecea148d2e79ba
9f10b11 Set version to 1.0.0-b28
195f974 Fix HTTP split parse edge case:
264fd41 Restyle async result constructions
572a0eb Split out and rename test stream classes
95b6646 Tidy up some WebSocket javadocs
f6938d3 Set version to 1.0.0-b27
a6120cd Update copyright dates
c7bfe7d Add documentation building instructions
f6c91ce Tidy up tests and docs:
f03985f Move basic_streambuf to streambuf.hpp (API Change):
b8639a7 Invoke callback on pings and pongs (API Change):
git-subtree-dir: src/beast
git-subtree-split: 9f10b11eff58aeb793b673c8a8cb6e2bee3db621
* This fixes an uncommon, but annoying, spurious failure running this
test, particularly in release builds. This appears to be an issue with
Windows of the FS where quickly creating and deleting the same
directory repeatedly will eventually fail.
* RIPD-1390
06f74f0 Set version to 1.0.0-b26
68f535f Tidy up warnings and tests:
4ee5fa9 Set version to 1.0.0-b25
229d390 Update README.md for CppCast 2017
c3e3a55 Fix deflate setup bug
439a224 WebSocket server examples and test tidying:
29565c8 Remove unnecessary include
caa3b39 Fix 32-bit arm7 warnings
0474cc5 Better handler_ptr (API Change):
ca38657 Fixes for websocket echo server:
797631c Set version to 1.0.0-b24
a450968 Add permessage-deflate WebSocket extension:
67e965e Make decorator copyable
42899fc Add optional yield_to arguments
61aef03 Simplify Travis package install specification
9d0d7c9 bjam use clang on MACOSX
git-subtree-dir: src/beast
git-subtree-split: 06f74f05f7de51d7f791a17c2b06840183332cbe
* CMake defaults CMAKE_CXX_FLAGS_RELEASE, etc. to include defining
NDEBUG, regardless of other options set elsewhere, for most or all
generators. This change explicitly removes that flag from the relevant
variables.
* Also move the project command earlier, since it wipes out some local
changes.
Avoid custom overflow code; simply use 128-bit math to
maintain precision and return a saturated 64-bit value
as the final result.
Disallow use of negative values in the `fee_mult_max`
and `fee_div_max` fields. This change could potentially
cause submissions with negative values that would have
previously succeeded to now fail.
9d560f9 Merge #428: Exhaustive recovery
2cee5fd exhaustive tests: add recovery module
8225239 Merge #433: Make the libcrypto detection fail the newer API.
12de863 Make the libcrypto detection fail the newer API.
678b0e5 exhaustive tests: remove erroneous comment from ecdsa_sig_sign
2928420 Merge #427: Remove Schnorr from travis as well
03ff8c2 group_impl.h: remove unused `secp256k1_ge_set_infinity` function
a724d72 configure: add --enable-coverage to set options for coverage analysis
b595163 recovery: add tests to cover API misusage
8eecc4a Remove Schnorr from travis as well
6f8ae2f ecdh: test NULL-checking of arguments
25e3cfb ecdsa_impl: replace scalar if-checks with VERIFY_CHECKs in ecdsa_sig_sign
a8abae7 Merge #310: Add exhaustive test for group functions on a low-order subgroup
b4ceedf Add exhaustive test for verification
83836a9 Add exhaustive tests for group arithmetic, signing, and ecmult on a small group
20b8877 Add exhaustive test for group functions on a low-order subgroup
80773a6 Merge #425: Remove Schnorr experiment
e06e878 Remove Schnorr experiment
04c8ef3 Merge #407: Modify parameter order of internal functions to match API parameter order
6e06696 Merge #411: Remove guarantees about memcmp-ability
40c8d7e Merge #421: Update scalar_4x64_impl.h
a922365 Merge #422: Restructure nonce clearing
3769783 Restructure nonce clearing
0f9e69d Restructure nonce clearing
9d67afa Update scalar_4x64_impl.h
7d15cd7 Merge #413: fix auto-enabled static precompuatation
00c5d2e fix auto-enabled static precompuatation
91219a1 Remove guarantees about memcmp-ability
7a49cac Merge #410: Add string.h include to ecmult_impl
0bbd5d4 Add string.h include to ecmult_impl
353c1bf Fix secp256k1_ge_set_table_gej_var parameter order
541b783 Fix secp256k1_ge_set_all_gej_var parameter order
7d893f4 Fix secp256k1_fe_inv_all_var parameter order
c5b32e1 Merge #405: Make secp256k1_fe_sqrt constant time
926836a Make secp256k1_fe_sqrt constant time
e2a8e92 Merge #404: Replace 3M + 4S doubling formula with 2M + 5S one
8ec49d8 Add note about 2M + 5S doubling formula
5a91bd7 Merge #400: A couple minor cleanups
ac01378 build: add -DSECP256K1_BUILD to benchmark_internal build flags
a6c6f99 Remove a bunch of unused stdlib #includes
65285a6 Merge #403: configure: add flag to disable OpenSSL tests
a9b2a5d configure: add flag to disable OpenSSL tests
b340123 Merge #402: Add support for testing quadratic residues
e6e9805 Add function for testing quadratic residue field/group elements.
efd953a Add Jacobi symbol test via GMP
fa36a0d Merge #401: ecmult_const: unify endomorphism and non-endomorphism skew cases
c6191fd ecmult_const: unify endomorphism and non-endomorphism skew cases
0b3e618 Merge #378: .gitignore build-aux cleanup
6042217 Merge #384: JNI: align shared files copyright/comments to bitcoinj's
24ad20f Merge #399: build: verify that the native compiler works for static precomp
b3be852 Merge #398: Test whether ECDH and Schnorr are enabled for JNI
aa0b1fd build: verify that the native compiler works for static precomp
eee808d Test whether ECDH and Schnorr are enabled for JNI
7b0fb18 Merge #366: ARM assembly implementation of field_10x26 inner (rebase of #173)
001f176 ARM assembly implementation of field_10x26 inner
0172be9 Merge #397: Small fixes for sha256
3f8b78e Fix undefs in hash_impl.h
2ab4695 Fix state size in sha256 struct
6875b01 Merge #386: Add some missing `VERIFY_CHECK(ctx != NULL)`
2c52b5d Merge #389: Cast pointers through uintptr_t under JNI
43097a4 Merge #390: Update bitcoin-core GitHub links
31c9c12 Merge #391: JNI: Only call ecdsa_verify if its inputs parsed correctly
1cb2302 Merge #392: Add testcase which hits additional branch in secp256k1_scalar_sqr
d2ee340 Merge #388: bench_ecdh: fix call to secp256k1_context_create
093a497 Add testcase which hits additional branch in secp256k1_scalar_sqr
a40c701 JNI: Only call ecdsa_verify if its inputs parsed correctly
faa2a11 Update bitcoin-core GitHub links
47b9e78 Cast pointers through uintptr_t under JNI
f36f9c6 bench_ecdh: fix call to secp256k1_context_create
bcc4881 Add some missing `VERIFY_CHECK(ctx != NULL)` for functions that use `ARG_CHECK`
6ceea2c align shared files copyright/comments to bitcoinj's
70141a8 Update .gitignore
7b549b1 Merge #373: build: fix x86_64 asm detection for some compilers
bc7c93c Merge #374: Add note about y=0 being possible on one of the sextic twists
e457018 Merge #364: JNI rebased
86e2d07 JNI library: cleanup, removed unimplemented code
3093576 JNI library
bd2895f Merge pull request #371
e72e93a Add note about y=0 being possible on one of the sextic twists
3f8fdfb build: fix x86_64 asm detection for some compilers
e5a9047 [Trivial] Remove double semicolons
c18b869 Merge pull request #360
3026daa Merge pull request #302
03d4611 Add sage verification script for the group laws
a965937 Merge pull request #361
83221ec Add experimental features to configure
5d4c5a3 Prevent damage_array in the signature test from going out of bounds.
419bf7f Merge pull request #356
6c527ec Merge pull request #357
445f7f1 Fix for Windows compile issue
03d84a4 Benchmark against OpenSSL verification
2bfb82b Merge pull request #351
06aeea5 Turn secp256k1_ec_pubkey_serialize outlen to in/out
970164d Merge pull request #348
64666251 Improvements for coordinate decompression
e2100ad Merge pull request #347
8e48787 Change secp256k1_ec_pubkey_combine's count argument to size_t.
c69dea0 Clear output in more cases for pubkey_combine, adds tests.
269d422 Comment copyediting.
b4d17da Merge pull request #344
4709265 Merge pull request #345
26abce7 Adds 32 static test vectors for scalar mul, sqr, inv.
5b71a3f Better error case handling for pubkey_create & pubkey_serialize, more tests.
3b7bc69 Merge pull request #343
eed87af Change contrib/laxder from headers-only to files compilable as standalone C
d7eb1ae Merge pull request #342
7914a6e Make lax_der_privatekey_parsing.h not depend on internal code
73f64ff Merge pull request #339
9234391 Overhaul flags handling
1a36898 Make flags more explicit, add runtime checks.
1a3e03a Merge pull request #340
96be204 Add additional tests for eckey and arg-checks.
bb5aa4d Make the tweak function zeroize-output-on-fail behavior consistent.
4a243da Move secp256k1_ec_privkey_import/export to contrib.
1b3efc1 Move secp256k1_ecdsa_sig_recover into the recovery module.
e3cd679 Eliminate all side-effects from VERIFY_CHECK() usage.
b30fc85 Avoid nonce_function_rfc6979 algo16 argument emulation.
70d4640 Make secp256k1_ec_pubkey_create skip processing invalid secret keys.
6c476a8 Minor comment improvements.
131afe5 Merge pull request #334
0c6ab2f Introduce explicit lower-S normalization
fea19e7 Add contrib/lax_der_parsing.h
3bb9c44 Rewrite ECDSA signature parsing code
fa57f1b Use secp256k1_rand_int and secp256k1_rand_bits more
49b3749 Add new tests for the extra testrand functions
f684d7d Faster secp256k1_rand_int implementation
251b1a6 Improve testrand: add extra random functions
31994c8 Merge pull request #338
f79aa88 Bugfix: swap arguments to noncefp
c98df26 Merge pull request #319
67f7da4 Extensive interface and operations tests for secp256k1_ec_pubkey_parse.
ee2cb40 Add ARG_CHECKs to secp256k1_ec_pubkey_parse/secp256k1_ec_pubkey_serialize
7450ef1 Merge pull request #328
68a3c76 Merge pull request #329
98135ee Merge pull request #332
37100d7 improve ECDH header-doc
b13d749 Fix couple of typos in API comments
7c823e3 travis: fixup module configs
cc3141a Merge pull request #325
ee58fae Merge pull request #326
213aa67 Do not force benchmarks to be statically linked.
338fc8b Add API exports to secp256k1_nonce_function_default and secp256k1_nonce_function_rfc6979.
52fd03f Merge pull request #320
9f6993f Remove some dead code.
357f8cd Merge pull request #314
118cd82 Use explicit symbol visibility.
4e64608 Include public module headers when compiling modules.
1f41437 Merge pull request #316
fe0d463 Merge pull request #317
cfe0ed9 Fix miscellaneous style nits that irritate overactive static analysis.
2b199de Use the explicit NULL macro for pointer comparisons.
9e90516 Merge pull request #294
dd891e0 Get rid of _t as it is POSIX reserved
201819b Merge pull request #313
912f203 Eliminate a few unbraced statements that crept into the code.
eeab823 Merge pull request #299
486b9bb Use a flags bitfield for compressed option to secp256k1_ec_pubkey_serialize and secp256k1_ec_privkey_export
05732c5 Callback data: Accept pointers to either const or non-const data
1973c73 Bugfix: Reinitialise buffer lengths that have been used as outputs
788038d Use size_t for lengths (at least in external API)
c9d7c2a secp256k1_context_set_{error,illegal}_callback: Restore default handler by passing NULL as function argument
9aac008 secp256k1_context_destroy: Allow NULL argument as a no-op
64b730b secp256k1_context_create: Use unsigned type for flags bitfield
cb04ab5 Merge pull request #309
a551669 Merge pull request #295
81e45ff Update group_impl.h
85e3a2c Merge pull request #112
b2eb63b Merge pull request #293
dc0ce9f [API BREAK] Change argument order to out/outin/in
6d947ca Merge pull request #298
c822693 Merge pull request #301
6d04350 Merge pull request #303
7ab311c Merge pull request #304
5fb3229 Fixes a bug where bench_sign would fail due to passing in too small a buffer.
263dcbc remove unused assignment
b183b41 bugfix: "ARG_CHECK(ctx != NULL)" makes no sense
6da1446 build: fix parallel build
5eb4356 Merge pull request #291
c996d53 Print success
9f443be Move pubkey recovery code to separate module
d49abbd Separate ECDSA recovery tests
439d34a Separate recoverable and normal signatures
a7b046e Merge pull request #289
f66907f Improve/reformat API documentation secp256k1.h
2f77487 Add context building benchmarks
cc623d5 Merge pull request #287
de7e398 small typo fix
9d96e36 Merge pull request #280
432e1ce Merge pull request #283
14727fd Use correct name in gitignore
356b0e9 Actually test static precomputation in Travis
ff3a5df Merge pull request #284
2587208 Merge pull request #212
a5a66c7 Add support for custom EC-Schnorr-SHA256 signatures
d84a378 Merge pull request #252
72ae443 Improve perf. of cmov-based table lookup
92e53fc Implement endomorphism optimization for secp256k1_ecmult_const
ed35d43 Make `secp256k1_scalar_add_bit` conditional; make `secp256k1_scalar_split_lambda_var` constant time
91c0ce9 Add benchmarks for ECDH and const-time multiplication
0739bbb Add ECDH module which works by hashing the output of ecmult_const
4401500 Add constant-time multiply `secp256k1_ecmult_const` for ECDH
e4ce393 build: fix hard-coded usage of "gen_context"
b8e39ac build: don't use BUILT_SOURCES for the static context header
baa75da tests: add a couple tests
ae4f0c6 Merge pull request #278
995c548 Introduce callback functions for dealing with errors.
c333074 Merge pull request #282
18c329c Remove the internal secp256k1_ecdsa_sig_t type
74a2acd Add a secp256k1_ecdsa_signature_t type
23cfa91 Introduce secp256k1_pubkey_t type
4c63780 Merge pull request #269
3e6f1e2 Change rfc6979 implementation to be a generic PRNG
ed5334a Update configure.ac to make it build on OpenBSD
1b68366 Merge pull request #274
a83bb48 Make ecmult static precomputation default
166b32f Merge pull request #276
c37812f Add gen_context src/ecmult_static_context.h to CLEANFILES to fix distclean.
125c15d Merge pull request #275
76f6769 Fix build with static ecmult altroot and make dist.
5133f78 Merge pull request #254
b0a60e6 Merge pull request #258
733c1e6 Add travis build to test the static context.
fbecc38 Add ability to use a statically generated ecmult context.
4fb174d Merge pull request #263
4ab8990 Merge pull request #270
bdf0e0c Merge pull request #271
31d0c1f Merge pull request #273
eb2c8ff Add missing casts to SECP256K1_FE_CONST_INNER
55399c2 Further performance improvements to _ecmult_wnaf
99fd963 Add secp256k1_ec_pubkey_compress(), with test similar to the related decompress() function.
145cc6e Improve performance of _ecmult_wnaf
36b305a Verify the result of GMP modular inverse using non-GMP code
e2a07c7 Fix compilation with C++
2b4cf41 Use pkg-config always when possible, with failover to manual checks for libcrypto
git-subtree-dir: src/secp256k1
git-subtree-split: 9d560f992db26612ce2630b194aef5f44d63a530
The default SSL cipher list introduced with 0.50.0 in
commit 2c87739 was overly restrictive and resulted in
clients unable to negotiate SSL connections.
Adjust the default cipher to the more sensible:
HIGH:MEDIUM:!aNULL:!MD5:!DSS:!3DES:!RC4:!EXPORT
Correct a bug that would not allow an SSL handshake
to properly complete if the port was configured using
the `wss` keyword.
The existing configuration includes 512 and 1024 bit DH
parameters and supports ciphers such as RC4 and 3DES and
hash algorithms like SHA-1 which are no longer considered
secure.
Going forward, use only 2048-bit DH parameters and define
a new default set of modern ciphers to use:
HIGH:!aNULL:!MD5:!DSS:!SHA1:!3DES:!RC4:!EXPORT:!DSS
Additionally, allow administrators who wish to have different
settings to configure custom global and per-port ciphers suites
in the configuration file using the `ssl_ciphers` directive.
c00cd37 Set version to 1.0.0-b23
f662e36 Travis CI improvements:
b05fa33 Fix message constructor and special members
b4722cc Add copy special members
420d1c7 Better logging in async echo server
149e3a2 Add file and line number to thrown exceptions
3e88b83 Tune websocket echo server for performance
git-subtree-dir: src/beast
git-subtree-split: c00cd37b8a441a92755658014fdde97d515ec7ed
The specification requires that we verify that the
signature and modulus of an RSA-SHA256 fulfillment
are both the same length (specifically that they
have "the same number of octets") referring to the
encoded length.
We were, instead, checking the number of bytes that
the signature and modulus had after decoding.
This combines two enhancements to the ledger_data RPC
command and related commands.
The ledger_data RPC command will now return the ledger header
in the first query (the one with no marker specified).
Also, ledger_data and related commands will now provide the
ledger header in binary if binary output is specified.
Modified existing ledgerdata unit test to cover new functionality.
When started with "--start", put all known, non-vetoed
amendments in the genesis ledger. This avoids the need
to wait 256 ledgers before amendments are enabled when
testing with a fresh ledger.
This will allow code that looks at the ledger header to know what version the
SHAMap uses. This is helpful for code that rebuilds ledger binary structures
from the leaves.
Add an amendment to allow gateways to set a "tick size"
for assets they issue. There are no changes unless the
amendment is enabled (since the tick size option cannot
be set).
With the amendment enabled:
AccountSet transactions may set a "TickSize" parameter.
Legal values are 0 and 3-15 inclusive. Zero removes the
setting. 3-15 allow that many decimal digits of precision
in the pricing of offers for assets issued by this account.
For asset pairs with XRP, the tick size imposed, if any,
is the tick size of the issuer of the non-XRP asset. For
asset pairs without XRP, the tick size imposed, if any,
is the smaller of the two issuer's configured tick sizes.
The tick size is imposed by rounding the offer quality
down to nearest tick and recomputing the non-critical
side of the offer. For a buy, the amount offered is
rounded down. For a sell, the amount charged is rounded up.
Gateways must enable a TickSize on their account for this
feature to benefit them.
The primary expected benefit is the elimination of bots
fighting over the tip of the order book. This means:
- Quicker price discovery as outpricing someone by a
microscopic amount is made impossible. Currently
bots can spend hours outbidding each other with no
significant price movement.
- A reduction in offer creation and cancellation spam.
- More offers left on the books as priority means
something when you can't outbid by a microscopic amount.
Migrate tests in uniport-test.js to cpp/jtx. Handle exceptions in
WSClient and JSONRPClient constructors. Use shorter timeout
for HTTP and WS Peers when client is localhost. Add missing call to
start_timer in HTTP Peer. Add incomplete WS Upgrade request test
to prove that server timeout is working.
* Force jtx to request/receive the 2.0 API
* Force the JSON and WebSocket tests to use 2.0 API
* This specifically allows the Websocket to create 2.0 json/ripple
and get back a 2.0 response.
* Add test for malformed json2
* Add check for parse failure
* Add check for params to be in array form.
* Correct type-o discovered in tests due to stricter checking.
* Add API version to the WSClient & JSONRPCClient test
* Update source.dox with more headers
Previously, manifests sent to new peers were marked as history so that
they would not be forwarded. However, this prevented a starting up
node's new manifest from being forwarded beyond its directly connected
peers. Stale or invalid manifests are still not forwarded.
2016-12-23 14:36:10 -08:00
4720 changed files with 522257 additions and 1021133 deletions
*If there is any impact to the public API methods (HTTP / WebSocket), please update https://github.com/xrplf/rippled/blob/develop/API-CHANGELOG.md
*Update API-CHANGELOG.md and add the change directly in this PR by pushing to your PR branch.
*libxrpl: See https://github.com/XRPLF/rippled/blob/develop/docs/build/depend.md
*Peer Protocol: See https://xrpl.org/peer-protocol.html
-->
- [ ] Public API: New feature (new methods and/or new fields)
- [ ] Public API: Breaking change (in general, breaking changes should only impact the next api_version)
- [ ]`libxrpl` change (any change that may affect `libxrpl` or dependents of `libxrpl`)
- [ ] Peer protocol change (must be backward compatible or bump the peer protocol version)
<!--
## Before / After
If relevant, use this section for an English description of the change at a technical level.
If this change affects an API, examples should be included here.
For performance-impacting changes, please provide these details:
1. Is this a new feature, bug fix, or improvement to existing functionality?
2. What behavior/functionality does the change impact?
3. In what processing can the impact be measured? Be as specific as possible - e.g. RPC client call, payment transaction that involves LOB, AMM, caching, DB operations, etc.
4. Does this change affect concurrent processing - e.g. does it involve acquiring locks, multi-threaded processing, or async processing?
-->
<!--
## Test Plan
If helpful, please describe the tests that you ran to verify your changes and provide instructions so that others can reproduce.
This section may not be needed if your change includes thoroughly commented unit tests.
parser.add_argument('-a','--all',help='Set to generate all configurations (generally used when merging a PR) or leave unset to generate a subset of configurations (generally used when committing to a PR).',action="store_true")
parser.add_argument('-c','--config',help='Path to the JSON file containing the strategy matrix configurations.',required=False,type=Path)
This changelog is intended to list all updates to the [public API methods](https://xrpl.org/public-api-methods.html).
For info about how [API versioning](https://xrpl.org/request-formatting.html#api-versioning) works, including examples, please view the [XLS-22d spec](https://github.com/XRPLF/XRPL-Standards/discussions/54). For details about the implementation of API versioning, view the [implementation PR](https://github.com/XRPLF/rippled/pull/3155). API versioning ensures existing integrations and users continue to receive existing behavior, while those that request a higher API version will experience new behavior.
The API version controls the API behavior you see. This includes what properties you see in responses, what parameters you're permitted to send in requests, and so on. You specify the API version in each of your requests. When a breaking change is introduced to the `rippled` API, a new version is released. To avoid breaking your code, you should set (or increase) your version when you're ready to upgrade.
For a log of breaking changes, see the **API Version [number]** headings. In general, breaking changes are associated with a particular API Version number. For non-breaking changes, scroll to the **XRP Ledger version [x.y.z]** headings. Non-breaking changes are associated with a particular XRP Ledger (`rippled`) release.
## API Version 2
API version 2 is available in `rippled` version 2.0.0 and later. To use this API, clients specify `"api_version" : 2` in each request.
#### Removed methods
In API version 2, the following deprecated methods are no longer available: (https://github.com/XRPLF/rippled/pull/4759)
-`tx_history` - Instead, use other methods such as `account_tx` or `ledger` with the `transactions` field set to `true`.
-`ledger_header` - Instead, use the `ledger` method.
#### Modifications to JSON transaction element in V2
In API version 2, JSON elements for transaction output have been changed and made consistent for all methods which output transactions. (https://github.com/XRPLF/rippled/pull/4775)
This helps to unify the JSON serialization format of transactions. (https://github.com/XRPLF/clio/issues/722, https://github.com/XRPLF/rippled/issues/4727)
- JSON transaction element is named `tx_json`
- Binary transaction element is named `tx_blob`
- JSON transaction metadata element is named `meta`
- Binary transaction metadata element is named `meta_blob`
Additionally, these elements are now consistently available next to `tx_json` (i.e. sibling elements), where possible:
-`hash` - Transaction ID. This data was stored inside transaction output in API version 1, but in API version 2 is a sibling element.
-`ledger_index` - Ledger index (only set on validated ledgers)
-`ledger_hash` - Ledger hash (only set on closed or validated ledgers)
-`close_time_iso` - Ledger close time expressed in ISO 8601 time format (only set on validated ledgers)
-`validated` - Bool element set to `true` if the transaction is in a validated ledger, otherwise `false`
This change affects the following methods:
-`tx` - Transaction data moved into element `tx_json` (was inline inside `result`) or, if binary output was requested, moved from `tx` to `tx_blob`. Renamed binary transaction metadata element (if it was requested) from `meta` to `meta_blob`. Changed location of `hash` and added new elements
-`account_tx` - Renamed transaction element from `tx` to `tx_json`. Renamed binary transaction metadata element (if it was requested) from `meta` to `meta_blob`. Changed location of `hash` and added new elements
-`transaction_entry` - Renamed transaction metadata element from `metadata` to `meta`. Changed location of `hash` and added new elements
-`subscribe` - Renamed transaction element from `transaction` to `tx_json`. Changed location of `hash` and added new elements
-`sign`, `sign_for`, `submit` and `submit_multisigned` - Changed location of `hash` element.
#### Modification to `Payment` transaction JSON schema
When reading Payments, the `Amount` field should generally **not** be used. Instead, use [delivered_amount](https://xrpl.org/partial-payments.html#the-delivered_amount-field) to see the amount that the Payment delivered. To clarify its meaning, the `Amount` field is being renamed to `DeliverMax`. (https://github.com/XRPLF/rippled/pull/4733)
- In `Payment` transaction type, JSON RPC field `Amount` is renamed to `DeliverMax`. To enable smooth client transition, `Amount` is still handled, as described below: (https://github.com/XRPLF/rippled/pull/4733)
- On JSON RPC input (e.g. `submit_multisigned` etc. methods), `Amount` is recognized as an alias to `DeliverMax` for both API version 1 and version 2 clients.
- On JSON RPC input, submitting both `Amount` and `DeliverMax` fields is allowed _only_ if they are identical; otherwise such input is rejected with `rpcINVALID_PARAMS` error.
- On JSON RPC output (e.g. `subscribe`, `account_tx` etc. methods), `DeliverMax` is present in both API version 1 and version 2.
- On JSON RPC output, `Amount` is only present in API version 1 and _not_ in version 2.
#### Modifications to account_info response
-`signer_lists` is returned in the root of the response. (In API version 1, it was nested under `account_data`.) (https://github.com/XRPLF/rippled/pull/3770)
- When using an invalid `signer_lists` value, the API now returns an "invalidParams" error. (https://github.com/XRPLF/rippled/pull/4585)
- (`signer_lists` must be a boolean. In API version 1, strings were accepted and may return a normal response - i.e. as if `signer_lists` were `true`.)
#### Modifications to [account_tx](https://xrpl.org/account_tx.html#account_tx) response
- Using `ledger_index_min`, `ledger_index_max`, and `ledger_index` returns `invalidParams` because if you use `ledger_index_min` or `ledger_index_max`, then it does not make sense to also specify `ledger_index`. In API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4571)
- The same applies for `ledger_index_min`, `ledger_index_max`, and `ledger_hash`. (https://github.com/XRPLF/rippled/issues/4545#issuecomment-1565065579)
- Using a `ledger_index_min` or `ledger_index_max` beyond the range of ledgers that the server has:
- returns `lgrIdxMalformed` in API version 2. Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/issues/4288)
- Attempting to use a non-boolean value (such as a string) for the `binary` or `forward` parameters returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4620)
#### Modifications to [noripple_check](https://xrpl.org/noripple_check.html#noripple_check) response
- Attempting to use a non-boolean value (such as a string) for the `transactions` parameter returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4620)
## API Version 1
This version is supported by all `rippled` versions. For WebSocket and HTTP JSON-RPC requests, it is currently the default API version used when no `api_version` is specified.
The [commandline](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/request-formatting/#commandline-format) always uses the latest API version. The command line is intended for ad-hoc usage by humans, not programs or automated scripts. The command line is not meant for use in production code.
### Inconsistency: server_info - network_id
The `network_id` field was added in the `server_info` response in version 1.5.0 (2019), but it is not returned in [reporting mode](https://xrpl.org/rippled-server-modes.html#reporting-mode). However, use of reporting mode is now discouraged, in favor of using [Clio](https://github.com/XRPLF/clio) instead.
## XRP Ledger server version 2.5.0
As of 2025-04-04, version 2.5.0 is in development. You can use a pre-release version by building from source or [using the `nightly` package](https://xrpl.org/docs/infrastructure/installation/install-rippled-on-ubuntu).
### Additions and bugfixes in 2.5.0
-`channel_authorize`: If `signing_support` is not enabled in the config, the RPC is disabled.
## XRP Ledger server version 2.4.0
[Version 2.4.0](https://github.com/XRPLF/rippled/releases/tag/2.4.0) was released on March 4, 2025.
### Additions and bugfixes in 2.4.0
-`ledger_entry`: `state` is added an alias for `ripple_state`.
-`ledger_entry`: Enables case-insensitive filtering by canonical name in addition to case-sensitive filtering by RPC name.
-`validators`: Added new field `validator_list_threshold` in response.
-`simulate`: A new RPC that executes a [dry run of a transaction submission](https://github.com/XRPLF/XRPL-Standards/tree/master/XLS-0069d-simulate#2-rpc-simulate)
- Signing methods autofill fees better and properly handle transactions that don't have a base fee, and will also autofill the `NetworkID` field.
## XRP Ledger server version 2.3.0
[Version 2.3.0](https://github.com/XRPLF/rippled/releases/tag/2.3.0) was released on Nov 25, 2024.
### Breaking changes in 2.3.0
-`book_changes`: If the requested ledger version is not available on this node, a `ledgerNotFound` error is returned and the node does not attempt to acquire the ledger from the p2p network (as with other non-admin RPCs).
Admins can still attempt to retrieve old ledgers with the `ledger_request` RPC.
### Additions and bugfixes in 2.3.0
-`book_changes`: Returns a `validated` field in its response, which was missing in prior versions.
## XRP Ledger server version 2.2.0
[Version 2.2.0](https://github.com/XRPLF/rippled/releases/tag/2.2.0) was released on Jun 5, 2024. The following additions are non-breaking (because they are purely additive):
- The `feature` method now has a non-admin mode for users. (It was previously only available to admin connections.) The method returns an updated list of amendments, including their names and other information. ([#4781](https://github.com/XRPLF/rippled/pull/4781))
## XRP Ledger server version 2.0.0
[Version 2.0.0](https://github.com/XRPLF/rippled/releases/tag/2.0.0) was released on Jan 9, 2024. The following additions are non-breaking (because they are purely additive):
-`server_definitions`: A new RPC that generates a `definitions.json`-like output that can be used in XRPL libraries.
- In `Payment` transactions, `DeliverMax` has been added. This is a replacement for the `Amount` field, which should not be used. Typically, the `delivered_amount` (in transaction metadata) should be used. To ease the transition, `DeliverMax` is present regardless of API version, since adding a field is non-breaking.
- API version 2 has been moved from beta to supported, meaning that it is generally available (regardless of the `beta_rpc_api` setting).
## XRP Ledger server version 2.2.0
The following is a non-breaking addition to the API.
- The `feature` method now has a non-admin mode for users. (It was previously only available to admin connections.) The method returns an updated list of amendments, including their names and other information. ([#4781](https://github.com/XRPLF/rippled/pull/4781))
## XRP Ledger server version 1.12.0
[Version 1.12.0](https://github.com/XRPLF/rippled/releases/tag/1.12.0) was released on Sep 6, 2023. The following additions are non-breaking (because they are purely additive).
-`server_info`: Added `ports`, an array which advertises the RPC and WebSocket ports. This information is also included in the `/crawl` endpoint (which calls `server_info` internally). `grpc` and `peer` ports are also included. (https://github.com/XRPLF/rippled/pull/4427)
-`ports` contains objects, each containing a `port` for the listening port (a number string), and a `protocol` array listing the supported protocols on that port.
- This allows crawlers to build a more detailed topology without needing to port-scan nodes.
- (For peers and other non-admin clients, the info about admin ports is excluded.)
- Clawback: The following additions are gated by the Clawback amendment (`featureClawback`). (https://github.com/XRPLF/rippled/pull/4553)
- Adds an [AccountRoot flag](https://xrpl.org/accountroot.html#accountroot-flags) called `lsfAllowTrustLineClawback` (https://github.com/XRPLF/rippled/pull/4617)
- Adds the corresponding `asfAllowTrustLineClawback` [AccountSet Flag](https://xrpl.org/accountset.html#accountset-flags) as well.
- Clawback is disabled by default, so if an issuer desires the ability to claw back funds, they must use an `AccountSet` transaction to set the AllowTrustLineClawback flag. They must do this before creating any trust lines, offers, escrows, payment channels, or checks.
- Adds the [Clawback transaction type](https://github.com/XRPLF/XRPL-Standards/blob/master/XLS-39d-clawback/README.md#331-clawback-transaction), containing these fields:
-`Account`: The issuer of the asset being clawed back. Must also be the sender of the transaction.
-`Amount`: The amount being clawed back, with the `Amount.issuer` being the token holder's address.
`tfWithdrawAll`, `tfOneAssetWithdrawAll` which allow a trader to specify different fields combination
for `AMMDeposit` and `AMMWithdraw` transactions.
- Adds new transaction result codes:
- tecUNFUNDED_AMM: insufficient balance to fund AMM. The account does not have funds for liquidity provision.
- tecAMM_BALANCE: AMM has invalid balance. Calculated balances greater than the current pool balances.
- tecAMM_FAILED: AMM transaction failed. Fails due to a processing failure.
- tecAMM_INVALID_TOKENS: AMM invalid LP tokens. Invalid input values, format, or calculated values.
- tecAMM_EMPTY: AMM is in empty state. Transaction requires AMM in non-empty state (LP tokens > 0).
- tecAMM_NOT_EMPTY: AMM is not in empty state. Transaction requires AMM in empty state (LP tokens == 0).
- tecAMM_ACCOUNT: AMM account. Clawback of AMM account.
- tecINCOMPLETE: Some work was completed, but more submissions required to finish. AMMDelete partially deletes the trustlines.
## XRP Ledger server version 1.11.0
[Version 1.11.0](https://github.com/XRPLF/rippled/releases/tag/1.11.0) was released on Jun 20, 2023.
### Breaking changes in 1.11
- Added the ability to mark amendments as obsolete. For the `feature` admin API, there is a new possible value for the `vetoed` field. (https://github.com/XRPLF/rippled/pull/4291)
- The value of `vetoed` can now be `true`, `false`, or `"Obsolete"`.
- Removed the acceptance of seeds or public keys in place of account addresses. (https://github.com/XRPLF/rippled/pull/4404)
- This simplifies the API and encourages better security practices (i.e. seeds should never be sent over the network).
- For the `ledger_data` method, when all entries are filtered out, the `state` field of the response is now an empty list (in other words, an empty array, `[]`). (Previously, it would return `null`.) While this is technically a breaking change, the new behavior is consistent with the documentation, so this is considered only a bug fix. (https://github.com/XRPLF/rippled/pull/4398)
- If and when the `fixNFTokenRemint` amendment activates, there will be a new AccountRoot field, `FirstNFTSequence`. This field is set to the current account sequence when the account issues their first NFT. If an account has not issued any NFTs, then the field is not set. ([#4406](https://github.com/XRPLF/rippled/pull/4406))
- There is a new account deletion restriction: an account can only be deleted if `FirstNFTSequence` + `MintedNFTokens` + `256` is less than the current ledger sequence.
- This is potentially a breaking change if clients have logic for determining whether an account can be deleted.
- NetworkID
- For sidechains and networks with a network ID greater than 1024, there is a new [transaction common field](https://xrpl.org/transaction-common-fields.html), `NetworkID`. (https://github.com/XRPLF/rippled/pull/4370)
- This field helps to prevent replay attacks and is now required for chains whose network ID is 1025 or higher.
- The field must be omitted for Mainnet, so there is no change for Mainnet users.
- There are three new local error codes:
-`telNETWORK_ID_MAKES_TX_NON_CANONICAL`: a `NetworkID` is present but the chain's network ID is less than 1025. Remove the field from the transaction, and try again.
-`telREQUIRES_NETWORK_ID`: a `NetworkID` is required, but is not present. Add the field to the transaction, and try again.
-`telWRONG_NETWORK`: a `NetworkID` is specified, but it is for a different network. Submit the transaction to a different server which is connected to the correct network.
### Additions and bug fixes in 1.11
- Added `nftoken_id`, `nftoken_ids` and `offer_id` meta fields into NFT `tx` and `account_tx` responses. (https://github.com/XRPLF/rippled/pull/4447)
- Added an `account_flags` object to the `account_info` method response. (https://github.com/XRPLF/rippled/pull/4459)
- Added `NFTokenPages` to the `account_objects` RPC. (https://github.com/XRPLF/rippled/pull/4352)
- Fixed: `marker` returned from the `account_lines` command would not work on subsequent commands. (https://github.com/XRPLF/rippled/pull/4361)
- If the `XRPFees` feature is enabled, the `fee_ref` field will be
removed from the [ledger subscription stream](https://xrpl.org/subscribe.html#ledger-stream), because it will no longer
have any meaning.
# Unit tests for API changes
The following information is useful to developers contributing to this project:
The purpose of unit tests is to catch bugs and prevent regressions. In general, it often makes sense to create a test function when there is a breaking change to the API. For APIs that have changed in a new API version, the tests should be modified so that both the prior version and the new version are properly tested.
To take one example: for `account_info` version 1, WebSocket and JSON-RPC behavior should be tested. The latest API version, i.e. API version 2, should be tested over WebSocket, JSON-RPC, and command line.
| These instructions assume you have a C++ development environment ready with Git, Python, Conan, CMake, and a C++ compiler. For help setting one up on Linux, macOS, or Windows, [see this guide](./docs/build/environment.md). |
> These instructions also assume a basic familiarity with Conan and CMake.
> If you are unfamiliar with Conan, you can read our
> [crash course](./docs/build/conan.md) or the official [Getting Started][3]
> walkthrough.
## Branches
For a stable release, choose the `master` branch or one of the [tagged
You can manage multiple Conan profiles in the directory
`$(conan config home)/profiles`, for example renaming `default` to a different
name and then creating a new `default` profile for a different compiler.
#### Select language
The default profile created by Conan will typically select different C++ dialect
than C++20 used by this project. You should set `20` in the profile line
starting with `compiler.cppstd=`. For example:
```bash
sed -i.bak -e 's|^compiler\.cppstd=.*$|compiler.cppstd=20|'$(conan config home)/profiles/default
```
#### Select standard library in Linux
**Linux** developers will commonly have a default Conan [profile][] that
compiles with GCC and links with libstdc++. If you are linking with libstdc++
(see profile setting `compiler.libcxx`), then you will need to choose the
`libstdc++11` ABI:
```bash
sed -i.bak -e 's|^compiler\.libcxx=.*$|compiler.libcxx=libstdc++11|'$(conan config home)/profiles/default
```
#### Select architecture and runtime in Windows
**Windows** developers may need to use the x64 native build tools. An easy way
to do that is to run the shortcut "x64 Native Tools Command Prompt" for the
version of Visual Studio that you have installed.
Windows developers must also build `rippled` and its dependencies for the x64
architecture:
```bash
sed -i.bak -e 's|^arch=.*$|arch=x86_64|'$(conan config home)/profiles/default
```
**Windows** developers also must select static runtime:
```bash
sed -i.bak -e 's|^compiler\.runtime=.*$|compiler.runtime=static|'$(conan config home)/profiles/default
```
#### Clang workaround for grpc
If your compiler is clang, version 19 or later, or apple-clang, version 17 or
later, you may encounter a compilation error while building the `grpc`
dependency:
```text
In file included from .../lib/promise/try_seq.h:26:
.../lib/promise/detail/basic_seq.h:499:38: error: a template argument list is expected after a name prefixed by the template keyword [-Wmissing-template-arg-list-after-template-kw]
If your compiler is gcc, version 12, and you have enabled `werr` option, you may
encounter a compilation error such as:
```text
/usr/include/c++/12/bits/char_traits.h:435:56: error: 'void* __builtin_memcpy(void*, const void*, long unsigned int)' accessing 9223372036854775810 or more bytes at offsets [2, 9223372036854775807] and 1 may overlap up to 9223372036854775813 bytes at offset -3 [-Werror=restrict]
| `coverage` | OFF | Prepare the coverage report. |
| `san` | N/A | Enable a sanitizer with Clang. Choices are `thread` and `address`. |
| `tests` | OFF | Build tests. |
| `unity` | OFF | Configure a unity build. |
| `xrpld` | OFF | Build the xrpld (`rippled`) application, and not just the libxrpl library. |
| `werr` | OFF | Treat compilation warnings as errors |
| `wextra` | OFF | Enable additional compilation warnings |
[Unity builds][5] may be faster for the first build
(at the cost of much more memory) since they concatenate sources into fewer
translation units. Non-unity builds may be faster for incremental builds,
and can be helpful for detecting `#include` omissions.
## Troubleshooting
### Conan
After any updates or changes to dependencies, you may need to do the following:
1. Remove your build directory.
2. Remove individual libraries from the Conan cache, e.g.
```bash
conan remove 'grpc/*'
```
**or**
Remove all libraries from Conan cache:
```bash
conan remove '*'
```
3. Re-run [conan export](#patched-recipes) if needed.
4. [Regenerate lockfile](#conan-lockfile).
5. Re-run [conan install](#build-and-test).
#### ERROR: Package not resolved
If you're seeing an error like `ERROR: Package 'snappy/1.1.10' not resolved: Unable to find 'snappy/1.1.10#968fef506ff261592ec30c574d4a7809%1756234314.246' in remotes.`,
please add `xrplf` remote or re-run `conan export` for [patched recipes](#patched-recipes).
### `protobuf/port_def.inc` file not found
If `cmake --build .` results in an error due to a missing a protobuf file, then
you might have generated CMake files for a different `build_type` than the
`CMAKE_BUILD_TYPE` you passed to Conan.
```
/rippled/.build/pb-xrpl.libpb/xrpl/proto/ripple.pb.h:10:10: fatal error: 'google/protobuf/port_def.inc' file not found
10 | #include <google/protobuf/port_def.inc>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
For example, if you want to build Debug:
1. For conan install, pass `--settings build_type=Debug`
2. For cmake, pass `-DCMAKE_BUILD_TYPE=Debug`
## Add a Dependency
If you want to experiment with a new package, follow these steps:
1. Search for the package on [Conan Center](https://conan.io/center/).
2. Modify [`conanfile.py`](./conanfile.py):
- Add a version of the package to the `requires` property.
- Change any default options for the package by adding them to the
Ripple is a network of computers which use the [Ripple consensus algorithm]
(https://www.youtube.com/watch?v=pj1QVb1vlC0) to atomically settle and record
transactions on a secure distributed database, the Ripple Consensus Ledger
(RCL). Because of its distributed nature, the RCL offers transaction immutability
without a central operator. The RCL contains a built-in currency exchange and its
path-finding algorithm finds competitive exchange rates across order books
and currency pairs.
# The XRP Ledger
### Key Features
- **Distributed**
- Direct account-to-account settlement with no central operator
- Decentralized global market for competitive FX
- **Secure**
- Transactions are cryptographically signed using ECDSA or Ed25519
- Multi-signing capabilities
- **Scalable**
- Capacity to process the world’s cross-border payments volume
- Easy access to liquidity through a competitive FX marketplace
The [XRP Ledger](https://xrpl.org/) is a decentralized cryptographic ledger powered by a network of peer-to-peer nodes. The XRP Ledger uses a novel Byzantine Fault Tolerant consensus algorithm to settle and record transactions in a secure distributed database without a central operator.
## Cross-border payments
Ripple enables banks to settle cross-border payments in real-time, with
end-to-end transparency, and at lower costs. Banks can provide liquidity
for FX themselves or source it from third parties.
## XRP
As Ripple adoption grows, so do the number of currencies and counterparties.
Liquidity providers need to maintain accounts with each counterparty for
each currency – a capital- and time-intensive endeavor that spreads liquidity
thin. Further, some transactions, such as exotic currency trades, will require
multiple trading parties, who each layer costs to the transaction. Thin
liquidity and many intermediary trading parties make competitive pricing
challenging.
[XRP](https://xrpl.org/xrp.html) is a public, counterparty-free crypto-asset native to the XRP Ledger, and is designed as a gas token for network services and to bridge different currencies. XRP is traded on the open-market and is available for anyone to access. The XRP Ledger was created in 2012 with a finite supply of 100 billion units of XRP.
## rippled
### XRP as a Bridge Currency
Ripple can bridge even exotic currency pairs directly through XRP. Similar to
USD in today’s currency market, XRP allows liquidity providers to focus on
offering competitive FX rates on fewer pairs and adding depth to order books.
Unlike USD, trading through XRP does not require bank accounts, service fees,
counterparty risk, or additional operational costs. By using XRP, liquidity
providers can specialize in certain currency corridors, reduce operational
costs, and ultimately, offer more competitive FX pricing.
The server software that powers the XRP Ledger is called `rippled` and is available in this repository under the permissive [ISC open-source license](LICENSE.md). The `rippled` server software is written primarily in C++ and runs on a variety of platforms. The `rippled` server software can run in several modes depending on its [configuration](https://xrpl.org/rippled-server-modes.html).
If you are interested in running an **API Server** (including a **Full History Server**), take a look at [Clio](https://github.com/XRPLF/clio). (rippled Reporting Mode has been replaced by Clio.)
# rippled - Ripple server
`rippled` is the reference server implementation of the Ripple
protocol. To learn more about how to build and run a `rippled`
- [Read the build instructions in `BUILD.md`](BUILD.md)
- If you encounter any issues, please [open an issue](https://github.com/XRPLF/rippled/issues)
### License
`rippled` is open source and permissively licensed under the
ISC license. See the LICENSE file for more details.
## Key Features of the XRP Ledger
#### Repository Contents
- **[Censorship-Resistant Transaction Processing][]:** No single party decides which transactions succeed or fail, and no one can "roll back" a transaction after it completes. As long as those who choose to participate in the network keep it healthy, they can settle transactions in seconds.
- **[Fast, Efficient Consensus Algorithm][]:** The XRP Ledger's consensus algorithm settles transactions in 4 to 5 seconds, processing at a throughput of up to 1500 transactions per second. These properties put XRP at least an order of magnitude ahead of other top digital assets.
- **[Finite XRP Supply][]:** When the XRP Ledger began, 100 billion XRP were created, and no more XRP will ever be created. The available supply of XRP decreases slowly over time as small amounts are destroyed to pay transaction fees.
- **[Responsible Software Governance][]:** A team of full-time developers at Ripple & other organizations maintain and continually improve the XRP Ledger's underlying software with contributions from the open-source community. Ripple acts as a steward for the technology and an advocate for its interests.
- **[Secure, Adaptable Cryptography][]:** The XRP Ledger relies on industry standard digital signature systems like ECDSA (the same scheme used by Bitcoin) but also supports modern, efficient algorithms like Ed25519. The extensible nature of the XRP Ledger's software makes it possible to add and disable algorithms as the state of the art in cryptography advances.
- **[Modern Features][]:** Features like Escrow, Checks, and Payment Channels support financial applications atop of the XRP Ledger. This toolbox of advanced features comes with safety features like a process for amending the network and separate checks against invariant constraints.
- **[On-Ledger Decentralized Exchange][]:** In addition to all the features that make XRP useful on its own, the XRP Ledger also has a fully-functional accounting system for tracking and trading obligations denominated in any way users want, and an exchange built into the protocol. The XRP Ledger can settle long, cross-currency payment paths and exchanges of multiple currencies in atomic transactions, bridging gaps of trust with XRP.
| Folder | Contents |
|---------|----------|
| ./bin | Scripts and data files for Ripple integrators. |
| ./build | Intermediate and final build outputs. |
| ./Builds| Platform or IDE-specific project files. |
| ./doc | Documentation and example configuration files. |
Some of the directories under `src` are external repositories inlined via
git-subtree. See the corresponding README for more details.
Here are some good places to start learning the source code:
##For more information:
- Read the markdown files in the source tree: `src/ripple/**/*.md`.
- Read [the levelization document](.github/scripts/levelization) to get an idea of the internal dependency graph.
- In the big picture, the `main` function constructs an `ApplicationImp` object, which implements the `Application` virtual interface. Almost every component in the application takes an `Application&` parameter in its constructor, typically named `app` and stored as a member variable `app_`. This allows most components to depend on any other component.
For more details on operating an XRP Ledger server securely, please visit https://xrpl.org/manage-the-rippled-server.html.
# Security Policy
## Supported Versions
Software constantly evolves. In order to focus resources, we only generally only accept vulnerability reports that affect recent and current versions of the software. We always accept reports for issues present in the **master**, **release** or **develop** branches, and with proposed, [open pull requests](https://github.com/ripple/rippled/pulls).
## Identifying and Reporting Vulnerabilities
We take security seriously and we do our best to ensure that all our releases are bug free. But we aren't perfect and sometimes things will slip through.
### Responsible Investigation
We urge you to examine our code carefully and responsibly, and to disclose any issues that you identify in a responsible fashion.
Responsible investigation includes, but isn't limited to, the following:
- Not performing tests on the main network. If testing is necessary, use the [Testnet or Devnet](https://xrpl.org/xrp-testnet-faucet.html).
- Not targeting physical security measures, or attempting to use social engineering, spam, distributed denial of service (DDOS) attacks, etc.
- Investigating bugs in a way that makes a reasonable, good faith effort not to be disruptive or harmful to the XRP Ledger and the broader ecosystem.
### Responsible Disclosure
If you discover a vulnerability or potential threat, or if you _think_
you have, please reach out by dropping an email using the contact
information below.
Your report should include the following:
- Your contact information (typically, an email address);
- The description of the vulnerability;
- The attack scenario (if any);
- The steps to reproduce the vulnerability;
- Any other relevant details or artifacts, including code, scripts or patches.
In your email, please describe the issue or potential threat. If possible, include a "repro" (code that can reproduce the issue) or describe the best way to reproduce and replicate the issue. Please make your report as detailed and comprehensive as possible.
For more information on responsible disclosure, please read this [Wikipedia article](https://en.wikipedia.org/wiki/Responsible_disclosure).
## Report Handling Process
Please report the bug directly to us and limit further disclosure. If you want to prove that you knew the bug as of a given time, consider using a cryptographic precommitment: hash the content of your report and publish the hash on a medium of your choice (e.g. on Twitter or as a memo in a transaction) as "proof" that you had written the text at a given point in time.
Once we receive a report, we:
1. Assign two people to independently evaluate the report;
2. Consider their recommendations;
3. If action is necessary, formulate a plan to address the issue;
4. Communicate privately with the reporter to explain our plan.
5. Prepare, test and release a version which fixes the issue; and
6. Announce the vulnerability publicly.
We will triage and respond to your disclosure within 24 hours. Beyond that, we will work to analyze the issue in more detail, formulate, develop and test a fix.
While we commit to responding with 24 hours of your initial report with our triage assessment, we cannot guarantee a response time for the remaining steps. We will communicate with you throughout this process, letting you know where we are and keeping you updated on the timeframe.
## Bug Bounty Program
[Ripple](https://ripple.com) is generously sponsoring a bug bounty program for vulnerabilities in [`rippled`](https://github.com/XRPLF/rippled) (and other related projects, like [`xrpl.js`](https://github.com/XRPLF/xrpl.js), [`xrpl-py`](https://github.com/XRPLF/xrpl-py), [`xrpl4j`](https://github.com/XRPLF/xrpl4j)).
This program allows us to recognize and reward individuals or groups that identify and report bugs. In summary, in order to qualify for a bounty, the bug must be:
1.**In scope**. Only bugs in software under the scope of the program qualify. Currently, that means `rippled`, `xrpl.js`, `xrpl-py`, `xrpl4j`.
2.**Relevant**. A security issue, posing a danger to user funds, privacy, or the operation of the XRP Ledger.
3.**Original and previously unknown**. Bugs that are already known and discussed in public do not qualify. Previously reported bugs, even if publicly unknown, are not eligible.
4.**Specific**. We welcome general security advice or recommendations, but we cannot pay bounties for that.
5.**Fixable**. There has to be something we can do to permanently fix the problem. Note that bugs in other people’s software may still qualify in some cases. For example, if you find a bug in a library that we use which can compromise the security of software that is in scope and we can get it fixed, you may qualify for a bounty.
6.**Unused**. If you use the exploit to attack the XRP Ledger, you do not qualify for a bounty. If you report a vulnerability used in an ongoing or past attack and there is specific, concrete evidence that suggests you are the attacker we reserve the right not to pay a bounty.
The amount paid varies dramatically. Vulnerabilities that are harmless on their own, but could form part of a critical exploit will usually receive a bounty. Full-blown exploits can receive much higher bounties. Please don’t hold back partial vulnerabilities while trying to construct a full-blown exploit. We will pay a bounty to anyone who reports a complete chain of vulnerabilities even if they have reported each component of the exploit separately and those vulnerabilities have been fixed in the meantime. However, to qualify for a the full bounty, you must to have been the first to report each of the partial exploits.
### Contacting Us
To report a qualifying bug, please send a detailed report to:
echo"Use the following command on your local machine to download from your rippled instance: scp <remote_rippled_username>@<remote_host>:${tmp_loc}/info-package.tar.gz <path/to/local_machine/directory>"| tee /dev/fd/3
# This code comes from http://github.com/warner/python-ecdsa
from._versionimportget_versions
__version__=get_versions()['version']
delget_versions
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
