Compare commits

...

1660 Commits

Author SHA1 Message Date
Elliot Lee
22cdb5728b Set version to 2.0.1 2024-01-29 08:36:10 -08:00
Bronek Kozicki
901152bd93 chore: retry codecov upload (#4896)
Update to #4849, using a workaround for spurious codecov upload errors.

Spurious codecov upload errors are expected in public repos which rely
on PRs via forks. Retrying uploads is a decent and easy workaround.
2024-01-24 16:46:24 -08:00
John Freeman
d9a5bca625 docs: fix broken links in docs/build/conan.md (#4699) 2024-01-23 20:35:22 -08:00
Elliot Lee
1676e9fe21 Set version to 2.0.1-rc1 2024-01-22 14:48:05 -08:00
Bronek Kozicki
fad9d639bf test: improve code coverage reporting (#4849)
* Speed up the generation of coverage reports by using multiple cores.

* Add codecov step to coverage workflow.
2024-01-22 13:09:18 -08:00
Chenna Keshava B S
efe6722bf8 docs: update help message about unit test-suite pattern matching (#4846)
Update the "rippled --help" message for the "-u" parameter. This
documents the unit test name pattern matching rule implemented by #4634.

Fix #4800
2024-01-19 21:55:57 -08:00
Mark Travis
a41f38547b Revert "Asynchronously write batches to NuDB. (#4503)" (#4882)
This reverts commit 1d9db1bfdd.

This improves the stability of online deletion.
2024-01-19 13:26:26 -08:00
Elliot Lee
87ee7868ea Set version to 2.0.1-b1 2024-01-17 16:40:20 -08:00
Elliot Lee
861bd1a96e docs: add Performance type to PR template (#4875) 2024-01-17 16:37:28 -08:00
Bronek Kozicki
6ac2b705dd test: add DeliverMax to more JSONRPC tests (#4826)
Minor change in unit tests to improve testing scope.
2024-01-16 20:38:35 -08:00
John Freeman
fe4d6c68b5 fix: change default send_queue_limit to 500 (#4867)
Clients subscribed to `transactions` over WebSocket are being
disconnected because the traffic exceeds the default `send_queue_limit`
of 100.

This commit changes the default configuration, not the default in code.

Fix #4866
2024-01-16 11:06:46 -08:00
Chenna Keshava B S
5a7af5bb77 fix: clang warning about deprecated sprintf usage (#4747)
Resolves a warning that was emitted from the clang compiler. Switches
usage of the sprintf function to the recommended snprintf function.

Warning was observed in Apple clang version 15.0.0 (clang-1500.0.40.1).

Fix #4569
2024-01-15 22:32:52 -08:00
Ed Hennis
d9f90c84c0 Improve lifetime management of ledger objects (SLEs) to prevent runaway memory usage: (#4822)
* Add logging for Application.cpp sweep()
* Improve lifetime management of ledger objects (`SLE`s)
* Only store SLE digest in CachedView; get SLEs from CachedSLEs
* Also force release of last ledger used for path finding if there are
  no path finding requests to process
* Count more ST objects (derive from `CountedObject`)
* Track CachedView stats in CountedObjects
* Rename the CachedView counters
* Fix the scope of the digest lookup lock

Before this patch, if you asked "is it caching?" It was always caching.
2024-01-12 12:42:33 -05:00
Ed Hennis
4308407dc1 WebSocket should only call async_close once (#4848)
Prevent WebSocket connections from trying to close twice.

The issue only occurs in debug builds (assertions are disabled in
release builds, including published packages), and when the WebSocket
connections are unprivileged. The assert (and WRN log) occurs when a
client drives up the resource balance enough to be forcibly disconnected
while there are still messages pending to be sent.

Thanks to @lathanbritz for discovering this issue in #4822.
2024-01-11 20:11:09 -05:00
Elliot Lee
2b0313d60c Set version to 2.0.0 2024-01-08 13:22:59 -08:00
Michael Legleux
350d213ee8 Set version to 2.0.0-rc7
* Ignore python error about modifying system python (#4863)
2024-01-05 12:55:45 -08:00
Elliot Lee
ca3198164c Set version to 2.0.0-rc6 2023-12-20 09:33:33 -08:00
Scott Schurr
c53a5e7a72 Revert "Apply transaction batches in periodic intervals (#4504)" (#4852)
This reverts commit 002893f280.

There were two files with conflicts in the automated revert:

- src/ripple/rpc/impl/RPCHelpers.h and
- src/test/rpc/JSONRPC_test.cpp

Those files were manually resolved.
2023-12-20 09:30:12 -08:00
Bronek Kozicki
ffb53f2085 Revert "Add ProtocolStart and GracefulClose P2P protocol messages (#3839)" (#4850)
This reverts commit 8f89694fae.
2023-12-19 12:52:25 -08:00
Elliot Lee
3b191a3097 docs(API-CHANGELOG): clarify changes for V2 (#4773) 2023-12-06 14:27:33 -08:00
Hussein Badakhchani
656948cd0f fix typo: 'of' instead of 'on' (#4821)
Co-authored-by: Hussein Badakhchani <hoos@alsoug.com>
2023-12-06 14:26:18 -08:00
Elliot Lee
46f3d3ef61 Set version to 2.0.0-rc5 2023-11-30 21:46:37 -08:00
Bronek Kozicki
06251aa76f Workarounds for gcc-13 compatibility (#4817)
Workaround for compilation errors with gcc-13 and other compilers
relying on `libstdc++` version 13. This is temporary until actual fix is
available for us to use: https://github.com/boostorg/beast/pull/2682

Some boost.beast files (which we do use) rely on an old gcc-12 behaviour
where `#include <cstdint>` was not needed even though types from this
header were used. This was broken by a change in libstdc++ version 13:
https://gcc.gnu.org/gcc-13/porting_to.html#header-dep-changes

The necessary fix was implemented in boost.beast, however it is not yet
available. Until it is available, we can use this workaround to enable
compilation of `rippled` with gcc-13, clang-16, etc.
2023-11-30 21:46:06 -08:00
Sophia Xie
5aef102f4f Revert #4505, #4760 (#4842)
* Revert "Optimize calculation of close time to avoid impasse and minimize gratuitous proposal changes (#4760)"

This reverts commit 8ce85a9750.

* Revert "Several changes to improve Consensus stability: (#4505)"

This reverts commit f259cc1ab6.

* Add missing include

---------

Co-authored-by: seelabs <scott.determan@yahoo.com>
2023-11-30 21:00:50 -08:00
Elliot Lee
431646437e Set version to 2.0.0-rc4 2023-11-29 12:20:43 -08:00
Bronek Kozicki
fe8621b00f APIv2: show DeliverMax in submit, submit_multisigned (#4827)
Show `DeliverMax` instead of `Amount` in output from `submit`,
`submit_multisigned`, `sign`, and `sign_for`.

Fix #4829
2023-11-29 12:19:32 -08:00
Bronek Kozicki
c045060560 APIv2: consistently return ledger_index as integer (#4820)
For api_version 2, always return ledger_index as integer in JSON output.

api_version 1 retains prior behavior.
2023-11-29 12:12:38 -08:00
Manoj Doshi
8ec475b1fd Set version to 2.0.0-rc3 2023-11-27 15:24:25 -08:00
Elliot Lee
e33a6d5c2b docs(API-CHANGELOG): add extra bullet about DeliverMax (#4784) 2023-11-27 13:29:47 -08:00
Bronek Kozicki
923e1cef99 Update API-CHANGELOG.md for release 2.0 (#4828) 2023-11-27 13:28:34 -08:00
Michael Legleux
2e93dd57eb Add Debian 12 Bookworm; ignore core-utils in almalinux (#4836)
Add codename `bookworm` to the distro matrix during Artifactory uploads
allowing Debian 12 clients to install `rippled` packages.

Ignore installing conflicting `core-utils` packages during almalinux
package smoke tests.
2023-11-27 13:26:41 -08:00
Elliot Lee
92957d685d Merge pull request #4823 from XRPLF/release
Merging release into develop
2023-11-27 13:16:34 -08:00
Elliot Lee
f05acbd782 Merge pull request #4824 from manojsdoshi/develop
Merging rc2 to develop
2023-11-27 13:12:18 -08:00
Manoj Doshi
44a9266c9e Set version to 2.0.0-rc2 2023-11-20 16:30:17 -08:00
Mark Travis
c1710900b4 Optimize calculation of close time to avoid impasse and minimize gratuitous proposal changes (#4760)
* Optimize the calculation of close time to avoid
impasse and minimize gratuitous proposal changes.

* git apply clang-format.patch

* Review (Howard) fixes.

* Review fix for impasse discovered by John.

* Review fixes (comments) from John.

* Scott S review fixes. Also clang-format.
2023-11-20 16:29:11 -08:00
Bronek Kozicki
d5059b11b9 Fix 2.0 regression in tx method with binary output (#4812)
* Fix binary output from tx method

* Formatting fix

* Minor test improvement

* Minor test improvements
2023-11-20 16:28:52 -08:00
Michael Legleux
f95fa338c9 Update Linux smoketest distros (#4813)
Co-authored-by: manoj <mdoshi@ripple.com>
2023-11-20 16:28:28 -08:00
Bronek Kozicki
96c926c71e Promote API version 2 to supported (#4803)
* Promote API version 2 to supported

* Switch command line to API version 1

* Fix LedgerRequestRPC test

* Remove obsolete tx_account method

This method is not implemented, the only parts which are removed are related to command-line parsing

* Fix RPCCall test

* Reduce diff size, small test improvements

* Minor fixes

* Support for the mold linker

* [fold] handle case where both mold and gold are installed

* [fold] Use first non-default linker

* Fix TransactionEntry_test

* Fix AccountTx_test

---------

Co-authored-by: seelabs <scott.determan@yahoo.com>
2023-11-20 16:28:17 -08:00
Scott Determan
4dff203787 Support for the mold linker (#4807) 2023-11-20 16:28:00 -08:00
manoj
4977a5d43c Proposed 2.0.0-rc2 (#4818)
* Support for the mold linker (#4807)

* Promote API version 2 to supported (#4803)

* Promote API version 2 to be supported

* Switch the command line to API version 1

* Fix LedgerRequestRPC test

* Remove obsolete tx_account method

This method is not implemented, the only parts which are removed are related to command-line parsing

* Fix RPCCall test

* Reduce diff size, small test improvements

* Minor fixes

* Support for the mold linker

* Fix TransactionEntry_test

* Fix AccountTx_test

---------

Co-authored-by: seelabs <scott.determan@yahoo.com>

* Update Linux smoketest distros (#4813)

* Fix 2.0 regression in tx method with binary output (#4812)

* Fix binary output from tx method

* Formatting fix

* Minor test improvement

* Minor test improvements

* Optimize calculation of close time to avoid impasse and minimize gratuitous proposal changes (#4760)

* Optimize the calculation of close time to avoid
impasse and minimize gratuitous proposal changes.

* git apply clang-format.patch

* Scott S review fixes. Also clang-format.

* Set version to 2.0.0-rc2

---------

Co-authored-by: manoj <mdoshi@ripple.com>
Co-authored-by: Scott Determan <scott.determan@yahoo.com>
Co-authored-by: Bronek Kozicki <brok@incorrekt.com>
Co-authored-by: Michael Legleux <legleux@users.noreply.github.com>
Co-authored-by: Mark Travis <mtrippled@users.noreply.github.com>
2023-11-20 13:34:59 -08:00
Mark Travis
8ce85a9750 Optimize calculation of close time to avoid impasse and minimize gratuitous proposal changes (#4760)
* Optimize the calculation of close time to avoid
impasse and minimize gratuitous proposal changes.

* git apply clang-format.patch

* Review (Howard) fixes.

* Review fix for impasse discovered by John.

* Review fixes (comments) from John.

* Scott S review fixes. Also clang-format.
2023-11-16 20:37:53 -08:00
Bronek Kozicki
d5939727e0 Fix 2.0 regression in tx method with binary output (#4812)
* Fix binary output from tx method

* Formatting fix

* Minor test improvement

* Minor test improvements
2023-11-15 16:40:10 -08:00
Michael Legleux
7b49f1e1de Update Linux smoketest distros (#4813)
Co-authored-by: manoj <mdoshi@ripple.com>
2023-11-14 16:56:48 -08:00
Bronek Kozicki
ac27089c69 Promote API version 2 to supported (#4803)
* Promote API version 2 to supported

* Switch command line to API version 1

* Fix LedgerRequestRPC test

* Remove obsolete tx_account method

This method is not implemented, the only parts which are removed are related to command-line parsing

* Fix RPCCall test

* Reduce diff size, small test improvements

* Minor fixes

* Support for the mold linker

* [fold] handle case where both mold and gold are installed

* [fold] Use first non-default linker

* Fix TransactionEntry_test

* Fix AccountTx_test

---------

Co-authored-by: seelabs <scott.determan@yahoo.com>
2023-11-13 15:04:27 -08:00
Scott Determan
4cb0bcb003 Support for the mold linker (#4807) 2023-11-13 12:28:25 -08:00
Manoj Doshi
cf4e9e5578 Set version to 2.0.0-rc1 2023-11-09 11:28:45 -08:00
Bronek Kozicki
32ced493de Unify JSON serialization format of transactions (#4775)
* Remove include <ranges>

* Formatting fix

* Output for subscriptions

* Output from sign, submit etc.

* Output from ledger

* Output from account_tx

* Output from transaction_entry

* Output from tx

* Store close_time_iso in API v2 output

* Add small APIv2 unit test for subscribe

* Add unit test for transaction_entry

* Add unit test for tx

* Remove inLedger from API version 2

* Set ledger_hash and ledger_index

* Move isValidated from RPCHelpers to LedgerMaster

* Store closeTime in LedgerFill

* Time formatting fix

* additional tests for Subscribe unit tests

* Improved comments

* Rename mInLedger to mLedgerIndex

* Minor fixes

* Set ledger_hash on closed ledger, even if not validated

* Update API-CHANGELOG.md

* Add ledger_hash, ledger_index to transaction_entry

* Fix validated and close_time_iso in account_tx

* Fix typos

* Improve getJson for Transaction and STTx

* Minor improvements

* Replace class enum JsonOptions with struct

We may consider turning this into a general-purpose template and using it elsewhere

* simplify the extraction of transactionID from Transaction object

* Remove obsolete comments

* Unconditionally set validated in account_tx output

* Minor improvements

* Minor fixes

---------

Co-authored-by: Chenna Keshava <ckeshavabs@gmail.com>
2023-11-08 10:36:24 -08:00
Scott Determan
09e0f103f4 fix: check for valid public key in attestations (#4798) 2023-11-02 15:19:21 -07:00
pwang200
056255e396 Fix unit test api_version to enable api_version 2 (#4785)
The command line API still uses `apiMaximumSupportedVersion`.
The unit test RPCs use `apiMinimumSupportedVersion` if unspecified.

Context:
- #4568
- #4552
2023-11-02 09:59:19 -07:00
Elliot Lee
85342b21c8 chore: delete unused Dockerfile (#4791) 2023-10-31 23:00:11 -07:00
Stefan van Kessel
26b0322aa5 fix: remove unused variable (#4677)
With clang 15, an unused-but-set-variable warning was emitted:

PostgresDatabase.cpp:178:14: warning: variable 'expNumResults' set but
not used [-Wunused-but-set-variable]
    uint32_t expNumResults = 1;
2023-10-30 20:35:28 -07:00
Gregory Tsipenyuk
3b624d8bf8 fixFillOrKill: fix offer crossing with tfFillOrKill (#4694)
Introduce the `fixFillOrKill` amendment.

Fix an edge case occurring when an offer with `tfFillOrKill` set (but
without `tfSell` set) fails to cross an offer with a better rate. If
`tfFillOrKill` is set, then the owner must receive the full TakerPays.
Without this amendment, an offer fails if the entire `TakerGets` is not
spent. With this amendment, when `tfSell` is not set, the entire
`TakerGets` does not have to be spent.

For details about OfferCreate, see: https://xrpl.org/offercreate.html

Fix #4684

---------

Co-authored-by: Scott Schurr <scott@ripple.com>
2023-10-30 14:05:46 -07:00
Bronek Kozicki
ac02e56519 fix: remove include <ranges> (#4788)
Remove dependency on `<ranges>` header, since it is not implemented by
all compilers which we want to support.

This code change only affects unit tests.

Resolve https://github.com/XRPLF/rippled/issues/4787
2023-10-30 11:13:25 -07:00
Bronek Kozicki
1eac4d2c07 APIv2: remove tx_history and ledger_header (#4759)
Remove `tx_history` and `ledger_header` methods from API version 2.

Update `RPC::Handler` to allow for methods (or method implementations)
to be API version specific. This partially resolves #4727. We can now
store multiple handlers with the same name, as long as they belong to
different (non-overlapping) API versions. This necessarily impacts the
handler lookup algorithm and its complexity; however, there is no
performance loss on x86_64 architecture, and only minimal performance
loss on arm64 (around 10ns). This design change gives us extra
flexibility evolving the API in the future, including other parts of
#4727.

In API version 2, `tx_history` and `ledger_header` are no longer
recognised; if they are called, `rippled` will return error
`unknownCmd`

Resolve #3638

Resolve #3539
2023-10-24 15:57:49 -07:00
Mark Travis
3e5f770a38 docs: clarify definition of network health (#4729)
Update the documentation to describe network health with more nuance as
well as context about related factors.
2023-10-24 14:23:44 -07:00
Elliot Lee
2a66bb3fc7 Set version to 2.0.0-b4 2023-10-23 11:51:45 -07:00
Bronek Kozicki
397268394b APIv2(DeliverMax): add alias for Amount in Payment transactions (#4733)
Using the "Amount" field in Payment transactions can cause incorrect
interpretation. There continue to be problems from the use of this
field. "Amount" is rarely the correct field to use; instead,
"delivered_amount" (or "DeliveredAmount") should be used.

Rename the "Amount" field to "DeliverMax", a less misleading name. With
api_version: 2, remove the "Amount" field from Payment transactions.

- Input: "DeliverMax" in `tx_json` is an alias for "Amount"
  - sign
  - submit (in sign-and-submit mode)
  - submit_multisigned
  - sign_for
- Output: Add "DeliverMax" where transactions are provided by the API
  - ledger
  - tx
  - tx_history
  - account_tx
  - transaction_entry
  - subscribe (transactions stream)
- Output: Remove "Amount" from API version 2

Fix #3484

Fix #3902
2023-10-23 11:26:16 -07:00
Ed Hennis
5026cbdaf3 perf(CI): use unity builds to speed up Windows CI (#4780)
The unity build speeds up compilation by bundling multiple source files
into one larger file. This reduces Windows CI build time by up to 50%.

As described in #4596, the automatic Windows builds take a very long
time. Unity builds are significantly faster - currently about 45 min,
much closer to the typical MacOS (35-40 minutes) and nix (~30 minutes)
run times.

This is intended as a stopgap solution until a more resourced and
reliable runner is available.

No C++ code was changed. This only affects CI.
2023-10-23 08:20:48 -07:00
shichengsg002
5af9dc5abd fix(CI): set permission for doxygen workflow (#4756) 2023-10-19 10:53:38 -07:00
Denis Angell
8d86c5e17d fix: allow pseudo-transactions to omit NetworkID (#4737)
The Network ID logic should not be applied to pseudo-transactions.

This allows amendments to enable on a network with an ID > 1024.

Context:
- NetworkID: https://github.com/XRPLF/rippled/pull/4370
- Pseudo-transactions: https://xrpl.org/pseudo-transaction-types.html

Fix #4736

---------

Co-authored-by: RichardAH <richard.holland@starstone.co.nz>
2023-10-19 10:00:49 -07:00
Mayukha Vadari
078bd606c7 feat(rpc): add server_definitions method (#4703)
Add a new RPC / WS call for `server_definitions`, which returns an
SDK-compatible `definitions.json` (binary enum definitions) generated by
the server. This enables clients/libraries to dynamically work with new
fields and features, such as ones that may become available on side
chains. Clients query `server_definitions` on a node from the network
they want to work with, and immediately know how to speak that node's
binary "language", even if new features are added to it in the future
(as long as there are no new serialized types that the software doesn't
know how to serialize/deserialize).

Example:

```js
> {"command": "server_definitions"}
< {
    "result": {
        "FIELDS": [
            [
                "Generic",
                {
                    "isSerialized": false,
                    "isSigningField": false,
                    "isVLEncoded": false,
                    "nth": 0,
                    "type": "Unknown"
                }
            ],
            [
                "Invalid",
                {
                    "isSerialized": false,
                    "isSigningField": false,
                    "isVLEncoded": false,
                    "nth": -1,
                    "type": "Unknown"
                }
            ],
            [
                "ObjectEndMarker",
                {
                    "isSerialized": false,
                    "isSigningField": true,
                    "isVLEncoded": false,
                    "nth": 1,
                    "type": "STObject"
                }
            ],
        ...
```

Close #3657

---------

Co-authored-by: Richard Holland <richard.holland@starstone.co.nz>
2023-10-18 16:16:40 -07:00
Mayukha Vadari
b421945e71 DID: Decentralized identifiers (DIDs) (XLS-40): (#4636)
Implement native support for W3C DIDs.

Add a new ledger object: `DID`.

Add two new transactions:
1. `DIDSet`: create or update the `DID` object.
2. `DIDDelete`: delete the `DID` object.

This meets the requirements specified in the DID v1.0 specification
currently recommended by the W3C Credentials Community Group.

The DID format for the XRP Ledger conforms to W3C DID standards.
The objects can be created and owned by any XRPL account holder.
The transactions can be integrated by any service, wallet, or application.
2023-10-18 13:01:12 -07:00
Denis Angell
41cd337506 Update the reserved hook error code name to tecHOOK_REJECTED (#4559)
The old name was `tecHOOK_ERROR`
2023-10-17 22:56:22 -07:00
Scott Schurr
b69156ac01 refactor(peerfinder): use LogicError in PeerFinder::Logic (#4562)
It might be possible for the server code to indirect through certain
`end()` iterators. While a debug build would catch this problem with
`assert()`s, a release build would crash. If there are problems in this
area in the future, it is best to get a definitive indication of the
nature of the error regardless of whether it's a debug or release build.
To accomplish this, these `assert`s are converted into `LogicError`s
that will produce a reasonable error message when they fire.
2023-10-17 22:52:33 -07:00
Elliot Lee
be6ac7e7a1 docs(pull_request_template): add API Impact section (#4757) 2023-10-17 21:27:03 -07:00
Elliot Lee
1fc1eb9f68 Set version to 2.0.0-b3 2023-10-16 17:59:19 -07:00
Ed Hennis
1fde585003 fix(CI): Call python to upgrade pip on Windows (#4768)
In Windows, we need to call `python` in order for the `pip` upgrade
command to work.

This changes the GitHub Actions Windows CI job to use the correct
command to upgrade PIP, fixing this error:

```
ERROR: To modify pip, please run the following command:
C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe -m pip install --upgrade pip
```

A future task is to make job run on heavy Windows runners so that it
doesn't take so long.

Context: #4596
2023-10-16 14:26:40 -07:00
Jackson Mills
c915984340 docs(conan.md): fix broken link to conanfile.py (#4740) 2023-10-12 10:12:17 -07:00
Ed Hennis
50cc1cf0c9 fix(PathRequest): remove incorrect assert (#4743)
The assert is saying that the only reason `pathFinder` would be null is
if the request was aborted (connection dropped, etc.). That's what
`continueCallback()` checks. But that is very clearly not true if you
look at `getPathFinder`, which calls `findPaths`, which can return false
for many reasons.

Fix #4744
2023-10-12 10:10:35 -07:00
Chenna Keshava B S
1151fba415 fix(CI): update workflow for uploading binaries to artifactory (#4746)
Update the nix CI runner. This commit does not modify any source code
files. The unix builds were successful, but the binaries were not
uploaded to the internal artifactory. This PR borrows an idea from
@ximinez to attempt to fix this issue.

After successful authentication, the `outcome` variable contains a
string. In the upload step, we are checking if outcome == 'success' as a
prerequisite for uploading the binary.

This commit updates the contents of the `outcome` variable.
2023-10-11 08:45:45 -07:00
Ed Hennis
3e08c390f5 ci: reenable Windows CI build with Artifactory support (#4596)
Artifactory support was added to the `nix` builds with #4556. This
extends that support to the Windows build. Now the Windows build works;
CI will build and test a Windows release build. This only affects CI and
does not change any C++ code.

* Copy the remote setup step outcome fix from #4716 discussion
* Allow the Windows job to succeed if tests fail:
  * Currently the tests do not always pass, even on a single threaded
    run on the GitHub runners. So we are using parallel runs and mark
    the test step as allowed to fail (continue-on-error).
  * At this point, it's more important that the build succeeds than that
    the tests succeed, because:
  * We've got plenty of test coverage on the other jobs.
  * Test failures are much rarer than build failures because of
    cross-platform issues.
  * Having a test failure locally doesn't interrupt a workflow nearly as
    much as a build failure.

Note that Conan Center cannot hold the binaries we need. They do not
build the configurations we need, and they will not add them.

## Future Tasks

This introduces a new bottleneck since the build and test takes over an
hour. Speed up the job by:

* Making this job run on heavy Windows runners.
* Increasing the number of hardware threads.
2023-10-09 15:51:22 -07:00
Ed Hennis
053b69c63f docs(API-CHANGELOG): add XRPFees change (#4741)
* Add a new API Changelog section for release 1.10.
* Mark `jss::fee_ref` as deprecated.
* Fix a copy-paste error in one of the unit tests.
2023-10-06 11:11:39 -07:00
Florent
ced14ec1ab docs(rippled-example.cfg): add P2P link compression (#4753)
P2P link compression is a feature added in 1.6.0 by #3287.

https://xrpl.org/enable-link-compression.html

If the default changes in the future - for example, as currently
proposed by #4387 - the comment will be updated at that time.

Fix #4656
2023-10-06 08:23:30 -07:00
Denis Angell
6ba9450c89 fixDisallowIncomingV1: allow issuers to authorize trust lines (#4721)
Context: The `DisallowIncoming` amendment provides an option to block
incoming trust lines from reaching your account. The
asfDisallowIncomingTrustline AccountSet Flag, when enabled, prevents any
incoming trust line from being created. However, it was too restrictive:
it would block an issuer from authorizing a trust line, even if the
trust line already exists. Consider:

1. Issuer sets asfRequireAuth on their account.
2. User sets asfDisallowIncomingTrustline on their account.
3. User submits tx to SetTrust to Issuer.

At this point, without `fixDisallowIncomingV1` active, the issuer would
not be able to authorize the trust line.

The `fixDisallowIncomingV1` amendment, once activated, allows an issuer
to authorize a trust line even after the user sets the
asfDisallowIncomingTrustline flag, as long as the trust line already
exists.
2023-10-05 16:25:16 -07:00
Scott Determan
ec8626046b refactor: reduce boilerplate in applySteps: (#4710)
When a new transactor is added, there are several places in applySteps
that need to be modified. This patch refactors the code so only one
function needs to be modified.
2023-10-05 15:29:23 -07:00
Rome Reginelli
4e84ad6cd7 refactor: reunify transaction common fields: (#4715)
Make transactions and pseudo-transactions share the same commonFields
again. This regularizes the code in a nice way.

While this technically allows pseudo-transactions to have a
TicketSequence field, pseudo-transactions are only ever constructed by
code paths that don't add such a field, so this is not a transaction
processing change. It may be possible to add a separate check to ensure
TicketSequence (and other fields that don't make sense on
pseudo-transactions) are never added to pseudo-transactions, but that
should not be necessary. (TicketSequence is not the only common field
that can not and does not appear in pseudo-transactions.) Note:
TicketSequence is already documented as a common field.

Related: #4637

Fix #4714
2023-10-04 17:03:14 -07:00
sokkaofthewatertribe
40ebbecac8 fix typo in SECURITY.md (#4662) 2023-10-03 21:32:16 -07:00
Elliot Lee
0c43eb3973 docs(API-CHANGELOG): clarify account_info response (#4724) 2023-10-03 21:31:24 -07:00
Stefan van Kessel
3dea78d34b fix: asan stack-use-after-scope in soci::use with rvalues (#4676)
Address a stack-use-after-scope issue when using rvalues with
`soci::use`. Replace rvalues with lvalues to ensure the scope extends
beyond the end of the expression.

The issue arises from `soci` taking a reference to the rvalue without
copying its value or extending its lifetime. `soci` references rvalues
in `soci::use_container` and then the address in `soci_use_type`. For
types like `int`, memory access post-lifetime is unlikely to cause
issues. However, for `std::string`, the backing heap memory can be freed
and potentially reused, leading to a potential segmentation fault.

This was detected on x86_64 using clang-15 with asan. asan confirms
resolution of the issue.

Fix #4675
2023-10-03 21:21:36 -07:00
Chenna Keshava B S
e27d24ba00 docs(BUILD.md): require GCC 11 or higher (#4700)
Update minimum compiler requirement for building the codebase. The
feature "using enum" is required. This feature was introduced in C++20.

Updating the C++ compiler to version 11 or later fixes this error:

```
Building CXX object CMakeFiles/xrpl_core.dir/src/ripple/protocol/impl/STAmount.cpp.o
/build/ripple/binary/src/ripple/protocol/impl/STAmount.cpp: In lambda function:
/build/ripple/binary/src/ripple/protocol/impl/STAmount.cpp:1577:15: error: expected nested-name-specifier before 'enum'
 1577 |         using enum Number::rounding_mode;
      |               ^~~~
```

Fix #4693
2023-10-02 16:43:29 -07:00
Scott Determan
925aca764b fix(XLS-38): disallow the same bridge on one chain: (#4720)
Modify the `XChainBridge` amendment.

Before this patch, two door accounts on the same chain could could own
the same bridge spec (of course, one would have to be the issuer and one
would have to be the locker). While this is silly, it does not violate
any bridge invariants. However, on further review, if we allow this then
the `claim` transactions would need to change. Since it's hard to see a
use case for two doors to own the same bridge, this patch disallows
it. (The transaction will return tecDUPLICATE).
2023-10-02 07:49:33 -07:00
Scott Schurr
2bb8de030f fix: stabilize voting threshold for amendment majority mechanism (#4410)
Amendment "flapping" (an amendment repeatedly gaining and losing
majority) usually occurs when an amendment is on the verge of gaining
majority, and a validator not in favor of the amendment goes offline or
loses sync. This fix makes two changes:

1. The number of validators in the UNL determines the threshold required
   for an amendment to gain majority.
2. The AmendmentTable keeps a record of the most recent Amendment vote
   received from each trusted validator (and, with `trustChanged`, stays
   up-to-date when the set of trusted validators changes). If no
   validation arrives from a given validator, then the AmendmentTable
   assumes that the previously-received vote has not changed.

In other words, when missing an `STValidation` from a remote validator,
each server now uses the last vote seen. There is a 24 hour timeout for
recorded validator votes.

These changes do not require an amendment because they do not impact
transaction processing, but only the threshold at which each individual
validator decides to propose an EnableAmendment pseudo-transaction.

Fix #4350
2023-09-28 11:47:48 -07:00
Ed Hennis
b92d511558 fix(build): uint is not defined on Windows platform (#4731)
Fix the Windows build by using `unsigned int` (instead of `uint`).

The error, introduced by #4618, looks something like:
  rpc\impl\RPCHelpers.h(299,5): error C2061: syntax error: identifier
  'uint' (compiling source file app\ledger\Ledger.cpp)
2023-09-27 13:12:04 -07:00
Nik Bougalis
548c91ebb6 Eliminate the built-in SNTP support (fixes #4207): (#4628) 2023-09-26 17:35:31 -07:00
Elliot Lee
2c56d9fc3e Set version to 2.0.0-b2 2023-09-22 16:17:35 -07:00
John Freeman
6b61505ac2 fix: accept all valid currency codes in API (#4566)
A few methods, including `book_offers`, take currency codes as
parameters. The XRPL doesn't care if the letters in those codes are
lowercase or uppercase, as long as they come from an alphabet defined
internally. rippled doesn't care either, when they are submitted in a
hex representation. When they are submitted in an ASCII string
representation, rippled, but not XRPL, is more restrictive, preventing
clients from interacting with some currencies already in the XRPL.

This change gets rippled out of the way and lets clients submit currency
codes in ASCII using the full alphabet.

Fixes #4112
2023-09-22 16:07:43 -07:00
Bronek Kozicki
e4db0fba2b chore: add .build to .gitignore (#4722)
Currently, the `BUILD.md` instructions suggest using `.build` as the
build directory, so this change helps to reduce confusion.

An alternative would be to instruct developers to add `/.build/` to
`.git/info/exclude` or to user-level `.gitignore` (although the latter
is very intrusive). However, it is being added here because it is a good
practice to have a sensible default that's consistent with the build
instructions.
2023-09-22 16:01:00 -07:00
Gregory Tsipenyuk
8f89694fae Add ProtocolStart and GracefulClose P2P protocol messages (#3839)
Clean up the peer-to-peer protocol start/close sequences by introducing
START_PROTOCOL and GRACEFUL_CLOSE messages, which sync inbound/outbound
peer send/receive. The GRACEFUL_CLOSE message differentiates application
and link layer failures.

* Introduce the `InboundHandoff` class to manage inbound peer
  instantiation and synchronize the send/receive protocol messages
  between peers.
* Update `OverlayImpl` to utilize the `InboundHandoff` class to manage
  inbound handshakes.
* Update `PeerImp` for improved handling of protocol messages.
* Modify the `Message` class for better maintainability.
* Introduce P2P protocol version `2.3`.
2023-09-22 15:56:44 -07:00
ForwardSlashBack
5433e133d5 Fix typo in BUILD.md (#4718)
Co-authored-by: Chenna Keshava B S <21219765+ckeshava@users.noreply.github.com>
2023-09-21 15:16:22 -07:00
Peter Chen
2487dab194 APIv2(gateway_balances, channel_authorize): update errors (#4618)
gateway_balances
* When `account` does not exist in the ledger, return `actNotFound`
  * (Previously, a normal response was returned)
  * Fix #4290
* When required field(s) are missing, return `invalidParams`
  * (Previously, `invalidHotWallet` was incorrectly returned)
  * Fix #4548

channel_authorize
* When the specified `key_type` is invalid, return `badKeyType`
  * (Previously, `invalidParams` was returned)
  * Fix #4289

Since these are breaking changes, they apply only to API version 2.

Supersedes #4577
2023-09-21 13:33:24 -07:00
John Freeman
77e0912a0e build: use Boost 1.82 and link Boost.Json (#4632)
Add Boost::json to the list of linked Boost libraries.

This seems to be required for macOS.
2023-09-21 08:58:39 -07:00
Chenna Keshava B S
a948203dae docs(overlay): add URL of blog post and clarify wording (#4635) 2023-09-18 22:00:59 -07:00
Elliot Lee
8f65bc24a0 docs(API-CHANGELOG): api_version 2 is expected with 2.0 (#4633)
* Also update for #4585
2023-09-18 21:59:47 -07:00
Elliot Lee
9f102fc99d docs(RELEASENOTES): update 1.12.0 notes to match dev blog (#4691)
* Reorganize some changelog entries
* Add note about portable binaries
* Dev blog: https://xrpl.org/blog
2023-09-18 21:57:25 -07:00
John Freeman
7bff9dc7f0 Update secp256k1 to 0.3.2 (#4653)
Copy the new code to `src/secp256k1` without changes:
`src/secp256k1` is identical to bitcoin-core/secp256k1@acf5c55 (v0.3.2).

We could consider changing to a Git submodule, though that would require
changes to the build instructions because we are not using submodules
anywhere else.
2023-09-18 13:26:14 -07:00
Chenna Keshava B S
e86181c096 docs: fix comment for LedgerHistory::fixIndex return value (#4574)
`LedgerHistory::fixIndex` returns `false` if a repair was performed.

Fix #4572
2023-09-18 10:01:43 -07:00
Ed Hennis
65df4bceaa fix: remove unused variable causing clang 14 build errors (#4672)
Removed the unused variable `none` from `Writer.cpp` which was causing
build errors on clang version 14.
2023-09-18 08:38:51 -07:00
Elliot Lee
3397922989 docs(BUILD): make it easier to find environment.md (#4507)
Make the instructions a bit easier to follow. Users on different
platforms can look for their platform name to find relevant information.
2023-09-15 21:54:25 -07:00
Elliot Lee
046d0c2f0a Set version to 2.0.0-b1 2023-09-15 08:55:09 -07:00
Michael Legleux
01fc4b0203 Revert CMake changes (#4707)
This was likely put back when #4292 was rebased.
2023-09-15 08:54:46 -07:00
Scott Determan
5b7d2c133a Change XChainBridge amendment to Supported::yes (#4709) 2023-09-15 08:54:39 -07:00
Scott Determan
237b406e8c Fix Windows build by removing two unused declarations (#4708)
Remove the `verify` and `message` function declarations. The explicit
instantiation requests could not be completed because there were no
implementations for those two member functions. It is helpful that the
Microsoft (MSVC) compiler on Windows appears to be strict when it comes
to template instantiation.

This resolves the warning:

  XChainAttestations.h(450): warning C4661: 'bool
  ripple::XChainAttestationsBase<ripple::XChainClaimAttestation>::verify(void)
  const': no suitable definition provided for explicit template
  instantiation request
2023-09-14 21:02:31 -07:00
Ed Hennis
5427321260 Match unit tests on start of test name (#4634)
* For example, without this change, to run the TxQ tests, must specify
  `--unittest=TxQ1,TxQ2` on the command line. With this change, can use
  `--unittest=TxQ`, and both will be run.
* An exact match will prevent any further partial matching.
* This could have some side effects for different tests with a common
  name beginning. For example, NFToken, NFTokenBurn, NFTokenDir. This
  might be useful. If not, the shorter-named test(s) can be renamed. For
  example, NFToken to NFTokens.
* Split the NFToken, NFTokenBurn, and Offer test classes. Potentially speeds
  up parallel tests by a factor of 5.
2023-09-14 13:19:15 -07:00
Howard Hinnant
ce570c166d Revert ThreadName due to problems on Windows (#4702)
* Revert "Remove CurrentThreadName.h from RippledCore.cmake (#4697)"

This reverts commit 3b5fcd5873.

* Revert "Introduce replacement for getting and setting thread name: (#4312)"

This reverts commit 36cb5f90e2.
2023-09-14 13:16:50 -07:00
Scott Determan
649c11a78e XChainBridge: Introduce sidechain support (XLS-38): (#4292)
A bridge connects two blockchains: a locking chain and an issuing
chain (also called a mainchain and a sidechain). Both are independent
ledgers, with their own validators and potentially their own custom
transactions. Importantly, there is a way to move assets from the
locking chain to the issuing chain and a way to return those assets from
the issuing chain back to the locking chain: the bridge. This key
operation is called a cross-chain transfer. A cross-chain transfer is
not a single transaction. It happens on two chains, requires multiple
transactions, and involves an additional server type called a "witness".

A bridge does not exchange assets between two ledgers. Instead, it locks
assets on one ledger (the "locking chain") and represents those assets
with wrapped assets on another chain (the "issuing chain"). A good model
to keep in mind is a box with an infinite supply of wrapped assets.
Putting an asset from the locking chain into the box will release a
wrapped asset onto the issuing chain. Putting a wrapped asset from the
issuing chain back into the box will release one of the existing locking
chain assets back onto the locking chain. There is no other way to get
assets into or out of the box. Note that there is no way for the box to
"run out of" wrapped assets - it has an infinite supply.

Co-authored-by: Gregory Popovitch <greg7mdp@gmail.com>
2023-09-14 13:08:41 -07:00
Peter Chen
7fae1c1262 APIv2(account_tx, noripple_check): return error on invalid input (#4620)
For the `account_tx` and `noripple_check` methods, perform input
validation for optional parameters such as "binary", "forward",
"strict", "transactions". Previously, when these parameters had invalid
values (e.g. not a bool), no error would be returned. Now, it returns an
`invalidParams` error.

* This updates the behavior to match Clio
  (https://github.com/XRPLF/clio).
* Since this is potentially a breaking change, it only applies to
  requests specifying api_version: 2.
* Fix #4543.
2023-09-13 15:01:37 -07:00
Mark Travis
f259cc1ab6 Several changes to improve Consensus stability: (#4505)
* Verify accepted ledger becomes validated, and retry
   with a new consensus transaction set if not.
 * Always store proposals.
 * Track proposals by ledger sequence. This helps slow peers catch
   up with the rest of the network.
 * Acquire transaction sets for proposals with future ledger sequences.
   This also helps slow peers catch up.
 * Optimize timer delay for establish phase to wait based on how
   long validators have been sending proposals. This also helps slow
   peers to catch up.
 * Fix impasse achieving close time consensus.
 * Don't wait between open and establish phases.
2023-09-11 15:48:32 -07:00
Mark Travis
002893f280 Apply transaction batches in periodic intervals (#4504)
Add new transaction submission API field, "sync", which
determines behavior of the server while submitting transactions:
1) sync (default): Process transactions in a batch immediately,
   and return only once the transaction has been processed.
2) async: Put transaction into the batch for the next processing
   interval and return immediately.
3) wait: Put transaction into the batch for the next processing
   interval and return only after it is processed.
2023-09-11 15:47:40 -07:00
Mark Travis
1d9db1bfdd Asynchronously write batches to NuDB. (#4503) 2023-09-11 15:46:17 -07:00
Howard Hinnant
3b5fcd5873 Remove CurrentThreadName.h from RippledCore.cmake (#4697)
(File was already removed from the source)
2023-09-11 15:41:54 -07:00
Elliot Lee
a95505739d docs: update SECURITY.md (#4338) 2023-09-08 13:51:23 -07:00
Mayukha Vadari
31c8281922 refactor: simplify TxFormats common fields logic (#4637)
Minor refactor to `TxFormats.cpp`:
- Rename `commonFields` to `pseudoCommonFields` (since it is the common fields
  that all pseudo-transactions need)
- Add a new static variable, `commonFields`, which represents all the common
  fields that non-pseudo transactions need (essentially everything that
  `pseudoCommonFields` contains, plus `sfTicketSequence`)

This makes it harder to accidentally leave out `sfTicketSequence` in a new
transaction.
2023-09-08 10:24:51 -07:00
Peter Chen
c6f6375015 APIv2(ledger_entry): return invalidParams for bad parameters (#4630)
- Verify "check", used to retrieve a Check object, is a string.
- Verify "nft_page", used to retrieve an NFT Page, is a string.
- Verify "index", used to retrieve any type of ledger object by its
  unique ID, is a string.
- Verify "directory", used to retrieve a DirectoryNode, is a string or
  an object.

This change only impacts api_version 2 since it is a breaking change.

https://xrpl.org/ledger_entry.html

Fix #4550
2023-09-07 22:47:04 -07:00
Mark Pevec
6f74a745db docs(rippled-example.cfg): clarify ssl_cert vs ssl_chain (#4667)
Clarify usage of ssl_cert vs ssl_chain
2023-09-07 15:19:05 -07:00
Howard Hinnant
36cb5f90e2 Introduce replacement for getting and setting thread name: (#4312)
* In namespace ripple, introduces get_name function that takes a
  std:🧵:native_handle_type and returns a std::string.
* In namespace ripple, introduces get_name function that takes a
  std::thread or std::jthread and returns a std::string.
* In namespace ripple::this_thread, introduces get_name function
  that takes no parameters and returns the name of the current
  thread as a std::string.
* In namespace ripple::this_thread, introduces set_name function
  that takes a std::string_view and sets the name of the current
  thread.
* Intended to replace the beast utilities setCurrentThreadName
  and getCurrentThreadName.
2023-09-07 11:44:36 -07:00
Manoj Doshi
89780c8e4f Set version to 1.12.0 2023-09-06 13:58:34 -07:00
Elliot Lee
9d4d8c22d9 Set version to 1.12.0-rc4 2023-09-01 15:53:10 -07:00
Gregory Tsipenyuk
b014b79d88 amm_info: fetch by amm account id; add AMM object entry (#4682)
- Update amm_info to fetch AMM by amm account id.
  - This is an additional way to retrieve an AMM object.
  - Alternatively, AMM can still be fetched by the asset pair as well.
- Add owner directory entry for AMM object.

Context:

- Add back the AMM object directory entry, which was deleted by #4626.
  - This fixes `account_objects` for `amm` type.
2023-09-01 15:50:58 -07:00
Rome Reginelli
a61a88ea81 AMMBid: use tecINTERNAL for 'impossible' errors (#4674)
Modify two error cases in AMMBid transactor to return `tecINTERNAL` to
more clearly indicate that these errors should not be possible unless
operating in unforeseen circumstances. It likely indicates a bug.

The log level has been updated to `fatal()` since it indicates a
(potentially network-wide) unexpected condition when either of these
errors occurs.

Details:

The two specific transaction error cases changed are:

- `tecAMM_BALANCE` - In this case, this error (total LP Tokens
  outstanding is lower than the amount to be burned for the bid) is a
  subset of the case where the user doesn't have enough LP Tokens to pay
  for the bid. When this case is reached, the bidder's LP Tokens balance
  has already been checked first. The user's LP Tokens should always be
  a subset of total LP Tokens issued, so this should be impossible.
- `tecINSUFFICIENT_PAYMENT` - In this case, the amount to be refunded as
  a result of the bid is greater than the price paid for the auction
  slot. This should never occur unless something is wrong with the math
  for calculating the refund amount.

Both error cases in question are "defense in depth" measures meant to
protect against making things worse if the code has already reached a
state that is supposed to be impossible, likely due to a bug elsewhere.

Such "shouldn't ever occur" checks should use an error code that
categorically indicates a larger problem. This is similar to how
`tecINVARIANT_FAILED` is a warning sign that something went wrong and
likely could've been worse, but since there isn't an Invariant Check
applying here, `tecINTERNAL` is the appropriate error code.

This is "debatably" a transaction processing change since it could
hypothetically change how transactions are processed if there's a bug we
don't know about.
2023-09-01 13:44:48 -07:00
Elliot Lee
f31c50d04d Set version to 1.12.0-rc3 2023-08-30 15:47:07 -07:00
Elliot Lee
8ceb0f3a79 Revert "Asynchronously write batches to NuDB (#4503)"
This reverts commit 21c4aaf993.
2023-08-30 15:46:24 -07:00
Elliot Lee
d943c58b3d Revert "Apply transaction batches in periodic intervals (#4504)"
This reverts commit b580049ec0.
2023-08-30 15:46:12 -07:00
Elliot Lee
7ca1c644d1 Revert "Several changes to improve Consensus stability: (#4505)"
This reverts commit e8a7b2a1fc.
2023-08-29 16:52:31 -07:00
Manoj Doshi
300b7e078a Set version to 1.12.0-rc1 2023-08-21 16:23:06 -07:00
manoj
3574956e5e Set version to 1.12-b3
Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
2023-08-21 16:15:38 -07:00
Mark Travis
e8a7b2a1fc Several changes to improve Consensus stability: (#4505)
* Verify accepted ledger becomes validated, and retry
   with a new consensus transaction set if not.
 * Always store proposals.
 * Track proposals by ledger sequence. This helps slow peers catch
   up with the rest of the network.
 * Acquire transaction sets for proposals with future ledger sequences.
   This also helps slow peers catch up.
 * Optimize timer delay for establish phase to wait based on how
   long validators have been sending proposals. This also helps slow
   peers to catch up.
 * Fix impasse achieving close time consensus.
 * Don't wait between open and establish phases.
2023-08-21 16:15:31 -07:00
Mark Travis
b580049ec0 Apply transaction batches in periodic intervals (#4504)
Add new transaction submission API field, "sync", which
determines behavior of the server while submitting transactions:
1) sync (default): Process transactions in a batch immediately,
   and return only once the transaction has been processed.
2) async: Put transaction into the batch for the next processing
   interval and return immediately.
3) wait: Put transaction into the batch for the next processing
   interval and return only after it is processed.
2023-08-21 16:15:31 -07:00
Mark Travis
21c4aaf993 Asynchronously write batches to NuDB (#4503) 2023-08-21 16:15:31 -07:00
Ikko Eltociear Ashimine
213491d94b refactor: fix typo in FeeUnits.h (#4644)
covert -> convert
2023-08-21 16:15:31 -07:00
Michael Legleux
da203b241b Update Ubuntu build image (#4650) 2023-08-21 16:15:31 -07:00
Arihant Kothari
5d88b726c2 test: add forAllApiVersions helper function (#4611)
Introduce a new variadic template helper function, `forAllApiVersions`,
that accepts callables to execute a set of functions over a range of
versions - from RPC::apiMinimumSupportedVersion to RPC::apiBetaVersion.
This avoids the duplication of code.

Context: #4552
2023-08-21 16:15:31 -07:00
Mayukha Vadari
dcb7dd8d27 add view updates for account SLEs (#4629)
Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
2023-08-21 16:15:31 -07:00
John Freeman
91e9658217 Fix the package recipe for consumers of libxrpl (#4631)
- "Rename" the type `LedgerInfo` to `LedgerHeader` (but leave an alias
  for `LedgerInfo` to not yet disturb existing uses). Put it in its own
  public header, named after itself, so that it is more easily found.
- Move the type `Fees` and NFT serialization functions into public
  (installed) headers.
- Compile the XRPL and gRPC protocol buffers directly into `libxrpl` and
  install their headers. Fix the Conan recipe to correctly export these
  types.

Addresses change (2) in
https://github.com/XRPLF/XRPL-Standards/discussions/121.

For context: This work supports Clio's dependence on libxrpl. Clio is
just an example consumer. These changes should benefit all current and
future consumers.

---------

Co-authored-by: cyan317 <120398799+cindyyan317@users.noreply.github.com>
Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
2023-08-21 16:15:31 -07:00
Alphonse Noni Mousse
b28f62bbd9 refactor: improve checking of path lengths (#4519)
Improve the checking of the path lengths during Payments. Previously,
the code that did the check of the payment path lengths was sometimes
executed, but without any effect. This changes it to only check when it
matters, and to not make unnecessary copies of the path vectors.

Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
2023-08-21 16:15:30 -07:00
Alphonse N. Mousse
4f4951022c refactor: use C++20 function std::popcount (#4389)
- Replace custom popcnt16 implementation with std::popcount from C++20
- Maintain compatibility with older compilers and MacOS by providing a
  conditional compilation fallback to __builtin_popcount and a lookup
  table method
- Move and inline related functions within SHAMapInnerNode for
  performance and readability

Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
2023-08-21 16:15:30 -07:00
Gregory Tsipenyuk
58f7aecb0b fix(AMM): prevent orphaned objects, inconsistent ledger state: (#4626)
When an AMM account is deleted, the owner directory entries must be
deleted in order to ensure consistent ledger state.

* When deleting AMM account:
  * Clean up AMM owner dir, linking AMM account and AMM object
  * Delete trust lines to AMM
* Disallow `CheckCreate` to AMM accounts
  * AMM cannot cash a check
* Constrain entries in AuthAccounts array to be accounts
  * AuthAccounts is an array of objects for the AMMBid transaction
* SetTrust (TrustSet): Allow on AMM only for LP tokens
  * If the destination is an AMM account and the trust line doesn't
    exist, then:
    * If the asset is not the AMM LP token, then fail the tx with
      `tecNO_PERMISSION`
    * If the AMM is in empty state, then fail the tx with `tecAMM_EMPTY`
      * This disallows trustlines to AMM in empty state
* Add AMMID to AMM root account
  * Remove lsfAMM flag and use sfAMMID instead
* Remove owner dir entry for ltAMM
* Add `AMMDelete` transaction type to handle amortized deletion
  * Limit number of trust lines to delete on final withdraw + AMMDelete
  * Put AMM in empty state when LPTokens is 0 upon final withdraw
  * Add `tfTwoAssetIfEmpty` deposit option in AMM empty state
  * Fail all AMM transactions in AMM empty state except special deposit
  * Add `tecINCOMPLETE` to indicate that not all AMM trust lines are
    deleted (i.e. partial deletion)
    * This is handled in Transactor similar to deleted offers
  * Fail AMMDelete with `tecINTERNAL` if AMM root account is nullptr
  * Don't validate for invalid asset pair in AMMDelete
* AMMWithdraw deletes AMM trust lines and AMM account/object only if the
  number of trust lines is less than max
  * Current `maxDeletableAMMTrustLines` = 512
  * Check no directory left after AMM trust lines are deleted
  * Enable partial trustline deletion in AMMWithdraw
* Add `tecAMM_NOT_EMPTY` to fail any transaction that expects an AMM in
  empty state
* Clawback considerations
  * Disallow clawback out of AMM account
  * Disallow AMM create if issuer can claw back

This patch applies to the AMM implementation in #4294.

Acknowledgements:
Richard Holland and Nik Bougalis for responsibly disclosing this issue.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the project code and urge researchers to
responsibly disclose any issues they may find.

To report a bug, please send a detailed report to:

    bugs@xrpl.org

Signed-off-by: Manoj Doshi <mdoshi@ripple.com>
2023-08-21 16:15:08 -07:00
RichardAH
5530a0b665 feat: support Concise Transaction Identifier (CTID) (XLS-37) (#4418)
* add CTIM to tx rpc


---------

Co-authored-by: Rome Reginelli <mduo13@gmail.com>
Co-authored-by: Elliot Lee <github.public@intelliot.com>
Co-authored-by: Denis Angell <dangell@transia.co>
2023-08-17 18:43:47 -07:00
Elliot Lee
aded4a7a92 docs: add API Changelog (#4612)
Introduce an API Changelog, which logs the changes that have been made
to the API.

Without this changelog, it is difficult to keep track of the changes and
additions that are made to the API. While all changes are surfaced in
PRs and Release Notes, these are mixed in with other non-API-affecting
changes. PRs that affect the API have the `API Change` label applied,
but it is hard to identify which PRs have been included in each release.
Furthermore, some API changes will take effect based on `api_version`
(starting with rippled version 1.12, which will introduce `api_version:
2`), while others are based on the `rippled` version.

The API Changelog clarifies the details of the changes in a way that is
easily understood by API consumers, and breaks down the changes to be
clear which ones are gated by api_version (versus `rippled` version).

From now on, all PR authors are responsible for updating the API
Changelog according to the additions/changes that their PR makes to the
APIs.
2023-07-19 14:03:26 -07:00
Elliot Lee
01df19c08b Set version to 1.12.0-b2 2023-07-18 11:57:56 -07:00
Elliot Lee
c197c4cdd9 BUILD: list steps after dependencies update (#4623)
After dependencies update, developers may need to effectively "start
over" with the build steps.
2023-07-17 22:16:42 -07:00
Shawn Xie
5ba1f984df Rename allowClawback flag to allowTrustLineClawback (#4617)
Reason for this change is here XRPLF/XRPL-Standards#119

We would want to be explicit that this flag is exclusively for trustline. For new token types(eg. CFT), they will not utilize this flag for clawback, instead, they will turn clawback on/off on the token-level, which is more versatile.
2023-07-13 18:00:32 -07:00
John Freeman
cb09e61d2f Update dependencies (#4595)
Use the most recent versions in ConanCenter.

* Due to a bug in Clang 16, you may get a compile error:
  "call to 'async_teardown' is ambiguous"
  * A compiler flag workaround is documented in `BUILD.md`.
* At this time, building this with gcc 13 may require editing some files
  in `.conan/data`
  * A patch to support gcc13 may be added in a later PR.

---------

Co-authored-by: Scott Schurr <scott@ripple.com>
2023-07-13 13:25:08 -04:00
Gregory Tsipenyuk
3c9db4b69e Introduce AMM support (XLS-30d): (#4294)
Add AMM functionality:
- InstanceCreate
- Deposit
- Withdraw
- Governance
- Auctioning
- payment engine integration

To support this functionality, add:
- New RPC method, `amm_info`, to fetch pool and LPT balances
- AMM Root Account
- trust line for each IOU AMM token
- trust line to track Liquidity Provider Tokens (LPT)
- `ltAMM` object

The `ltAMM` object tracks:
- fee votes
- auction slot bids
- AMM tokens pair
- total outstanding tokens balance
- `AMMID` to AMM `RootAccountID` mapping

Add new classes to facilitate AMM integration into the payment engine.
`BookStep` uses these classes to infer if AMM liquidity can be consumed.

The AMM formula implementation uses the new Number class added in #4192.
IOUAmount and STAmount use Number arithmetic.

Add AMM unit tests for all features.

AMM requires the following amendments:
- featureAMM
- fixUniversalNumber
- featureFlowCross

Notes:
- Current trading fee threshold is 1%
- AMM currency is generated by: 0x03 + 152 bits of sha256{cur1, cur2}
- Current max AMM Offers is 30

---------

Co-authored-by: Howard Hinnant <howard.hinnant@gmail.com>
2023-07-12 13:52:50 -04:00
John Freeman
eeb8b41889 Adapt to change in Conan recipe for NuDB (#4615)
The recipe was updated a few days ago and the exported library target was renamed.
2023-07-11 16:26:15 -05:00
Elliot Lee
f7dd37e355 docs(CONTRIBUTING): push beta releases to release (#4589)
Sections that were rewrapped were wrapped to 72 characters, the same as
the recommendation for commit messages.
2023-07-07 14:31:09 -07:00
Arihant Kothari
a45a95e5ea APIv2(ledger_entry): return "invalidParams" when fields missing (#4552)
Improve error handling for ledger_entry by returning an "invalidParams"
error when one or more request fields are specified incorrectly, or one
or more required fields are missing.

For example, if none of of the following fields is provided, then the
API should return an invalidParams error:
* index, account_root, directory, offer, ripple_state, check, escrow,
  payment_channel, deposit_preauth, ticket

Prior to this commit, the API returned an "unknownOption" error instead.
Since the error was actually due to invalid parameters, rather than
unknown options, this error was misleading.

Since this is an API breaking change, the "invalidParams" error is only
returned for requests using api_version: 2 and above. To maintain
backward compatibility, the "unknownOption" error is still returned for
api_version: 1.

Related: #4573

Fix #4303
2023-07-06 11:58:53 -07:00
Chenna Keshava B S
c6fee28b92 refactor: change the return type of mulDiv to std::optional (#4243)
- Previously, mulDiv had `std::pair<bool, uint64_t>` as the output type.
  - This is an error-prone interface as it is easy to ignore when
    overflow occurs.
- Using a return type of `std::optional` should decrease the likelihood
  of ignoring overflow.
  - It also allows for the use of optional::value_or() as a way to
    explicitly recover from overflow.
- Include limits.h header file preprocessing directive in order to
  satisfy gcc's numeric_limits incomplete_type requirement.

Fix #3495

---------

Co-authored-by: John Freeman <jfreeman08@gmail.com>
2023-07-05 20:11:19 -07:00
Shawn Xie
77dc63b549 fix: add allowClawback flag for account_info (#4590)
* Update the `account_info` API so that the `allowClawback` flag is
  included in the response.
  * The proposed `Clawback` amendement added an `allowClawback` flag in
    the `AccountRoot` object.
  * In the API response, under `account_flags`, there is now an
    `allowClawback` field with a boolean (`true` or `false`) value.
  * For reference, the XLS-39 Clawback implementation can be found in
    #4553

Fix #4588
2023-07-05 08:46:23 -07:00
John Freeman
66bfe909e6 build: add binary hardening compile and link flags (#4603)
Enhance security during the build process:

* The '-fstack-protector' flag enables stack protection for preventing
  buffer overflow vulnerabilities. If an attempt is made to overflow the
  buffer, the program will terminate, thus protecting the integrity of
  the stack.
* The '-Wl,-z,relro,-z,now' linker flag enables Read-only Relocations
  (RELRO), a feature that helps harden the binary against certain types
  of exploits, particularly those that involve overwriting the Global
  Offset Table (GOT).
  * This flag is only set for Linux builds, due to compatibility issues
    with apple-clang.
  * The `relro` option makes certain sections of memory read-only after
    initialization to prevent them from being overwritten, while `now`
    ensures that all dynamic symbols are resolved immediately on program
    start, reducing the window of opportunity for attacks.
2023-07-03 07:41:12 -07:00
Mayukha Vadari
9c50415ebe add clang-format pre-commit hook (#4599)
* Add a new YAML file (.pre-commit-config.yaml) to set up pre-commit
  hook for clang-format
  * The pre-commit hook is opt-in and needs to be installed in order to
    run automatically
* Update CONTRIBUTING.md with instructions on how to set up and use the
  clang-format linter

Automating the process of running clang-format before committing code
helps to save time by removing the need to fix formatting issues later.

This is a tooling improvement and doesn't change C++ code.
2023-07-01 15:23:57 -07:00
Ed Hennis
516ffb2147 chore: update checkout action version to v3: (#4598)
* Update the version of the checkout action (for GitHub Actions) in
  `clang-format.yml` and `levelization.yml`.
  * The previous version, v2, was raising deprecation warnings due to
    its reliance on Node.js 12.
  * The latest checkout action version, v3, uses Node.js 16.
2023-07-01 00:13:37 -07:00
Peter Chen
f18c6dfea7 APIv2(account_info): handle invalid "signer_lists" value (#4585)
When requesting `account_info` with an invalid `signer_lists` value, the
API should return an "invalidParams" error.

`signer_lists` should have a value of type boolean. If it is not a
boolean, then it is invalid input. The response now indicates that.

* This is an API breaking change, so the change is only reflected for
  requests containing `"api_version": 2`
* Fix #4539
2023-06-29 23:05:21 -07:00
Michael Legleux
1cb67fbd20 fix: deb package build (#4591)
The debug packages were named with the extension ".ddeb", but due to a
bug in Artifactory, they need to have the ".deb" extension. Debug symbol
packages with ".ddeb" extensions are not indexed, and thus are not
visible in apt clients.

* Fix the issue by renaming the debug packages in the build script.
* Use GCC-11 and update GCC Conan profile.
  * This software requires GCC 11 and C++20. However, reporting mode is
    built with C++17.

This is a quick band-aid to fix the build. Later, it will be better to
remove this package-building code.

For context, a Debian (deb) package contains bundled software and
resources necessary for installing and managing software on a
Debian-based system, including Ubuntu and derivatives.
2023-06-29 20:15:11 -07:00
John Freeman
54afdaa101 ci: cancel overridden workflows (#4597)
Small quality-of-life improvements to workflows using new concurrency
control features:

https://docs.github.com/en/actions/using-jobs/using-concurrency

At time of this commit, macOS runners are oversubscribed. This may help.
2023-06-29 15:31:36 -07:00
Chenna Keshava B S
1c2ae10dc0 fix: Update Handler::Condition enum values #3417 (#4239)
- Use powers of two to clearly indicate the bitmask
- Replace bitmask with explicit if-conditions to better indicate predicates

Change enum values to be powers of two (fix #3417) #4239

Implement the simplified condition evaluation
removes the complex bitwise and(&) operator
Implement the second proposed solution in Nik Bougalis's comment - Software does not distinguish between different Conditions (Version: 1.5) #3417 (comment)
I have tested this code change by performing RPC calls with the commands server_info, server_state, peers and validation_info. These commands worked as expected.
2023-06-29 09:12:15 -07:00
Peter Chen
d34e8be316 APIv2: add error messages for account_tx (#4571)
Certain inputs for the AccountTx method should return an error. In other
words, an invalid request from a user or client now results in an error
message.

Since this can change the response from the API, it is an API breaking
change. This commit maintains backward compatibility by keeping the
existing behavior for existing requests. When clients specify
"api_version": 2, they will be able to get the updated error messages.

Update unit tests to check the error based on the API version.

* Fix #4288
* Fix #4545
2023-06-29 08:41:13 -07:00
Ed Hennis
4111382a31 Fix build references to deleted ServerHandlerImp: (#4592)
* Commits 0b812cd (#4427) and 11e914f (#4516) conflict. The first added
  references to `ServerHandlerImp` in files outside of that class's
  organizational unit (which is technically incorrect). The second
  removed `ServerHandlerImp`, but was not up to date with develop. This
  results in the build failing.
* Fixes the build by changing references to `ServerHandlerImp` to
  the more correct `ServerHandler`.
2023-06-28 13:23:12 -07:00
Scott Schurr
11e914fbe9 refactor: rename ServerHandlerImp to ServerHandler (#4516)
Rename `ServerHandlerImp` to `ServerHandler`. There was no other
ServerHandler definition despite the existence of a header suggesting
that there was.

This resolves a piece of historical confusion in the code, which was
identified during a code review.

The changes in the diff may look more extensive than they actually are.
The contents of `impl/ServerHandlerImp.h` were merged into
`ServerHandler.h`, making the latter file appear to have undergone
significant modifications. However, this is a non-breaking refactor that
only restructures code.
2023-06-27 17:57:52 -07:00
Chenna Keshava B S
0e983528e1 fix: remove deprecated fields in ledger method (#4244)
Remove deprecated fields from the ledger command:
* accepted
* hash (use ledger_hash instead)
* seqNum (use ledger_index instead)
* totalCoins (use total_coins instead)

Update SHAMapStore unit tests to use `jss:ledger_hash` instead of the
deprecated `hash` field.

Fix #3214
2023-06-27 17:52:15 -07:00
John Freeman
6b4437db39 Fix package definition for Conan (#4485)
Fix the libxrpl library target for consumers using Conan.

* Fix installation issues and update includes.
* Update requirements in the Conan package info.
  * libxrpl requires openssl::crypto.

(Conan is a software package manager for C++.)
2023-06-27 01:23:52 -07:00
Denis Angell
534a36536c refactor: replace hand-rolled lexicalCast (#4473)
Replace hand-rolled code with std::from_chars for better
maintainability.

The C++ std::from_chars function is intended to be as fast as possible,
so it is unlikely to be slower than the code it replaces. This change is
a net gain because it reduces the amount of hand-rolled code.
2023-06-26 23:50:03 -07:00
Elliot Lee
beba87129e Set version to 1.12.0-b1 2023-06-26 14:24:00 -07:00
Shawn Xie
b7e902dccc XLS-39 Clawback: (#4553)
Introduces:
* AccountRoot flag: lsfAllowClawback
* New Clawback transaction
* More info on clawback spec: https://github.com/XRPLF/XRPL-Standards/tree/master/XLS-39d-clawback
2023-06-26 14:07:20 -07:00
Howard Hinnant
9eb30d4316 refactor: remove TypedField's move constructor (#4567)
Apply a minor cleanup in `TypedField`:
* Remove a non-working and unused move constructor.
* Constrain the remaining constructor to not be overly generic enough as
  to be used as a copy or move constructor.
2023-06-26 12:32:10 -07:00
John Freeman
8fdad0d7fd ci: use Artifactory remote in nix workflow (#4556)
There is now an Artifactory (thanks @shichengripple001 and team!) to
hold dependency binaries for the builds.

* Rewrite the `nix` workflow to use it and cut the time down to a mere
  21 minutes.
  * This workflow should continue to work (just more slowly) for forks
    that do not have access to the Artifactory.
2023-06-23 14:20:20 -07:00
drlongle
0b812cdece Add RPC/WS ports to server_info (#4427)
Enhance the /crawl endpoint by publishing WebSocket/RPC ports in the
server_info response. The function processing requests to the /crawl
endpoint actually calls server_info internally, so this change enables a
server to advertise its WebSocket/RPC port(s) to peers via the /crawl
endpoint. `grpc` and `peer` ports are included as well.

The new `ports` array contains objects, each containing a `port` for the
listening port (number string), and an array `protocol` listing the
supported protocol(s).

This allows crawlers to build a richer topology without needing to
port-scan nodes. For non-admin users (including peers), the info about
*admin* ports is excluded.

Also increase test coverage for RPC ServerInfo.

Fix #2837.
2023-06-23 10:19:26 -07:00
Scott Schurr
724a301599 fixReducedOffersV1: prevent offers from blocking order books: (#4512)
Curtail the occurrence of order books that are blocked by reduced offers
with the implementation of the fixReducedOffersV1 amendment.

This commit identifies three ways in which offers can be reduced:

1. A new offer can be partially crossed by existing offers, so the new
   offer is reduced when placed in the ledger.

2. An in-ledger offer can be partially crossed by a new offer in a
   transaction. So the in-ledger offer is reduced by the new offer.

3. An in-ledger offer may be under-funded. In this case the in-ledger
   offer is scaled down to match the available funds.

Reduced offers can block order books if the effective quality of the
reduced offer is worse than the quality of the original offer (from the
perspective of the taker). It turns out that, for small values, the
quality of the reduced offer can be significantly affected by the
rounding mode used during scaling computations.

This commit adjusts some rounding modes so that the quality of a reduced
offer is always at least as good (from the taker's perspective) as the
original offer.

The amendment is titled fixReducedOffersV1 because additional ways of
producing reduced offers may come to light. Therefore, there may be a
future need for a V2 amendment.
2023-06-22 22:20:25 -07:00
Ed Hennis
71d7d67fa3 Enable the Beta RPC API (v2) for all unit tests: (#4573)
* Enable api_version 2, which is currently in beta. It is expected to be
  marked stable by the next stable release.
* This does not change any defaults.
* The only existing tests changed were one that set the same flag, which
  was now redundant, and a couple that tested versioning explicitly.
2023-06-21 11:51:37 -07:00
Elliot Lee
264280edd7 Set version to 1.11.0
* Add release notes
2023-06-20 11:40:11 -07:00
Elliot Lee
beb0904a32 Set version to 1.11.0-rc3 2023-06-09 17:34:40 -07:00
Chenna Keshava B S
77c0a62a74 fix: remove redundant moves (#4565)
- Resolve gcc compiler warning:
      AccountObjects.cpp:182:47: warning: redundant move in initialization [-Wredundant-move]
  - The std::move() operation on trivially copyable types may generate a
    compile warning in newer versions of gcc.
- Remove extraneous header (unused imports) from a unit test file.
2023-06-09 17:33:28 -07:00
Denis Angell
5d011c7e6b fix node size estimation (#4536)
Fix a bug in the `NODE_SIZE` auto-detection feature in `Config.cpp`.
Specifically, this patch corrects the calculation for the total amount
of RAM available, which was previously returned in bytes, but is now
being returned in units of the system's memory unit. Additionally, the
patch adjusts the node size based on the number of available hardware
threads of execution.
2023-06-09 09:37:18 -07:00
Scott Schurr
5644c8704f Trivial: add comments for NFToken-related invariants (#4558) 2023-06-08 17:31:19 -07:00
Scott Determan
c9a586c243 Add missing includes for gcc 13.1: (#4555)
gcc 13.1 failed to compile due to missing headers. This patch adds the
needed headers.
2023-06-05 15:50:03 -07:00
Scott Determan
f709311762 Fix unaligned load and stores: (#4528) (#4531)
Misaligned load and store operations are supported by both Intel and ARM
CPUs. However, in C++, these operations are undefined behavior (UB).
Substituting these operations with a `memcpy` fixes this UB. The
compiled assembly code is equivalent to the original, so there is no
performance penalty to using memcpy.

For context: The unaligned load and store operations fixed here were
originally introduced in the slab allocator (#4218).
2023-05-31 13:28:33 -07:00
oeggert
adde0c2d11 docs(BUILD): restructure content for better readability (#4514)
Follow-up to discussion #4433
2023-05-31 11:55:47 -07:00
Elliot Lee
adf672ff83 docs(README): add link to Clio (#4535) 2023-05-30 09:48:12 -07:00
Elliot Lee
029580886e Set version to 1.11.0-rc2 2023-05-23 14:29:51 -07:00
Ed Hennis
32f8ae1af1 Move faulty assert (#4533)
This assert was put in the wrong place, but it only triggers if shards
are configured. This change moves the assert to the right place and
updates it to ensure correctness.

The assert could be hit after the server downloads some shards. It may
be necessary to restart after the shards are downloaded.

Note that asserts are normally checked only in debug builds, so release
packages should not be affected.

Introduced in: #4319 (66627b26cf)
2023-05-23 14:25:18 -07:00
Scott Determan
ce997a6de8 Ensure that switchover vars are initialized before use: (#4527)
Global variables in different TUs are initialized in an undefined order.
At least one global variable was accessing a global switchover variable.
This caused the switchover variable to be accessed in an uninitialized
state.

Since the switchover is always explicitly set before transaction
processing, this bug can not effect transaction processing, but could
effect unit tests (and potentially the value of some global variables).
Note: at the time of this patch the offending bug is not yet in
production.
2023-05-22 19:36:46 -07:00
Shawn Xie
3620ac287e Add nftoken_id, nftoken_ids, offer_id fields for NFTokens (#4447)
Three new fields are added to the `Tx` responses for NFTs:

1. `nftoken_id`: This field is included in the `Tx` responses for
   `NFTokenMint` and `NFTokenAcceptOffer`. This field indicates the
   `NFTokenID` for the `NFToken` that was modified on the ledger by the
   transaction.
2. `nftoken_ids`: This array is included in the `Tx` response for
   `NFTokenCancelOffer`. This field provides a list of all the
   `NFTokenID`s for the `NFToken`s that were modified on the ledger by
   the transaction.
3. `offer_id`: This field is included in the `Tx` response for
   `NFTokenCreateOffer` transactions and shows the OfferID of the
   `NFTokenOffer` created.

The fields make it easier to track specific tokens and offers. The
implementation includes code (by @ledhed2222) from the Clio project to
extract NFTokenIDs from mint transactions.
2023-05-18 16:38:18 -07:00
John Freeman
629ed5c691 Switch to self-hosted runners for macOS (#4511) 2023-05-17 14:51:42 -07:00
drlongle
78076a6903 fix!: Prevent API from accepting seed or public key for account (#4404)
The API would allow seeds (and public keys) to be used in place of
accounts at several locations in the API. For example, when calling
account_info, you could pass `"account": "foo"`. The string "foo" is
treated like a seed, so the method returns `actNotFound` (instead of
`actMalformed`, as most developers would expect). In the early days,
this was a convenience to make testing easier. However, it allows for
poor security practices, so it is no longer a good idea. Allowing a
secret or passphrase is now considered a bug. Previously, it was
controlled by the `strict` option on some methods. With this commit,
since the API does not interpret `account` as `seed`, the option
`strict` is no longer needed and is removed.

Removing this behavior from the API is a [breaking
change](https://xrpl.org/request-formatting.html#breaking-changes). One
could argue that it shouldn't be done without bumping the API version;
however, in this instance, there is no evidence that anyone is using the
API in the "legacy" way. Furthermore, it is a potential security hole,
as it allows users to send secrets to places where they are not needed,
where they could end up in logs, error messages, etc. There's no reason
to take such a risk with a seed/secret, since only the public address is
needed.

Resolves: #3329, #3330, #4337

BREAKING CHANGE: Remove non-strict account parsing (#3330)
2023-05-16 17:22:10 -07:00
David Fuelling
67238b9fa6 Update environment.md build doc to install lzma: (#4498)
On macOS, if you have not installed something that depends on `xz`, then your
system may lack `lzma`, resulting in a build error similar to:

```
Downloading libarchive-3.6.0.tar.xz completed [6250.61k]
libarchive/3.6.0: 
ERROR: libarchive/3.6.0: Error in source() method, line 120
        get(self, **self.conan_data["sources"][self.version], strip_root=True)
        ReadError: file could not be opened successfully:
- method gz: ReadError('not a gzip file')
- method bz2: ReadError('not a bzip2 file')
- method xz: CompressionError('lzma module is not available')
- method tar: ReadError('invalid header')
```

The solution is to ensure that `lzma` is installed by installing `xz`.
2023-04-27 10:18:59 -07:00
John Freeman
c7ef4c9783 Add patched recipe for SOCI: (#4510)
SOCI is the C++ database access library. The SOCI recipe was updated in
Conan Center Index (CCI), and it breaks for our choice of options. This
breakage occurs when you build with a fresh Conan cache (e.g. when you
submit a PR, or delete `~/.conan/data`).

* Add a custom Conan recipe for SOCI v4.0.3
* Update dependency building to handle exporting and installing Snappy
  and SOCI
  * Fix workflows to use custom SOCI recipe
* Update BUILD.md to include instruction for exporting the SOCI Conan
  recipe:
  * `conan export external/soci soci/4.0.3@`

This solution has been verified on Ubuntu 20.04 and macOS.

Context:

* There is a compiler error that the `sqlite3.h` header is not available
  when building soci.
* When package B depends on package A, it finds the pieces it needs by
  importing the Package Configuration File (PCF) that Conan generates
  for package A.
  * Read the CMake written by package B to check that it is importing
    the PCF correctly and linking its exports correctly.
  * Since this can be difficult, it is often more efficient to check
    https://github.com/conan-io/conan-center-index/issues for package B
    to see if anyone else has seen a similar problem.
  * One of the issues points to a problem area in soci's CMake. To
    confirm the diagnosis, review soci's CMake (after any patches are
    applied) in the Conan build directory `build/$buildId/src/`.
  * Review the Conan-generated PCF in
    `build/$buildId/build/$buildType/generators/`.
  * In this case, the problem was likely (re)introduced by
    https://github.com/conan-io/conan-center-index/pull/17026
* If there is a problem in the source or in the Conan recipe, the
  fastest fix is to copy the recipe and either:
  * Add a source patch to fix any problems in the source.
  * Change the recipe to fix any problems in the recipe.
* In this case, this can be done by finding soci's Conan recipe at
  https://github.com/conan-io/conan-center-index/tree/master/recipes/soci
  and then copying the `all` directory as `external/$packageName` in our
  project. Then, make any changes.
  * Test packages can be removed from the recipe folder as they are not
    needed.
  * If adding a patch in the `patches` directory, add a description for
    it to `conandata.yml`.
  * Since `conanfile.py` has no `version` property on the recipe class,
    builders need to pass a version on the command line (like they do
    for our `snappy` recipe).
* Add an example command to `BUILD.md`.

Future work: It may make sense to refer to recipes by revision, by
checking in a lockfile.
2023-04-25 22:24:41 -07:00
solmsted
b21a05d465 Fix typo (#4508) 2023-04-25 14:11:08 -07:00
John Freeman
436de0e03a Expand Linux test matrix: (#4454)
This change makes progress on the plan in #4371. It does not replicate
the full [matrix] implemented in #3851, but it does replicate the 1.ii
section of the Linux matrix. It leverages "heavy" self-hosted runners,
and demonstrates a repeatable pattern for future matrices.

[matrix]: d794a0f3f1/.github/README.md (continuous-integration)
2023-04-24 16:17:51 -07:00
John Freeman
8d482d3557 Fix errors for Clang 16: (#4501)
Address issues related to the removal of `std::{u,bi}nary_function` in
C++17 and some warnings with Clang 16. Some warnings appeared with the
upgrade to Apple clang version 14.0.3 (clang-1403.0.22.14.1).

- `std::{u,bi}nary_function` were removed in C++17. They were empty
  classes with a few associated types. We already have conditional code
  to define the types. Just make it unconditional.
- libc++ checks a cast in an unevaluated context to see if a type
  inherits from a binary function class in the standard library, e.g.
  `std::equal_to`, and this causes an error when the type privately
  inherits from such a class. Change these instances to public
  inheritance.
- We don't need a middle-man for the empty base optimization. Prefer to
  inherit directly from an empty class than from
  `beast::detail::empty_base_optimization`.
- Clang warns when all the uses of a variable are removed by conditional
  compilation of assertions. Add a `[[maybe_unused]]` annotation to
  suppress it.
- As a drive-by clean-up, remove commented code.

See related work in #4486.
2023-04-21 12:20:35 -07:00
Mark Travis
c5003969de Use quorum specified via command line: (#4489)
If `--quorum` setting is present on the command line, use the specified
value as the minimum quorum. This allows for the use of a potentially
fork-unsafe quorum, but it is sometimes necessary for small and test
networks.

Fix #4488.

---------

Co-authored-by: RichardAH <richard.holland@starstone.co.nz>
2023-04-20 11:36:18 -07:00
John Freeman
1f417764c3 Add install instructions for package managers: (#4472)
Add instructions for installing rippled using the package managers APT
and YUM. Some steps were adapted from xrpl.org.

---------

Co-authored-by: Michael Legleux <mlegleux@ripple.com>
2023-04-13 10:41:16 -07:00
John Freeman
e75cd49313 Fix the fix for std::result_of (#4496)
Newer compilers, such as Apple Clang 15.0, have removed `std::result_of`
as part of C++20. The build instructions provided a fix for this (by
adding a preprocessor definition), but the fix was broken.

This fixes the fix by:
* Adding the `conf` prefix for tool configurations (which had been
  forgotten).
* Passing `extra_b2_flags` to `boost` package to fix its build.
  * Define `BOOST_ASIO_HAS_STD_INVOKE_RESULT` in order to build boost
    1.77 with a newer compiler.
2023-04-12 16:32:37 -07:00
RichardAH
4f95b9d7a6 Prevent replay attacks with NetworkID field: (#4370)
Add a `NetworkID` field to help prevent replay attacks on and from
side-chains.

The new field must be used when the server is using a network id > 1024.

To preserve legacy behavior, all chains with a network ID less than 1025
retain the existing behavior. This includes Mainnet, Testnet, Devnet,
and hooks-testnet. If `sfNetworkID` is present in any transaction
submitted to any of the nodes on one of these chains, then
`telNETWORK_ID_MAKES_TX_NON_CANONICAL` is returned.

Since chains with a network ID less than 1025, including Mainnet, retain
the existing behavior, there is no need for an amendment.

The `NetworkID` helps to prevent replay attacks because users specify a
`NetworkID` field in every transaction for that chain.

This change introduces a new UINT32 field, `sfNetworkID` ("NetworkID").
There are also three new local error codes for transaction results:

- `telNETWORK_ID_MAKES_TX_NON_CANONICAL`
- `telREQUIRES_NETWORK_ID`
- `telWRONG_NETWORK`

To learn about the other transaction result codes, see:
https://xrpl.org/transaction-results.html

Local error codes were chosen because a transaction is not necessarily
malformed if it is submitted to a node running on the incorrect chain.
This is a local error specific to that node and could be corrected by
switching to a different node or by changing the `network_id` on that
node. See:
https://xrpl.org/connect-your-rippled-to-the-xrp-test-net.html

In addition to using `NetworkID`, it is still generally recommended to
use different accounts and keys on side-chains. However, people will
undoubtedly use the same keys on multiple chains; for example, this is
common practice on other blockchain networks. There are also some
legitimate use cases for this.

A `app.NetworkID` test suite has been added, and `core.Config` was
updated to include some network_id tests.
2023-04-11 17:11:17 -07:00
Nik Bougalis
066f91ca07 Avoid using std::shared_ptr when not necessary: (#4218)
The `Ledger` class contains two `SHAMap` instances: the state and
transaction maps. Previously, the maps were dynamically allocated using
`std::make_shared` despite the fact that they did not require lifetime
management separate from the lifetime of the `Ledger` instance to which
they belong.

The two `SHAMap` instances are now regular member variables. Some smart
pointers and dynamic memory allocation was avoided by using stack-based
alternatives.

Commit 3 of 3 in #4218.
2023-04-11 15:50:25 -07:00
Nik Bougalis
c3acbce82d Optimize SHAMapItem and leverage new slab allocator: (#4218)
The `SHAMapItem` class contains a variable-sized buffer that
holds the serialized data associated with a particular item
inside a `SHAMap`.

Prior to this commit, the buffer for the serialized data was
allocated separately. Coupled with the fact that most instances
of `SHAMapItem` were wrapped around a `std::shared_ptr` meant
that an instantiation might result in up to three separate
memory allocations.

This commit switches away from `std::shared_ptr` for `SHAMapItem`
and uses `boost::intrusive_ptr` instead, allowing the reference
count for an instance to live inside the instance itself. Coupled
with using a slab-based allocator to optimize memory allocation
for the most commonly sized buffers, the net result is significant
memory savings. In testing, the reduction in memory usage hovers
between 400MB and 650MB. Other scenarios might result in larger
savings.

In performance testing with NFTs, this commit reduces memory size by
about 15% sustained over long duration.

Commit 2 of 3 in #4218.
2023-04-10 17:13:03 -07:00
Nik Bougalis
b7f588b789 Introduce support for a slabbed allocator: (#4218)
When instantiating a large amount of fixed-sized objects on the heap
the overhead that dynamic memory allocation APIs impose will quickly
become significant.

In some cases, allocating a large amount of memory at once and using
a slabbing allocator to carve the large block into fixed-sized units
that are used to service requests for memory out will help to reduce
memory fragmentation significantly and, potentially, improve overall
performance.

This commit introduces a new `SlabAllocator<>` class that exposes an
API that is _similar_ to the C++ concept of an `Allocator` but it is
not meant to be a general-purpose allocator.

It should not be used unless profiling and analysis of specific memory
allocation patterns indicates that the additional complexity introduced
will improve the performance of the system overall, and subsequent
profiling proves it.

A helper class, `SlabAllocatorSet<>` simplifies handling of variably
sized objects that benefit from slab allocations.

This commit incorporates improvements suggested by Greg Popovitch
(@greg7mdp).

Commit 1 of 3 in #4218.
2023-04-10 14:22:59 -07:00
ledhed2222
9346842eed Add jss fields used by Clio nft_info: (#4320)
Add Clio-specific JSS constants to ensure a common vocabulary of
keywords in Clio and this project. By providing visibility of the full
API keyword namespace, it reduces the likelihood of developers
introducing minor variations on names used by Clio, or unknowingly
claiming a keyword that Clio has already claimed. This change moves this
project slightly away from having only the code necessary for running
the core server, but it is a step toward the goal of keeping this
server's and Clio's APIs similar. The added JSS constants are annotated
to indicate their relevance to Clio.

Clio can be found here: https://github.com/XRPLF/clio

Signed-off-by: ledhed2222 <ledhed2222@users.noreply.github.com>
2023-04-06 11:33:20 -07:00
RichardAH
f191c911d4 Add NFTokenPages to account_objects RPC: (#4352)
- Include NFTokenPages in account_objects to make it easier to
  understand an account's Owner Reserve and simplify app development.
- Update related tests and documentation.
- Fix #4347.

For info about the Owner Reserve, see https://xrpl.org/reserves.html

---------

Co-authored-by: Scott Schurr <scott@ripple.com>
Co-authored-by: Ed Hennis <ed@ripple.com>
2023-04-05 13:58:55 -07:00
drlongle
e6f49040f5 Fix unit test app.LedgerData (#4484) 2023-03-31 11:18:42 -07:00
drlongle
2f3f6dcb03 Fix ledger_data to return an empty list: (#4398)
Change `ledger_data` to return an empty list when all entries are
filtered out.

When the `type` field is specified for the `ledger_data` method, it is
possible that no objects of the specified type are found. This can even
occur if those objects exist, but not in the section that the server
checked while serving your request. Previously, the `state` field of the
response has the value `null`, instead of an empty array `[]`. By
changing this to an empty array, the response is the same data type so
that clients can handle it consistently.

For example, in Python, `for entry in state` should now work correctly.
It would raise an exception if `state` is `null` (or `None`). 

This could break client code that explicitly checks for null. However,
this fix aligns the response with the documentation, where the `state`
field is an array.

Fix #4392.
2023-03-30 11:59:10 -07:00
drlongle
5ebcaf0a6c Add account flags to account_info response: (#4459)
Previously, the object `account_data` in the `account_info` response
contained a single field `Flags` that contains flags of an account. API
consumers must perform bitwise operations on this field to retrieve the
account flags.

This change adds a new object, `account_flags`, at the top level of the
`account_info` response `result`. The object contains relevant flags of
the account. This makes it easier to write simple code to check a flag's
value.

The flags included may depend on the amendments that are enabled.

Fix #2457.
2023-03-30 11:46:18 -07:00
drlongle
8bfdbcbab5 Add logging for exceptions: (#4400)
Log exception messages at several locations.

Previously, these were locations where an exception was caught, but the
exception message was not logged. Logging the exception messages can be
useful for analysis or debugging. The additional logging could have a
small negative performance impact.

Fix #3213.
2023-03-30 10:13:30 -07:00
Brandon Wilson
135b63dbe0 Update example [validator_list_sites] (#4448) 2023-03-29 23:01:41 -07:00
Elliot Lee
46167d1c46 Add link to BUILD.md: (#4450)
In the release notes (current and historical), there is a link to the
`Builds` directory. By creating `Builds/README.md` with a link to
`BUILD.md`, it is easier to find the build instructions.
2023-03-28 15:55:53 -07:00
Alloy Networks
79e621d96c Update README.md (#4463) 2023-03-28 12:04:06 -07:00
Ed Hennis
66627b26cf Refactor fee initialization and configuration: (#4319)
* Create the FeeSettings object in genesis ledger.
* Initialize with default values from the config. Removes the need to
  pass a Config down into the Ledger initialization functions, including
  setup().
* Drop the undocumented fee config settings in favor of the [voting]
  section.
  * Fix #3734.
  * If you previously used fee_account_reserve and/or fee_owner_reserve,
    you should change to using the [voting] section instead. Example:

```
[voting]
account_reserve=10000000
owner_reserve=2000000
```

* Because old Mainnet ledgers (prior to 562177 - yes, I looked it up)
  don't have FeeSettings, some of the other ctors will default them to
  the config values before setup() tries to load the object.
* Update default Config fee values to match Mainnet.
* Fix unit tests:
  * Updated fees: Some tests are converted to use computed values of fee
    object, but the default Env config was also updated to fix the rest.
  * Unit tests that check the structure of the ledger have updated
    hashes and counts.
2023-03-28 09:03:25 -07:00
Ed Hennis
7aad6e5127 feat: mark 4 amendments as obsolete: (#4291)
Add the ability to mark amendments as obsolete. There are some known
amendments that should not be voted for because they are broken (or
similar reasons).

This commit marks four amendments as obsolete:

1. `CryptoConditionsSuite`
2. `NonFungibleTokensV1`
3. `fixNFTokenDirV1`
4. `fixNFTokenNegOffer`

When an amendment is `Obsolete`, voting for the amendment is prevented.
A determined operator can still vote for the amendment by changing the
source, and doing so does not break any protocol rules.

The "feature" command now does not modify the vote for obsolete
amendments.

Before this change, there were two options for an amendment's
`DefaultVote` behavior: yes and no.

After this change, there are three options for an amendment's
`VoteBehavior`: DefaultYes, DefaultNo, and Obsolete.

To be clear, if an obsolete amendment were to (somehow) be activated by
consensus, the server still has the code to process transactions
according to that amendment, and would not be amendment blocked. It
would function the same as if it had been voting "no" on the amendment.

Resolves #4014.

Incorporates review feedback from @scottschurr.
2023-03-23 22:28:53 -07:00
Scott Schurr
dffcdea12b fix: Expected to return a value: (#4401)
Fix a case where `ripple::Expected` returned a json array, not a value.

The problem was that `Expected` invoked the wrong constructor for the
expected type, which resulted in a constructor that took multiple
arguments being interpreted as an array.

A proposed fix was provided by @godexsoft, which involved a minor
adjustment to three constructors that replaces the use of curly braces
with parentheses. This makes `Expected` usable for
[Clio](https://github.com/XRPLF/clio).

A unit test is also included to ensure that the issue doesn't occur
again in the future.
2023-03-23 17:32:17 -07:00
John Freeman
d7725837f5 build: add interface library libxrpl: (#4449)
Make it easy for projects to depend on libxrpl by adding an `ALIAS`
target named `xrpl::libxrpl` for projects to link.

The name was chosen because:

* The current library target is named `xrpl_core`. There is no other
  "non-core" library target against which we need to distinguish the
  "core" library. We only export one library target, and it should just
  be named after the project to keep things simple and predictable.
* Underscores in target or library names are generally discouraged.
* Every target exported in CMake should be prefixed with the project
  name.

By adding an `ALIAS` target, existing consumers who use the `xrpl_core`
target will not be affected.

* In the future, there can be a migration plan to make `xrpl_core` the
  `ALIAS` target (and `libxrpl` the "real" target, which will affect the
  filename of the compiled binary), and eventually remove it entirely.

Also:

* Fix the Conan recipe so that consumers using Conan import a target
  named `xrpl::libxrpl`. This way, every consumer can use the same
  instructions.
* Document the two easiest methods to depend on libxrpl. Both have been
  tested.
* See #4443.
2023-03-22 17:21:03 -07:00
John Freeman
7745c72b2c docs: update build instructions: (#4381)
* Remove obsolete build instructions.
* By using Conan, builders can choose which dependencies specifically to
  build and link as shared objects.
* Refactor the build instructions based on the plan in #4433.
2023-03-22 12:02:42 -07:00
Elliot Lee
acb373280b Merge branch 'master' (1.10.1) into develop 2023-03-22 11:13:36 -07:00
Elliot Lee
4f506599f6 Set version to 1.10.1
* Add release notes
2023-03-22 09:27:56 -07:00
Elliot Lee
383f1b6ab3 Set version to 1.10.1-rc1 2023-03-21 11:14:20 -07:00
Michael Legleux
da18c86cbf Build packages with Ubuntu 18.04
Restores Ubuntu 18.04 packages
Update docker images to use Conan
2023-03-21 11:13:03 -07:00
Elliot Lee
9fcb28acad docs: update protocol README (#4457) 2023-03-21 08:01:47 -07:00
Shawn Xie
305c9a8d61 fixNFTokenRemint: prevent NFT re-mint: (#4406)
Without the protocol amendment introduced by this commit, an NFT ID can
be reminted in this manner:

1. Alice creates an account and mints an NFT.
2. Alice burns the NFT with an `NFTokenBurn` transaction.
3. Alice deletes her account with an `AccountDelete` transaction.
4. Alice re-creates her account.
5. Alice mints an NFT with an `NFTokenMint` transaction with params:
   `NFTokenTaxon` = 0, `Flags` = 9).

This will mint a NFT with the same `NFTokenID` as the one minted in step
1. The params that construct the NFT ID will cause a collision in
`NFTokenID` if their values are equal before and after the remint.

With the `fixNFTokenRemint` amendment, there is a new sequence number
construct which avoids this scenario:

- A new `AccountRoot` field, `FirstNFTSequence`, stays constant over
  time.
  - This field is set to the current account sequence when the account
    issues their first NFT.
  - Otherwise, it is not set.
- The sequence of a newly-minted NFT is computed by: `FirstNFTSequence +
  MintedNFTokens`.
  - `MintedNFTokens` is then incremented by 1 for each mint.

Furthermore, there is a new account deletion restriction:

- An account can only be deleted if `FirstNFTSequence + MintedNFTokens +
  256` is less than the current ledger sequence.
  - 256 was chosen because it already exists in the current account
    deletion constraint.

Without this restriction, an NFT may still be remintable. Example
scenario:

1. Alice's account sequence is at 1.
2. Bob is Alice's authorized minter.
3. Bob mints 500 NFTs for Alice. The NFTs will have sequences 1-501, as
   NFT sequence is computed by `FirstNFTokenSequence + MintedNFTokens`).
4. Alice deletes her account at ledger 257 (as required by the existing
   `AccountDelete` amendment).
5. Alice re-creates her account at ledger 258.
6. Alice mints an NFT. `FirstNFTokenSequence` initializes to her account
   sequence (258), and `MintedNFTokens` initializes as 0. This
   newly-minted NFT would have a sequence number of 258, which is a
   duplicate of what she issued through authorized minting before she
   deleted her account.

---------

Signed-off-by: Shawn Xie <shawnxie920@gmail.com>
2023-03-20 14:47:46 -07:00
Ed Hennis
9b2d563dec fix: support RPC markers for any ledger object: (#4361)
There were situations where `marker`s  returned by `account_lines` did
not work on subsequent requests, returning "Invalid Parameters".

This was caused by the optimization implemented in "Enforce account RPC
limits by account objects traversed":

e28989638d

Previously, the ledger traversal would find up to `limit` account lines,
and if there were more, the marker would be derived from the key of the
next account line. After the change, ledger traversal would _consider_
up to `limit` account objects of any kind found in the account's
directory structure. If there were more, the marker would be derived
from the key of the next object, regardless of type.

With this optimization, it is expected that `account_lines` may return
fewer than `limit` account lines - even 0 - along with a marker
indicating that there are may be more available.

The problem is that this optimization did not update the
`RPC::isOwnedByAccount` helper function to handle those other object
types. Additionally, XLS-20 added `ltNFTOKEN_OFFER` ledger objects to
objects that have been added to the account's directory structure, but
did not update `RPC::isOwnedByAccount` to be able to handle those
objects. The `marker` provided in the example for #4354 includes the key
for an `ltNFTOKEN_OFFER`. When that `marker` is used on subsequent
calls, it is not recognized as valid, and so the request fails.

* Add unit test that walks all the object types and verifies that all of
  their indexes can work as a marker.
* Fix #4340
* Fix #4354
2023-03-20 10:22:15 -07:00
Nik Bougalis
150d4a47e4 refactor: optimize NodeStore object conversion: (#4353)
When writing objects to the NodeStore, we need to convert them from
the in-memory format to the binary format used by the node store.

The conversion is handled by the `EncodedBlob` class, which is only
instantiated on the stack. Coupled with the fact that most objects
are under 1024 bytes in size, this presents an opportunity to elide
a memory allocation in a critical path.

This commit also simplifies the interface of `EncodedBlob` and
eliminates a subtle corner case that could result in dangling
pointers.

These changes are not expected to cause a significant reduction in
memory usage. The change avoids the use of a `std::shared_ptr` when
unnecessary and tries to use stack-based memory allocation instead
of the heap whenever possible.

This is a net gain both in terms of memory usage (lower
fragmentation) and performance (less work to do at runtime).
2023-03-16 15:00:07 -07:00
Ed Hennis
1c9df69b33 fix(ValidatorSite): handle rare null pointer dereference in timeout: (#4420)
In rare circumstances, both `onRequestTimeout` and the response handler
(`onSiteFetch` or `onTextFetch`) can get queued and processed. In all
observed cases, the response handler processes a network error.
`onRequestTimeout` usually runs first, but on rare occasions, the
response handler runs first, which leaves `activeResource` empty.
2023-03-16 10:32:22 -07:00
RichardAH
10555faa92 fix(gateway_balances): handle overflow exception: (#4355)
* Prevent internal error by catching overflow exception in `gateway_balances`.
* Treat `gateway_balances` obligations overflow as max (largest valid) `STAmount`.
  * Note that very large sums of STAmount are approximations regardless.

---------

Co-authored-by: Scott Schurr <scott@ripple.com>
2023-03-16 10:25:40 -07:00
Elliot Lee
0f1ffff068 Set version to 1.10.1-b1 2023-03-14 21:21:50 -07:00
Chenna Keshava B S
9309b57364 Rectify the import paths of boost/iterator: (#4293)
- MSVC 19.x reported a warning about import paths in boost for
  function_output_iterator class (boost::function_output_iterator).
- Eliminate that warning by updating the import paths, as suggested by
  the compiler warnings.
2023-03-14 21:10:56 -07:00
RichardAH
cb08f2b6ec Allow port numbers be be specified with a colon: (#4328)
Port numbers can now be specified using either a colon or a space.

Examples:

1.2.3.4:51235

1.2.3.4 51235

- In the configuration file, an annoying "gotcha" for node operators is
  accidentally specifying IP:PORT combinations using a colon. The code
  previously expected a space, not a colon. It also does not provide
  good feedback when this operator error is made.
- This change simply allows this mistake (using a colon) to be fixed
  automatically, preserving the intention of the operator.
- Add unit tests, which test the functionality when specifying IP:PORT
  in the configuration file.
- The RPCCall test regime is not specific enough to test this
  functionality, it has been tested by hand.
- Ensure IPv6 addresses are not confused for ip:port

---------

Co-authored-by: Elliot Lee <github.public@intelliot.com>
2023-03-14 21:06:30 -07:00
drlongle
84cde3ce0b Use <=> operator for base_uint, Issue, and Book: (#4411)
- Implement the `operator==` and the `operator<=>` (aka the spaceship
  operator) in `base_uint`, `Issue`, and `Book`. 
- C++20-compliant compilers automatically provide the remaining
  comparison operators (e.g. `operator<`, `operator<=`, ...).
- Remove the function compare() because it is no longer needed.
- Maintain the same semantics as the existing code.
- Add some unit tests to gain further confidence.
- Fix #2525.
2023-03-14 20:54:54 -07:00
Mark Travis
f7b3ddd87b Reporting Mode: Do not attempt to acquire missing data from peer network (#4458)
In Reporting Mode, a server would core dump when it is not able to read
from Cassandra. This patch prevents the core dump when Cassandra is down
for reporting mode servers. This does not fix the root cause, but it
cuts down on some of the resulting noise.
2023-03-14 20:49:40 -07:00
Elliot Lee
1e7710eee2 docs: security bug bounty acknowledgements (#4460) 2023-03-14 13:08:56 -07:00
Elliot Lee
07f047b1e2 Set version to 1.10.0
Merge #4451
2023-03-14 09:30:22 -07:00
Elliot Lee
8687b5c3c9 Set version to 1.10.0-rc4 2023-03-02 14:28:08 -08:00
Kenny Lei
ecd49e1535 Rename 'NFT' to 'NFToken' in DisallowIncoming flags (#4442)
* Follow-up to #4336
* NFToken is the naming convention in the codebase (rather than NFT)
* Rename `lsfDisallowIncomingNFTOffer` to `lsfDisallowIncomingNFTokenOffer`
* Rename `asfDisallowIncomingNFTOffer` to `asfDisallowIncomingNFTokenOffer`
2023-03-02 13:01:50 -08:00
Elliot Lee
c77a8d5ec6 Update Docker.md (#4432)
* Add links to some related resources that may be helpful.
* Docker images can make testing easier to do.
2023-03-02 10:07:09 -08:00
Michael Legleux
e13676f709 Update package building scripts and images to use Conan (#4435)
* Set version to 1.10.0-rc3
* Test on Fedora 37
* Dependency builds are handled by Conan
2023-02-28 14:17:07 -08:00
Elliot Lee
74594d5348 Disable duplicate detector: (#4438)
Partially revert the functionality introduced
with #4195 / 5a15229 (part of 1.10.0-b1).

Acknowledgements:
Aaron Hook for responsibly disclosing this issue.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.

To report a bug, please send a detailed report to:

    bugs@xrpl.org

---------

Co-authored-by: Nik Bougalis <nikb@bougalis.net>
2023-02-28 14:12:28 -08:00
Ed Hennis
caf4827c0b Undo API changes introduced in XRPFees: (#4429)
* Original changes: e4b17d1cf2
* Resolve #4425
2023-02-24 08:41:28 -08:00
Elliot Lee
c2b03fecca Remove recipe for RocksDB and add recipe for Snappy (#4431)
- Copies the recipe for Snappy from Conan Center, but removes three
  lines that explicitly link the standard library, which prevents
  builders from statically linking it.
- Removes the recipe for RocksDB now that an official recipe for version
  6.27.3 is in Conan Center.

Developers will likely need to remove cached versions of both RocksDB
and Snappy:

```
conan remove -f rocksdb
conan remove -f snappy
```

---------

Co-authored-by: John Freeman <jfreeman08@gmail.com>
2023-02-23 12:41:18 -08:00
John Freeman
60c276d90b Fix Conan version constraint in workflows (#4430)
Conan 2.0 is not compatible with the current workflows.
2023-02-22 19:54:51 -08:00
Levin Winter
2929748898 Refactor getTrustedForLedger() (#4424)
Look for validations associated with a specific ledger ID and sequence
number.
2023-02-22 15:20:15 -08:00
Elliot Lee
5ec8783d35 README: Update "Build from source" section (#4426)
Fix #2987
2023-02-21 15:40:09 -08:00
Elliot Lee
96aab1288f README: Add a few source code starting points (#4421)
Co-authored-by: John Freeman <jfreeman08@gmail.com>
2023-02-17 22:27:27 -08:00
Ed Hennis
aebf2ac990 Resolve a couple of Github Action CI annoyances: (#4413)
* Set "fail-fast: false" so that multiple jobs in one workflow can
  finish independently. By default, if one job fails, other running jobs
  will be aborted, even if the other jobs are working fine and are
  almost done. This leads to wasted time and resources if the failure
  is, for example, OS specific, or due to a flaky unit test, and the
  failed job needs to be re-run, because all the jobs end up re-running.
* Put conditions back into the windows.yml job (manual, and for
  a specific branch name and that job). This prevents Github Actions
  from sending "No jobs were run" failure emails on every commit.
2023-02-15 11:34:32 -08:00
ledhed2222
ac78b7a9a7 Rename to fixNonFungibleTokensV1_2 and some cosmetic changes (#4419) 2023-02-13 15:52:40 -08:00
Denis Angell
b72a87c7d3 Only account specified as destination can settle through brokerage: (#4399)
Without this amendment, for NFTs using broker mode, if the sell offer contains a destination and that destination is the buyer account, anyone can broker the transaction. Also, if a buy offer contains a destination and that destination is the seller account, anyone can broker the transaction. This is not ideal and is misleading.

Instead, with this amendment: If you set a destination, that destination needs to be the account settling the transaction. So, the broker must be the destination if they want to settle. If the buyer is the destination, then the buyer must accept the sell offer, as you cannot broker your own offers.

If users want their offers open to the public, then they should not set a destination. On the other hand, if users want to limit who can settle the offers, then they would set a destination.

Unit tests:

1. The broker cannot broker a destination offer to the buyer and the buyer must accept the sell offer. (0 transfer)
2. If the broker is the destination, the broker will take the difference. (broker mode)
2023-02-13 15:52:39 -08:00
Scott Schurr
39c32561bd Prevent brokered sale of NFToken to owner: (#4403)
Fixes #4374

It was possible for a broker to combine a sell and a buy offer from an account that already owns an NFT. Such brokering extracts money from the NFT owner and provides no benefit in return.

With this amendment, the code detects when a broker is returning an NFToken to its initial owner and prohibits the transaction. This forbids a broker from selling an NFToken to the account that already owns the token. This fixes a bug in the original implementation of XLS-20.

Thanks to @nixer89 for suggesting this fix.
2023-02-13 15:52:39 -08:00
ledhed2222
89aa8b21ec Fix 3 issues around NFToken offer acceptance (#4380)
Fixes 3 issues:

In the following scenario, an account cannot perform NFTokenAcceptOffer even though it should be allowed to:

- BROKER has < S
- ALICE offers to sell token for S
- BOB offers to buy token for > S
- BROKER tries to bridge the two offers

This currently results in `tecINSUFFICIENT_FUNDS`, but should not because BROKER is not spending any funds in this transaction, beyond the transaction fee.

When trading an NFT using IOUs, and when the issuer of the IOU has any non-zero value set for TransferFee on their account via AccountSet (not a TransferFee on the NFT), and when the sale amount is equal to the total balance of that IOU that the buyer has, the resulting balance for the issuer of the IOU will become positive. This means that the buyer of the NFT was supposed to have caused a certain amount of IOU to be burned. That amount was unable to be burned because the buyer couldn't cover it. This results in the buyer owing this amount back to the issuer. In a real world scenario, this is appropriate and can be settled off-chain.

Currency issuers could not make offers for NFTs using their own currency, receiving `tecINSUFFICIENT_FUNDS` if they tried to do so.

With this fix, they are now able to buy/sell NFTs using their own currency.
2023-02-13 15:52:39 -08:00
Shawn Xie
a828e24cf0 Allow NFT to be burned when number of offers is greater than 500 (#4346)
* Allow offers to be removable
* Delete sell offers first

Signed-off-by: Shawn Xie <shawnxie920@gmail.com>
2023-02-13 15:52:38 -08:00
ledhed2222
f7a8d2de84 Add fixUnburnableNFToken feature (#4391) 2023-02-13 15:52:38 -08:00
Howard Hinnant
32559463ef Change default vote on fixUniversalNumber from yes to no (#4414) 2023-02-09 21:01:48 -08:00
Ed Hennis
8f514937a4 Update documented pathfinding configuration defaults: (#4409)
* Add recommended values for node which want to support "advanced"
  pathfinding, which is effectively the old default behavior.
2023-02-08 12:14:09 -08:00
John Freeman
36b34a7bd5 Update dependency: grpc (#4407) 2023-02-07 15:46:38 -08:00
Howard Hinnant
5edaec2bd0 Introduce min/max observers for Number
Three static member functions are introduced with
definitions consistent with std::numeric_limits:

static constexpr Number min() noexcept;

  Returns: The minimum positive value.  This is the value closest to zero.

static constexpr Number max() noexcept;

  Returns: The maximum possible value.

static constexpr Number lowest() noexcept;

  Returns: The negative value which is less than all other values.
2023-02-07 15:43:28 -08:00
Howard Hinnant
2f1f453052 Optimize uint128_t division by 10 within Number.cpp
* Optimization includes computing remainder from division.
* Used only within Number::operator*=.
2023-02-07 15:43:28 -08:00
Howard Hinnant
6eaaa7bcfa Replace Number division algorithm
* Replace division with faster algorithm.
* Correct some rounding bugs in multiplication.
* Add tests for rounding bugs.
2023-02-07 15:43:28 -08:00
Howard Hinnant
e354497f63 Include rounding mode in XRPAmount to STAmount conversion. 2023-02-07 15:43:28 -08:00
Howard Hinnant
6fcd654bee Remove undefined behavior
* Taking the negative of a signed negative is UB, but
  taking the negative of an unsigned is not.
2023-02-07 15:43:28 -08:00
Howard Hinnant
d275a2ab72 Silence warnings 2023-02-07 15:43:28 -08:00
Howard Hinnant
3f33471220 Introduce rounding modes for Number:
You can set a thread-local flag to direct Number how to round
non-exact results with the syntax:

    Number::rounding_mode prev_mode = Number::setround(Number::towards_zero);

This flag will stay in effect for this thread only until another call
to setround.  The previously set rounding mode is returned.

You can also retrieve the current rounding mode with:

    Number::rounding_mode current_mode = Number::getround();

The available rounding modes are:

* to_nearest : Rounds to nearest representable value.  On tie, rounds
               to even.
* towards_zero : Rounds towards zero.
* downward : Rounds towards negative infinity.
* upward : Rounds towards positive infinity.

The default rounding mode is to_nearest.
2023-02-07 15:43:28 -08:00
Howard Hinnant
a82ad5ba76 Use Number for IOUAmount and STAmount arithmetic
* Guarded by amendment fixUniversalNumber
* Produces slightly better accuracy in some computations.
2023-02-07 15:43:28 -08:00
Howard Hinnant
48e804c40c Add tests 2023-02-07 15:43:28 -08:00
Howard Hinnant
c9c54c9799 Add implicit conversion from STAmount to Number 2023-02-07 15:43:28 -08:00
Howard Hinnant
24fe5f9fd0 Add clip
* Return 0 if abs(x) < limit, else returns x
2023-02-07 15:43:28 -08:00
Howard Hinnant
476ee8a479 Add conversions between Number, XRPAmount and int64_t
* Conversions to Number are implicit
* Conversions away from Number are explicit and potentially lossy
* If lossy, round to nearest, and to even on tie
2023-02-07 15:43:28 -08:00
Howard Hinnant
0ee63b7c7b AMM Add Number class and associated algorithms 2023-02-07 15:43:28 -08:00
John Freeman
31e7e5a56e Revise CONTRIBUTING (#4382)
- Wrap text at 80 columns.
- Match capitalization of GitHub usernames.
- Prescribe more rules for pull requests.
- Link more reference documentation.
2023-02-05 16:08:26 -08:00
Ed Hennis
e4b17d1cf2 XRPFees: Fee setting and handling improvements (#4247)
* Introduces amendment `XRPFees`
* Convert fee voting and protocol messages to use XRPAmounts
* Includes Validations, Change transactions, the "Fees" ledger object,
  and subscription messages

* Improve handling of 0 drop reference fee with TxQ. For use with networks that do not want to require fees
* Note that fee escalation logic is still in place, which may cause the
  open ledger fee to rise if the network is busy. 0 drop transactions
  will still queue, and fee escalation can be effectively disabled by
  modifying the configuration on all nodes

* Change default network reserves to match Mainnet

* Name the new SFields *Drops (not *XRP)
* Reserve SField IDs for Hooks

* Clarify comments explaining the ttFEE transaction field validation
2023-02-02 16:20:35 -08:00
John Freeman
0ce15e0e35 Update BUILD.md (#4383)
Help readers of BUILD.md by moving the crash course to the top
2023-01-11 21:03:38 -08:00
John Freeman
b0e0f319a1 Make NodeToShardRPC a manual test (#4379)
Right now it is flaky, which creates noise in our automated testing.
2023-01-05 18:28:37 -06:00
John Freeman
2233f585f8 Update build instructions (#4376)
Document minimum compiler version; how to choose compiler; how to fix missing `std::result_of`; how to create Conan profile.
2023-01-05 18:27:53 -06:00
Scott Schurr
61d8c7a85b Add a unit test for invalid memos (#4287) 2023-01-04 15:45:19 -08:00
Scott Schurr
6f8750316c RPC tooBusy response has 503 HTTP status if "ripplerpc": "3.0": (#4143)
Fixes #4005

Makes it possible for internal RPC Error Codes to associate
themselves with a non-OK (200) HTTP status code.  There are
quite a number of RPC responses in addition to tooBusy that
now have non-OK HTTP status codes.

The new return HTTP return codes are only enabled by including
"ripplerpc": "3.0" or higher in the original request.
Otherwise the historical value, 200, continues to be returned.
This ensures that this is not a breaking change.
2023-01-03 09:24:45 -08:00
John Freeman
fda9e9a7ee Fix Doxygen workflow (#4372) 2022-12-21 10:41:16 -08:00
RichardAH
d8a84e9530 featureDisallowIncoming: Opt-out of incoming Checks, PayChans, NFTokenOffers and Trustlines (#4336)
featureDisallowIncoming is a new amendment that would allow users to opt-out of incoming Checks, Payment Channels, NFTokenOffers, and trust lines. This commit includes tests.

Adds four new AccountSet Flags:
1. asfDisallowIncomingNFTOffer
2. asfDisallowIncomingCheck
3. asfDisallowIncomingPayChan
4. asfDisallowIncomingTrustline
2022-12-19 17:35:35 -08:00
John Freeman
c3a9f3dbf3 Use the Conan package manager (#4367)
Introduces a conanfile.py (and a Conan recipe for RocksDB) to enable building the package with Conan, choosing more recent default versions of dependencies. It removes almost all of the CMake build files related to dependencies, and the configurations for Travis CI and GitLab CI. A new set of cross-platform build instructions are written in BUILD.md.

Includes example GitHub Actions workflow for each of Linux, macOS, Windows.

* Test on macos-12

We use the <concepts> library which was not added to Apple Clang until
version 13.1.6. The default Clang on macos-11 (the sometimes current
version of macos-latest) is 13.0.0, and the default Clang on macos-12 is
14.0.0.

Closes #4223.
2022-12-16 10:46:22 -08:00
CJ Cobb
df1300fb37 Catch transaction deserialization error in doLedgerGrpc (#4323)
* Allow clio to extract ledgers with transactions that can no longer be
  deserialized. The problem transactions will be skipped.
2022-12-13 17:09:53 -08:00
Scott Determan
648d6c3e2f Fix clang 15 warnings: (#4325)
Clang warned about the code removed in this patch with the warning:
```
warning: out-of-line definition of constexpr static data member is
redundant in C++17 and is deprecated [-Wdeprecated]
```
2022-12-13 16:21:22 -08:00
Scott Determan
47ffc392d7 Work around gdb bug by changing a template parameter: (#4332)
There's a bug in gdb where unsigned template parameters cause issues with
RTTI. This patch changes a template parameter from `size_t` to `int` to
work around this gdb bug.
2022-12-13 16:13:54 -08:00
Alloy Networks
0362e935af Reduce default reserves to 10/2 (#4329)
Reduce the reserve requirements from 20/5 to 10/2 in line with the current network votes. The requirements of 10/2 have been on the network long enough that new nodes should not still have the old reserve amount.

Co-authored-by: Richard Holland <richard.holland@starstone.co.nz>
2022-12-13 10:56:35 -08:00
Ed Hennis
a79fa2026b Update GitHub Actions to ubuntu-latest: (#4357)
* Per actions/runner-images#6002, ubuntu-18.04 is being deprecated. If
  latest ever fails in the future, we'll need to fix the jobs anyway, so
  catch it early.

* Use long option names

* Force clang-format to ubuntu-20.04 because LLVM 10 is not available for 22.04
2022-12-08 20:41:50 -08:00
Gregory Popovitch
c1e7fe2d93 Prevent unnecessary shared_ptr copies by accepting a value in SHAMapInnerNode::setChild (#4266)
* Do a move instead of a copy in `SHAMapInnerNode::setChild`

* Create the value directly in the call
2022-12-08 20:15:21 -08:00
Nik Bougalis
4a5ad4cfac Update affiliation: Nikolaos D Bougalis (#4360) 2022-12-08 16:30:05 -08:00
Elliot Lee
ffd453f7dd Set version to 1.10.0-rc1 2022-11-28 14:12:16 -08:00
Scott Determan
518fb6d208 Improve move semantics in Expected (#4326)
* Improve move semantics in Expected:

This patch unconditionally moves an `Unexpected<U>` value parameter as
long as `U` is not a reference. If `U` is a reference the code should
not compile. An error type that holds a reference is a strange use-case,
and an overload is not provided. If it is required in the future it can
be added.

The `Expected(U r)` overload should take a forwarding ref.

* Replace enable_if with concepts in Expected
2022-11-28 14:01:54 -08:00
Elliot Lee
093055c039 README - add link to ripple-server mailing list (#3890) 2022-11-28 13:57:38 -08:00
Ed Hennis
b7ac73c8e4 Don't try to read SLE with key 0 from the ledger:
* May resolve #4341
2022-11-28 13:55:29 -08:00
J. Scott Branson
004ec2d201 Update Linux Build Instructions (#4018)
* Removed a reference to the default number of workers varying based on whether a node has validation enabled. Workers default to the number of processor cores + 2: https://github.com/ripple/rippled/blob/develop/src/ripple/core/impl/JobQueue.cpp#L166

* Protobuf v2 and Ubuntu 16.04 are no longer supported.

* Updated protobuf version as v3 is now supported, fixed typos, automatically sent number of processors when building boost & rippled.
2022-11-28 13:49:38 -08:00
manojsdoshi
ebbf4b64a5 Set version to 1.10.0-b2 2022-10-14 13:15:44 -07:00
Scott Schurr
649ab872ff Add featureImmediateOfferKilled for tfImmediateOrCancel offers:
Fixes #4115; https://github.com/ripple/rippled/issues/4115
2022-10-14 12:54:59 -07:00
Scott Schurr
7e9e9104ea Rename SHAMapStoreImp::stopping() to healthWait() 2022-10-14 12:54:59 -07:00
greg7mdp
3726f8bf31 Release TaggedCache object memory outside the lock 2022-10-14 12:54:59 -07:00
Ed Hennis
e37dc710cf Improve timing of NodeToShardRPC unit test to reduce false failures 2022-10-14 12:54:59 -07:00
Ed Hennis
5d38e4cfbf Work around soci build issues 2022-10-13 15:44:49 -07:00
CJ Cobb
28f4cc7817 Remove gRPC code previously used for the xpring SDK 2022-10-13 15:44:43 -07:00
Chenna Keshava B S
95fabd5762 Remove inacessible code paths and outdated data format wchar_t 2022-10-11 18:11:32 -07:00
Alexander Kremer
23ce431876 Remove const_cast usage 2022-10-11 16:38:10 -07:00
seelabs
04ef885108 Support for boost 1.80.0:
Boost intrusive renamed the `comp()` function -> `get_comp()`
2022-10-11 16:36:30 -07:00
manojsdoshi
d33df35378 Merge master (1.9.4) into develop (1.10.0-b2) 2022-10-11 16:34:36 -07:00
Nik Bougalis
ba3c0e5145 Set version to 1.9.4 2022-09-20 18:23:13 -07:00
Michael Legleux
be1ce5eca9 Pin postgres, zlib, krb5, libuv and cassandra to stable versions 2022-09-20 18:22:37 -07:00
Ikko Ashimine
df66e4151e Fix typo in detail/Node.h
minumum -> minimum
2022-09-14 14:17:30 -07:00
seelabs
9a31f321cd Allow gcc 12 compilation:
Compiling with gcc 12 on manjaro (arch variant) had compilation errors
without adding an additional include file.
2022-09-14 14:16:40 -07:00
Scott Schurr
e40e38e8d3 Introduce fixRemoveNFTokenAutoTrustLine amendment:
It turns out that the feature enabled by the tfTrustLine flag
on an NFTokenMint transaction could be used as a means to
attack the NFToken issuer.  Details are in
https://github.com/XRPLF/rippled/issues/4300

The fixRemoveNFTokenAutoTrustLine amendment removes the
ability to set the tfTrustLine flag on an NFTokenMint
transaction.

Closes 4300.
2022-09-14 14:15:20 -07:00
Howard Hinnant
f5af42a640 Remove use of deprecated std::iterator 2022-09-14 14:14:18 -07:00
Nik Bougalis
fe05b8c4fe Set version to 1.10.0-b1 2022-08-25 08:49:18 -07:00
Nik Bougalis
7b3507bb87 Improve wrapper around OpenSSL RAND 2022-08-25 08:49:17 -07:00
Nik Bougalis
0ecfc7cb1a Improve self-signed certificate generation:
When starting, the code generates a new ephemeral private key and
a corresponding certificate to go along with it. This process can
take time and, while this is unlikely to matter for normal server
operations, it can have a significant impact for unit testing and
development. Profiling data suggests that ~20% of the time needed
for a unit test run can be attributed to this.

This commit does several things:

1. It restructures the code so that a new self-signed certificate
   and its corresponding private key are only initialized once at
   startup; this has minimal impact on the operation of a regular
   server.
2. It provides new default DH parameters. This doesn't impact the
   security of the connection, but those who compile from scratch
   can generate new parameters if they so choose.
3. It properly sets the version number in the certificate, fixing
   issue #4007; thanks to @donovanhide for the report.
4. It uses SHA-256 instead of SHA-1 as the hash algorithm for the
   certificate and adds some X.509 extensions as well as a random
   128-bit serial number.
5. It rounds the certificate's "start of validity" period so that
   the server's precise startup time cannot be easily deduced and
   limits the validity period to two years, down from ten years.
6. It removes some CBC-based ciphers from the default cipher list
   to avoid some potential security issues, such as CVE-2016-2107
   and CVE-2013-0169.
2022-08-25 08:49:17 -07:00
Nik Bougalis
e2eed966b0 Improve AccountID string conversion caching:
Caching the base58check encoded version of an `AccountID` has
performance advantages, because because of the computationally
heavy cost associated with the conversion, which requires the
application of SHA-256 twice.

This commit makes the cache significantly more efficient in terms
of memory used: it eliminates the map, using a vector with a size
that is determined by the configured size of the node, and a hash
function to directly map any given `AccountID` to a specific slot
in the cache; the eviction policy is simple: in case of collision
the existing entry is removed and replaced with the new data.

Previously, use of the cache was optional and required additional
effort by the programmer. Now the cache is automatic and does not
require any additional work or information.

The new cache also utilizes a 64-way spinlock, to help reduce any
contention that the pressure on the cache would impose.
2022-08-25 08:49:16 -07:00
Nik Bougalis
5a15229eeb Improve detection & handling of duplicate Node ID:
Each node on the network is supposed to have a unique cryptographic
identity. Typically, this identity is generated randomly at startup
and stored for later reuse in the (poorly named) file `wallet.db`.

If the file is copied, it is possible for two nodes to share the
same node identity. This is generally not desirable and existing
servers will detect and reject connections to other servers that
have the same key.

This commit achives three things:

1. It improves the detection code to pinpoint instances where two
   distinct servers with the same key connect with each other. In
   that case, servers will log an appropriate error and shut down
   pending intervention by the server's operator.
2. It makes it possible for server administrators to securely and
   easily generate new cryptographic identities for servers using
   the new `--newnodeid` command line arguments. When a server is
   started using this command, it will generate and save a random
   secure identity.
3. It makes it possible to configure the identity using a command
   line option, which makes it possible to derive it from data or
   parameters associated with the container or hardware where the
   instance is running by passing the `--nodeid` option, followed
   by a single argument identifying the infomation from which the
   node's identity is derived. For example, the following command
   will result in nodes with different hostnames having different
   node identities: `rippled --nodeid $HOSTNAME`

The last option is particularly useful for automated cloud-based
deployments that minimize the need for storing state and provide
unique deployment identifiers.

**Important note for server operators:**
Depending on variables outside of the the control of this code,
such as operating system version or configuration, permissions,
and more, it may be possible for other users or programs to be
able to access the command line arguments of other processes
on the system.

If you are operating in a shared environment, you should avoid
using this option, preferring instead to use the `[node_seed]`
option in the configuration file, and use permissions to limit
exposure of the node seed.

A user who gains access to the value used to derive the node's
unique identity could impersonate that node.

The commit also updates the minimum supported server protocol
version to `XRPL/2.1`, which has been supported since version
1.5.0 and eliminates support for `XPRL/2.0`.
2022-08-25 08:49:14 -07:00
Nik Bougalis
d318ab612a Avoid unnecessary copying and dynamic memory allocations
Co-authored-by: Chenna Keshava B S <ckbs.keshava56@gmail.com>
2022-08-25 08:49:14 -07:00
Chenna Keshava B S
9aaa0dff5f Build the command map at compile time (fixes #3298):
We profiled different algorithms and data structures to understand which
strategy is best from a performance standpoint:

- Linear search on an array;
- Binary search on a sorted array;
- Using `std::map`; and
- Using `std::unordered_map`.

Both linear search and std::unordered_map outperformed the other alternatives
so no change to the existing data structure is justified. If more handers are
added, this should be revisited.

For some additional details and timings, please see:
https://github.com/XRPLF/rippled/issues/3298#issuecomment-1185946010
2022-08-25 08:49:04 -07:00
Chenna Keshava B S
b88ed5a8ec Update command-line usage help message (fixes #3318) 2022-08-25 08:49:04 -07:00
Chenna Keshava B S
ce64f7a90f Remove deprecated AccountTxOld.cpp (fixes #2926) 2022-08-25 08:49:04 -07:00
Chenna Keshava B S
5e1cb09b88 Update broken link to hosted Doxygen content (fixes #4251) 2022-08-25 08:49:04 -07:00
Howard Hinnant
3d0c14f3e3 Remove use of deprecated std::iterator 2022-08-25 08:49:04 -07:00
Crypto Brad Garlinghouse
83ac141f65 Remove charUnHex 2022-08-25 08:49:04 -07:00
Crypto Brad Garlinghouse
e67f90588a Use constexpr to check memo validity 2022-08-25 08:38:28 -07:00
Crypto Brad Garlinghouse
69bb2be446 Introduce amendment to handle trustlines to self:
Trustlines must be between two different accounts but two trustlines exist
where an account extends trust to itself. They were created in the early
days, likely because of bugs that have been fixed. The new fixTrustLinesToSelf
amendment will remove those trustlines when it activates.
2022-08-25 08:38:28 -07:00
seelabs
92d35e54c7 Switch from C++17 to C++20 2022-08-25 08:38:28 -07:00
Nik Bougalis
47dec467ea Set version to 1.9.3 2022-08-24 19:41:07 -07:00
Wo Jake
cd3a6bf530 Document the "DefaultVote::no" policy in the code 2022-08-19 14:57:38 -07:00
Crypto Brad Garlinghouse
6a9c270776 Properly handle self-assignment of PublicKey 2022-08-19 14:27:33 -07:00
Howard Hinnant
21a3f4a5b5 Change by-value to by-reference to persist vote 2022-08-19 14:26:59 -07:00
manojsdoshi
e5275b8577 Set version to 1.9.2 2022-07-25 16:48:36 -07:00
Nik Bougalis
83faf43140 Set version to 1.9.2-rc1 2022-07-18 13:58:48 -07:00
Nik Bougalis
22b4de2e44 Fix a race condition during shutdown 2022-07-18 13:58:47 -07:00
Nik Bougalis
b95ca98965 Fix #4231 (wrong include guard) and an out-of-order construction warning 2022-07-17 22:17:34 -07:00
Nik Bougalis
7e46f5342b Correct a technical flaw with the spinlock locking:
The existing spinlock code, used to protect SHAMapInnerNode
child lists, has a mistake that can allow the same child to
be repeatedly locked under some circumstances.

The bug was in the `SpinBitLock::lock` loop condition check
and would result in the loop terminating early.

This commit fixes this and further simplifies the lock loop
making the correctness of the code easier to verify without
sacrificing performance.

It also promotes the spinlock class from an implementation
detail to a more general purpose, easier to use lock class
with clearer semantics. Two different lock types now allow
developers to easily grab either a single spinlock from an
a group of spinlocks (packed in an unsigned integer) or to
grab all of the spinlocks at once.

While this commit makes spinlocks more widely available to
developers, they are rarely the best tool for the job. Use
them judiciously and only after careful consideration.
2022-07-17 22:17:34 -07:00
Nik Bougalis
59326bbbc5 Introduce the NonFungibleTokensV1_1 amendment:
The XLS-20 implementation contained two bugs that would require the
introduction of amendments. This complicates the adoption of XLS-20
by requiring a staggered amendment activation, first of the two fix
amendments, followed by the `NonFungibleTokensV1` amendment.

After consideration, the consensus among node operators is that the
process should be simplified by the introduction of a new amendment
that, if enabled, would behaves as if the `NonFungibleTokensV1` and
the two fix amendments (`fixNFTokenDirV1` and `fixNFTokenNegOffer`)
were activated at once.

This commit implements this proposal; it does not introduce any new
functionality or additional features, above and beyond that offered
by the existing amendments.
2022-07-17 22:17:33 -07:00
Nik Bougalis
9eb303f8e8 Improve STVector256 deserialization 2022-07-17 22:17:33 -07:00
Nik Bougalis
47ccd0b579 Limit how often endpoint messages can be processed:
The peer discovery protocol depends on peers exchanging messages
listing IP addresses for other peers.

Under normal circumstances, these messages should not be sent
frequently; the existing code would track the earliest time a
new message should be processed, but did not actually enforce
that limit.
2022-07-17 22:17:32 -07:00
Nik Bougalis
5e6728dccd Set cluster timer only when in a cluster 2022-07-17 22:17:31 -07:00
Nathan Nichols
d458e9972b Improve JSON sanitization in reporting mode 2022-07-17 22:17:24 -07:00
Howard Hinnant
b0b44d32bd Fix amendment voting persistence:
An incorrect SQL query could cause the server to improperly
configure its voting state after a restart; typically, this
would manifest as an apparent failure to store a vote which
the administrator of the server had configured.

This commit fixes the broken SQL and ensures that amendment
votes are properly reloaded post-restart and closes #4220.
2022-07-17 22:17:24 -07:00
Scott Schurr
8266d9d598 Correct a technical flaw with NFT offers:
The existing code would, incorrectly, allow negative amounts in offers
for non-fungible tokens. Such offers would be handled very differently
depending on the context: a direct offer would fail with an error code
indicating an internal processing error, whereas brokered offers would
improperly succeed.

This commit introduces the `fixNFTokenNegOffer` amendment that detects
such offers during creation and returns an appropriate error code.

The commit also extends the existing code to allow for buy offers that
contain a `Destination` field, so that a specific broker can be set in
the offer.
2022-07-17 22:17:24 -07:00
Scott Schurr
0839a202c9 Reduce console noise coming from unit tests:
A few unit tests have historically generated a lot of noise
to the console from log writes.  This noise was not useful
and made it harder to locate actual test failures.

By changing the log level of these tests from
- severities::kError to
- severities::kDisabled
it was possible to remove that noise coming from the logs.
2022-07-17 22:17:24 -07:00
Ed Hennis
ee60b16b3a Lower log level of "addPathsForType" log message (fixes #4177) 2022-07-17 22:17:24 -07:00
Ed Hennis
18d437284e Lower the message level for missing optional doc components 2022-07-17 22:17:24 -07:00
Ed Hennis
1f75ba23ee Fix bitwise or on boolean operands warning / error 2022-07-17 22:17:24 -07:00
seelabs
723733a778 Catch missing node error when rotating database:
While there should never be a missing node when copying the SHAMap,
rippled should not terminate when there's an error rotating the
database. This patch aborts the database rotation rather than aborting rippled.
2022-07-17 22:17:24 -07:00
seelabs
8e6a0d418c Fix race conditions in shard:
ThreadSafetyAnalysis was used to identify race conditions in this file.
This analysis was modivated by a (rare) crash while running unit tests.

Add locks to Shard flagged by ThreadSafetyAnalysis
2022-07-17 22:17:24 -07:00
seelabs
f55913dcee Add support for clang's ThreadSafetyAnalysis 2022-07-17 22:17:24 -07:00
Chenna Keshava B S
e46d2bcf27 Correctly use the configured network_id parameter:
The existing code properly parses the network_id parameter from the
the configuration file, but it does not properly set up the code to
use the value correctly. As a result the configured `network_id` is
ignored.
2022-07-17 22:17:24 -07:00
Chenna Keshava B S
d632f9f6c8 Properly handle incorrect port numbers in parseURL (fixes #4200) 2022-07-11 09:56:39 -07:00
Chenna Keshava B S
3172a816fa Describe resolution for common SOCI-related build errors 2022-07-11 09:56:39 -07:00
Ikko Ashimine
4e724794c5 Fix typo in Consensus.h (#4179)
whenver -> whenever
2022-07-11 14:03:26 +02:00
RichardAH
610436d737 Add book_changes rpc (#4212) 2022-07-11 09:29:31 +02:00
RichardAH
0ee6f15b35 Create CONTRIBUTING.md, list maintainers and outline repo policies (#4214) 2022-07-09 20:37:13 +03:00
manojsdoshi
e32bc674aa Set version to 1.9.1 2022-05-22 13:18:18 -07:00
manojsdoshi
34786abd4f Set version to 1.9.1-rc2 2022-05-19 15:03:26 -07:00
natenichols
d0a813a19d Adding missing include <cassert> in beast/rngfill.h to fix rippled build 2022-05-19 15:02:28 -07:00
manojsdoshi
25474343a9 Set version to 1.9.1-rc1 2022-05-11 18:21:14 -07:00
Nik Bougalis
670bc22cfa Set version to 1.9.1-b1 2022-05-10 13:34:16 -07:00
Scott Schurr
80bda7cc48 Introduce fixNFTokenDirV1 amendment:
o Fixes an off-by-one when determining which NFTokenPage an
  NFToken belongs on.
o Improves handling of packed sets of 32 NFTs with
  identical low 96-bits.
o Fixes marker handling by the account_nfts RPC command.
o Tightens constraints of NFTokenPage invariant checks.

Adds unit tests to exercise the fixed cases as well as tests
for previously untested functionality.
2022-05-10 13:34:12 -07:00
Devon White
dac080f1c8 Advance ripple.app.rdb 2022-05-10 13:34:12 -07:00
Devon White
767dd4ff3f Address failures in DatabaseDownloader unittest 2022-05-10 13:34:12 -07:00
Richard Holland
01c37fed69 Introduce the ExpandedSignerList amendment:
The amendment increases the maximum sign of an account's signer
list from 8 to 32.

Like all new features, the associated amendment is configured with
a default vote of "no" and server operators will have to vote for
it explicitly if they believe it is useful.
2022-05-10 13:34:12 -07:00
Ed Hennis
04bd5878f1 Track total trustlines and avoid duplications 2022-05-10 13:34:12 -07:00
Ed Hennis
e836375d99 Don't load trust lines that can't participate in path finding
* "A path is considered invalid if and only if it enters and exits an
  address node through trust lines where No Ripple has been enabled for
  that address." (https://xrpl.org/rippling.html#specifics)
* When loading trust lines for an account "Alice" which was reached
  via a trust line that has the No Ripple flag set on Alice's side, do
  not use or cache any of Alice's trust lines which have the No Ripple
  flag set on Alice's side. For typical "end-user" accounts, this will
  return no trust lines.
2022-05-10 13:34:12 -07:00
Scott Schurr
aa4a5b7fe9 Adjust log levels to reflect actual severity (fixes #3996) 2022-05-10 13:34:12 -07:00
Mark Travis
5aedb0e07a Allow server to stabilize after online delete health check failure 2022-05-10 13:34:12 -07:00
greg7mdp
dfe69f1b76 Fix strict aliasing test failures with newer MSVC compilers 2022-05-10 13:34:12 -07:00
greg7mdp
87d06a2571 Check triggers cmake error unnecessarily 2022-05-10 13:34:12 -07:00
Nik Bougalis
7ca1f78446 Ensure that rngfill returns the requested amount of randomness:
One of the two versions of the `rngfill` function accepts a pointer
to a buffer and a size (in bytes). The function aims to fill the
provided `buffer` with `size` random bytes. It does this in chunks
of 8 bytes, for long as possible, and then fills any left-over gap
one byte at a time.

To avoid an annoying and incorrect warning about a potential buffer
overflow in the "trailing write", commit 78bc2727f7
used a `#pragma` to instruct the compiler to not generate the incorrect
diagnostic. Unfortunately, this change _also_ eliminated the trailing
write code, which means that, under some cases, the `rngfill` function
would generate between 1 and 7 fewer random bytes than requested.

This problem would only manifest on builds that do not define `__GNUC__`
which, as of this writing, means MSVC.
2022-05-10 13:34:12 -07:00
Nik Bougalis
b68a66928c Adjust the aggressiveness of the ledger acquisition engine:
Several hard-coded parameters control the behavior of the ledger
acquisition engine. The values of many of these parameters where
set by intuition and have complex and non-intuitive interactions
with each other and other parts of the code.

An earlier commit attempted to adjust several of these parameters
to improve syncing performance; initial testing was promising but
a number of operators reported experiencing syncing and stability
issues with their servers. As a result, this commit reverts parts
of commit 18235067af.

This commit further adjusts some tunables so as to increase the
aggressiveness of the ledger acquisition engine.
2022-05-06 11:38:19 -07:00
Nik Bougalis
245174c42c Adjust thread count calculation and request bundling:
This commit addresses minor bugs introduced with commit
6faaa91850:

- The number of threads used by the database engine was
  incorrectly clamped to the lower possible value, such
  that the database was effectively operating in single
  threaded mode.

- The number of requests to extract at once was so high
  that it could result in increased latency. The bundle
  size is now limited to 4 and can be adjusted by a new
  configuration option `rq_bundle` in the `[node_db]`
  stanza. This is an advanced tunable and adjusting it
  should not be needed.
2022-05-06 11:38:18 -07:00
manojsdoshi
7c66747d27 Set version to 1.9.0 2022-04-06 20:28:46 -07:00
manojsdoshi
cdd37a2a05 Set version to 1.9.0-rc1 2022-04-06 16:35:48 -07:00
manojsdoshi
c66be3e6cf Set version to 1.9.0-b3 2022-04-06 13:30:56 -07:00
Nik Bougalis
70779f6850 Introduce NFT support (XLS020) 2022-04-06 13:29:48 -07:00
greg7mdp
525aaecbca Fix build issue on M1 macs 2022-04-06 13:29:39 -07:00
Michael Legleux
9d3cd718e4 Do not install coreutils-single on rocky build image 2022-04-06 13:29:00 -07:00
Michael Legleux
656e9fe180 Upload rippled-reporting packages to Artifactory 2022-04-06 13:28:47 -07:00
Michael Legleux
8aa617d972 Fix failing Clio build 2022-04-06 13:28:10 -07:00
Nik Bougalis
711608e652 Set version to 1.9.0-b2 2022-03-30 15:16:40 -07:00
Ed Hennis
bc9773eb45 MSVC std::sample workaround 2022-03-30 15:16:27 -07:00
Michael Legleux
bea9610440 Remove checks for CMake < 3.16 2022-03-30 15:16:27 -07:00
Michael Legleux
375af87a86 Package builds of reporting mode rippled 2022-03-30 15:16:27 -07:00
Michael Legleux
1502e6e2cd Reduce the verbosity of CI scripts 2022-03-30 15:16:27 -07:00
Nik Bougalis
593677ee82 Squash warning about an unused variable 2022-03-30 15:16:27 -07:00
Nik Bougalis
8f58687091 Cleanup AcceptedLedger and AcceptedLedgerTx:
This commit modernizes the `AcceptedLedger` and `AcceptedLedgerTx`
classes, reduces their memory footprint and reduces unnecessary
dynamic memory allocations.
2022-03-30 15:16:27 -07:00
Nik Bougalis
c7e6803956 Simplify LedgerMaster:
- Eliminate `tune` member function and allow `LedgerHistory`
  to fully initialize itself.
2022-03-30 15:16:27 -07:00
Nik Bougalis
6faaa91850 Improve asynchronous database handlers:
This commit optimizes the way asynchronous nodestore operations are
processed both by reducing the amount of time locks are held and by
minimizing the number of memory allocations and data copying.
2022-03-30 15:16:27 -07:00
Nik Bougalis
d66d960d59 Miscellaneous Improvements:
- Adjust default tree cache sizing
- Various micro-optimizations
2022-03-30 15:16:26 -07:00
Nik Bougalis
18235067af Simplify and optimize the processing of inbound transactions 2022-03-30 15:16:26 -07:00
JoelKatz
3eb8aa8b80 Negative cache support for node store
Adds support to TaggedCache to support smart replacement
(Needed to avoid race conditions with negative caching.)

Create a "hotDUMMY" object that represents the absence
of an object.

Allow DatabaseNodeImp::asyncFetch to complete immediately
if object is in cache (positive or negative).

Fix a bug in asyncFetch where the object returned may not
be the correct canonical version because we stash the
object in the results array before we canonicalize it.
2022-03-30 15:16:12 -07:00
Nik Bougalis
b9903bbcc4 Simplify and improve order book tracking:
- Avoid using std::shared_ptr
- Prefer using unordered maps to avoid linear searches
- Increase the interval between full order book updates
2022-03-29 16:16:34 -07:00
Nik Bougalis
48803a48af Improve ledger-fetching logic:
When fetching ledgers, the existing code would isolate the peer
that sent the most useful responses and issue follow up queries
only to that peer.

This commit increases the query aggressiveness, and changes the
mechanism used to select which peers to issue follow-up queries
to so as to more evenly spread the load along those peers which
provided useful responses.
2022-03-29 16:16:34 -07:00
Nik Bougalis
1b9387eddc Eliminate SHAMapInnerNode lock contention:
The `SHAMapInnerNode` class had a global mutex to protect the
array of node children. Profiling suggested that around 4% of
all attempts to lock the global would block.

This commit removes that global mutex, and replaces it with a
new per-node 16-way spinlock (implemented so as not to effect
the size of an inner node objet), effectively eliminating the
lock contention.
2022-03-29 16:16:34 -07:00
Nik Bougalis
34ca457132 Improve stop signaling for Application 2022-03-29 16:15:42 -07:00
Edward Hennis
df60e46750 Improve deterministic transaction sorting in TxQ:
* Txs with the same fee level will sort by TxID XORed with the parent
  ledger hash.
* The TxQ is re-sorted after every ledger.
* Attempt to future-proof the TxQ tie breaking test
2022-03-23 23:28:04 -07:00
Edward Hennis
e7e672c3f8 Incremental improvements to path finding memory usage:
* Abort background path finding when closed or disconnected
* Exit pathfinding job thread if there are no requests left
* Don't bother creating the path find job if there are no requests
* Refactor to remove circular dependency between InfoSub and PathRequest
2022-03-23 23:28:04 -07:00
seelabs
4d5459d041 Optimize trust line caching:
The existing trust line caching code was suboptimal in that it stored
redundant information, pinned SLEs into memory and required multiple
memory allocations per cached object.

This commit eliminates redundant data, reducing the size of cached
objects and unpinning SLEs from memory, and uses value_types to
avoid the need for `std::shared_ptr`. As a result of these changes, the
effective size of a cached object, includes the overhead of the memory
allocator and the `std::shared_ptr` should be reduced by at least 64
bytes. This is significant, as there can easily be tens of millions
of these objects.
2022-03-23 23:28:04 -07:00
seelabs
59f5844381 Reduce lock contention in manifest cache:
This commit combines the `apply_mutex` and `read_mutex` into a single `mutex_`
var. This new `mutex_` var is a `shared_mutex`, and most operations only need to
lock it with a `shared_lock`. The only exception is `applyMutex`, which may need
a `unique_lock`.

One consequence of removing the `apply_mutex` is more than one `applyMutex`
function can run at the same time. To help reduce lock contention that a
`unique_lock` would cause, checks that only require reading data are run a
`shared_lock` (call these the "prewriteChecks"), then the lock is released, then
a `unique_lock` is acquired. Since a currently running `applyManifest` may write
data between the time a `shared_lock` is released and the `write_lock` is
acquired, the "prewriteChecks" need to be rerun. Duplicating this work isn't
ideal, but the "prewirteChecks" are relatively inexpensive.

A couple of other designs were considered. We could restrict more than one
`applyMutex` function from running concurrently - either with a `applyMutex` or
my setting the max number of manifest jobs on the job queue to one. The biggest
issue with this is if any other function ever adds a write lock for any reason,
`applyManifest` would not be broken - data could be written between the release
of the `shared_lock` and the acquisition of the `unique_lock`. Note: it is
tempting to solve this problem by not releasing the `shared_mutex` and simply
upgrading the lock. In the presence of concurrently running `applyManifest`
functions, this will deadlock (both function need to wait for the other to
release their read locks before they can acquire a write lock).
2022-03-23 23:28:04 -07:00
Richard Holland
a07a729e3d Reserve field codes for Hooks:
In order to preserve the Hooks ABI, it is important that field
values used for hooks be stable going forward.

This commit reserves the required codes so that they will not
be repurposed before Hooks can be proposed for inclusion in
the codebase.
2022-03-23 10:48:39 -07:00
Rome Reginelli
b65e279db6 Remove Xpring forum link from issue configs
The Xpring Forum shut down a while back, so it's no longer a relevant link for community discussions.
2022-03-23 10:48:39 -07:00
Steven Olmsted
1ddc966b31 Fix typo 2022-03-23 10:48:39 -07:00
Nik Bougalis
1a8eb5e6e3 Set version to 1.9.0-b1 2022-03-09 15:52:30 -08:00
Michael Legleux
6a8180c967 Use 3.16 as min cmake 2022-03-09 15:51:39 -08:00
Michael Legleux
eb57679085 Update cmake_minimum_required to 3.16, Ubuntu 20.04's version 2022-03-04 14:27:25 -08:00
Michael Legleux
297def5ed3 Update boost max to 1.77 2022-03-04 14:27:16 -08:00
Michael Legleux
a01cadbfd5 Move Beast & fix #include paths 2022-03-04 14:16:59 -08:00
CJ Cobb
11ca9a946c Add successor information to clio ETL messages
* Allow clio to ask for object successors and predecessors from rippled
* Add lower_bound and last_below to SHAMap
2022-03-04 13:38:48 -08:00
CJ Cobb
f326f019bf force build with c++17 2022-03-04 13:38:48 -08:00
natenichols
90326bf756 Proxy validation_quorum when in reporting mode 2022-03-04 13:38:48 -08:00
Edward Hennis
255bf829ca Fix Travis CI MacOS builds:
* Update boost version.
* Use latest macOS image.
* Credit to @donovanhide in #4025 for starting this ball rolling, and
  inspiring a boost update.
2022-03-04 13:38:48 -08:00
Edward Hennis
0623a40f02 Refactor to fix levelization:
* Remove Application & Database dependency in PerfLog. Replace it with
  a callback passed into the constructor.
* Fixes the circular dependency between ripple/nodestore and ripple/basics
2022-03-01 14:32:14 -08:00
Ikko Ashimine
a529b218f3 Fix typo in ReportingETL.cpp
respresent -> represent
2022-03-01 14:32:14 -08:00
Mark Travis
c0cb389b20 Fallback to normal sync if fast loading is not possible:
If fast loading is enabled but the last persisted ledger is not
entirely on disk, the server would fail to start without manual
intervention by the server operator.

This commit allows the server to detect this scenario and attempt
to automatically recover.
2022-03-01 11:29:07 -08:00
Mark Travis
8f82b62e0d Use CIDR notation for admin and secure_gateway 2022-03-01 11:27:50 -08:00
Mark Travis
dc213a4fab Make gateway_balances admin-only in reporting mode 2022-03-01 11:27:34 -08:00
Mark Travis
06e87e0f6a Fix deletion of orphan nodestore directories:
Orphaned nodestore directories should only be deleted
if the proper nodestore directories are confirmed to
exist.
2022-03-01 11:25:03 -08:00
John Freeman
c2a08a1f26 Simplify the Job Queue:
This is a refactor aimed at cleaning up and simplifying the existing
job queue.

As of now, all jobs are cancelled at the same time and in the same
way, so this commit removes the per-job cancellation token. If the
need for such support is demonstrated, support can be re-added.

* Revise documentation for ClosureCounter and Workers.
* Simplify code, removing unnecessary function arguments and
  deduplicating expressions
* Restructure job handlers to no longer need to pass a job's
  handle to the job.
2022-03-01 11:25:03 -08:00
Scott Schurr
df02eb125f Prefer using a local Sandbox over using the Context's view 2022-03-01 11:25:03 -08:00
Howard Hinnant
0c13676d5f Restrict access to the virtual functions move and copy
* These are meant to be used by detail::STVar only and
  are otherwise error-prone to call.
2022-03-01 11:25:03 -08:00
Howard Hinnant
74e6ed1af3 Improve the readability of STBase-derived types
* Increase the visibility of each type's API.
* No functional changes.
2022-03-01 11:25:03 -08:00
Nik Bougalis
72377e7bf2 Set version to 1.8.5 2022-02-07 16:36:55 -08:00
CJ Cobb
5b085a75fd Avoid dereferencing empty optional in ReportingETL 2022-02-07 12:38:50 -08:00
seelabs
61389a8bef Correct minor bug in TaggedPointer move constructor:
A typographical error would mishandle the case where a caller explicitly
tries to remove a child that is not actually part of the node. This case
is never invoked in practice, and so the bug would will never trigger.
2022-02-07 12:38:50 -08:00
Wo Jake
bd97e59254 Update README.md
XRPL Foundation's basic introduction to the XRPL, a great series that would *hopefully* continue...
2022-02-07 12:38:50 -08:00
Michael Legleux
95ecf296ad Install the protocol buffer deps during smoketests 2022-02-07 12:38:40 -08:00
Michael Legleux
b7e0306d0a Ensure protocol buffer prerequisites are present 2022-02-07 10:59:12 -08:00
Michael Legleux
a9ee802240 Update smoke test distributions 2022-02-07 10:59:12 -08:00
Edward Hennis
d23d37fcfd Correctly add GIT_COMMIT_HASH into version string:
Commit bf013c02ad added support
for incorporating a commit ID into the compiled version string
but did so in a way that did not follow the semantic versioning
standard.

This commit corrects that flaw by moving the commit ID into the
"metadata" part of the version string and properly handles the
case where the commit hash cannot be retrieved.
2022-02-07 10:52:31 -08:00
Nik Bougalis
289bc0afd9 Improve handling of endpoints during peer discovery 2022-02-07 10:33:02 -08:00
Nik Bougalis
c5dc00af74 Update RocksDB to version 6.27.3 2022-02-07 10:32:59 -08:00
Nik Bougalis
d49b486224 Set version to 1.8.4 2022-01-12 18:54:06 -08:00
Nik Bougalis
417cfc2fb0 Adjust pathfinding configuration defaults:
The pathfinding engine built into the code has several configurable
parameters to adjust the depth of the paths indexed and explored.

These parameters can dramatically impact the performance and memory
consumption of a server; higher values can result in resource usage
increasing exponentially.

These default values were decided early and somewhat arbitrarily at
a time when the network and the size of the network state were much
smaller.

This commit adjusts the default values to reduce the depth of paths
to more reasonable levels; unless explicitly overriden, the changes
mean that pathfinding operations will return fewer, shallower paths
than previous versions of the software.
2022-01-12 18:54:03 -08:00
seelabs
febbe14e6d Adjust mutex scope in walkMapParallel:
This commit corrects a technical flaw that was introduced with commit
7c12f01358: as written, a mutex that is
intended to help provide synchronization for multiple threads as they
are each walking the map, is declared so that each thread is passed a
dangling reference to a unique mutex.

This commit hoists the mutex outside the thread creation loop, so all
threads use a single mutex and eliminating the dangling reference.
2022-01-12 14:21:44 -08:00
Nik Bougalis
44514930f9 Set version to 1.8.3 2022-01-10 15:40:07 -08:00
Nik Bougalis
dc778536ed Provide sensible default values for nodestore cache:
The nodestore includes a built-in cache to reduce the disk I/O
load but, by default, this cache was not initialized unless it
was explicitly configured by the server operator.

This commit introduces sensible defaults based on the server's
configured node size.

It remains possible to completely disable the cache if desired
by explicitly configuring it the cache size and age parameters
to 0:

    [node_db]
    ...
    cache_size = 0
    cache_age = 0
2022-01-10 15:29:44 -08:00
Nik Bougalis
18584ef2fd Adjust number of concurrent ledger data jobs 2022-01-10 15:29:43 -08:00
Nik Bougalis
416ce35d73 Detect CVE-2021-3520 when decompressing using LZ4 2022-01-10 15:29:43 -08:00
Mark Travis
7c12f01358 Parallel ledger loader & I/O performance improvements:
- Only duplicate records from archive to writable during online_delete.
- Log duration of nodestore reads.
- Include nodestore counters in perf_log output.
- Remove gratuitous nodestore activity counting.
- Report initial sync duration in server_info and perfLog.
- Report state_accounting in perfLog.
- Make state_accounting durations more accurate.
- Parallel ledger loader.
- Config parameter to load ledgers on start.
2022-01-10 15:29:21 -08:00
JoelKatz
5a4654a0da Two small SHAMapSync improvements:
1) Don't acquire so many nodes per pass. It's likely
far more than we need.

2) Right-size the finishedReads_ vector on passes other
than just the first.
2022-01-07 04:41:58 -08:00
manojsdoshi
89766c5f21 Set version to 1.8.2 2021-12-20 14:50:58 -08:00
manojsdoshi
4ec11e692b Set version to 1.8.2-rc1 2021-12-16 23:23:25 -08:00
Michael Legleux
d02f0e11c5 Quiet builds and disable unity build 2021-12-16 22:45:38 -08:00
natenichols
e28989638d Enforce account RPC limits by account objects traversed 2021-12-16 11:57:08 -08:00
CJ Cobb
915fe31274 log request and duration for every RPC call 2021-12-15 12:12:28 -08:00
Mark Travis
db720a59e4 Log resource limit disconnections. 2021-12-15 11:26:13 -08:00
CJ Cobb
72752b1ee0 make cassandra io threads configurable 2021-12-15 11:25:32 -08:00
Mark Travis
c663f1f62b Make tx() function against a read-only postgres instance. 2021-12-15 11:25:10 -08:00
Scott Schurr
d54f6278bb Improve names returned by server_info counters 2021-12-15 11:21:51 -08:00
Mark Travis
fc04336caa Reporting mode always returns age in server_info. 2021-12-15 11:18:58 -08:00
CJ Cobb
47376a0cc3 Only forward to p2p nodes that are in sync 2021-12-15 11:18:07 -08:00
Nik Bougalis
45aa0142a6 Improve full & compressed inner node deserialization 2021-12-14 17:49:20 -08:00
Edward Hennis
e3acb61d57 Increase TxQ default minimum and target sizes 2021-12-14 17:47:48 -08:00
Edward Hennis
8fa33795a3 Reduce TxQ logging severity in several places 2021-12-14 17:47:48 -08:00
Edward Hennis
b1c9b134dc Make transaction queue order deterministic:
* Sort by fee level (which is the current behavior) then by transaction
  ID (hash).
* Edge case when the account at the end of the queue submits a higher
  paying transaction to walk backwards and compare against the cheapest
  transaction from a different account.
* Use std::if_any to simplify the JobQueue::isOverloaded loop.
2021-12-14 17:47:48 -08:00
Edward Hennis
ae9930b87d Consensus transaction recovery/deferral completely ignores the TxQ 2021-12-14 17:47:38 -08:00
Edward Hennis
aaa601841c Logging & minor optimizations:
* Log load fee values (at debug) received from validations.
* Log remote and cluster fee values (at trace) when changed.
* Refactor JobQueue::isOverloaded to return sooner if overloaded.
* Refactor Transactor::checkFee to only compute fee if ledger is open.
2021-12-14 17:47:38 -08:00
Scott Schurr
8ca2d98496 NuDBBackend destructor should not throw 2021-12-14 17:47:38 -08:00
Scott Schurr
ad805eb95b Improve handling of the X-Forwarded-For and Forwarded HTTP headers 2021-12-14 17:47:04 -08:00
Wo Jake
eb17325cbe Add load cost 2021-12-14 17:47:04 -08:00
wojak
b00787e161 Update README.md 2021-12-14 17:47:04 -08:00
Ikko Ashimine
81e7ec859d Fix typo in consensus.md
determing -> determining
2021-12-14 17:47:04 -08:00
Ikko Ashimine
6f6179abb4 Fix typo in RFC1751.cpp
seperated -> separated
2021-12-14 17:47:04 -08:00
Howard Hinnant
32a26a65d9 Make basic_seconds_clock::time_point atomic
* Just the rep is made atomic to workaround older compilers
2021-12-14 17:47:04 -08:00
Richard Holland
daccb5b4c0 Add ignore_default option to account_lines API:
This flag, if present, suppresses the output of incoming
trustlines in default state.

This is primarily motivated by observing that users of Xumm often
have many unwanted incoming trustlines in a default state, which are
not useful in the vast majority of cases.

Being able to suppress those when doing `account_lines` saves bandwidth
and resources.
2021-12-14 17:44:24 -08:00
Richard Holland
cf97dcb992 Make I/O and prefetch worker threads configurable 2021-12-14 17:43:50 -08:00
Richard Holland
6746b863b3 Configurable handling of untrusted validations and proposals 2021-12-14 17:43:04 -08:00
Richard Holland
bf013c02ad Add git commit hash to debug version string 2021-12-14 17:42:59 -08:00
manojsdoshi
fbedfb25ae Set version to 1.8.1 2021-11-24 10:32:37 -08:00
Nik Bougalis
9e877a929e Adjust RocksDB defaults 2021-11-24 10:29:49 -08:00
manojsdoshi
e0eae9725b Set version to 1.8.0 2021-11-22 18:44:47 -08:00
manojsdoshi
3083983fee Set version to 1.8.0-rc3 2021-11-22 18:42:33 -08:00
Edward Hennis
5050b366d9 Return correct error code for node_to_shard command errors 2021-11-22 18:40:32 -08:00
seelabs
f0c237e001 Set version to 1.8.0-rc2 2021-11-18 14:39:03 -05:00
CJ Cobb
970711f1fd Properly handle empty optionals in DB lookups 2021-11-18 14:38:42 -05:00
Mark Travis
19018e8959 Introduce partitioned unordered maps:
This commit implements partitioned unordered maps and makes it possible
to traverse such a map in parallel, allowing for more efficient use of
CPU resources.

The `CachedSLEs`, `TaggedCache`, and `KeyCache` classes make use of the
new functionality, which should improve performance.
2021-11-18 14:38:35 -05:00
Nik Bougalis
7edfbbd8bd Disable pathfinding indexing on validator nodes:
The pathfinding engine requires pre-building large tables which is a
resource-intensive operation. Typically, one would not expect that a
server configured as a validator would also support pathfinding APIs
and so, building those tables by default wastes resources.

This commit, if merged, will disable pathfinding on servers that are
configured as validators, unless the server operator opts to support
it explicitly, by configuring the `[path_search_max]` parameter.

Validator operators that wish to support pathfinding on a validator
and want to use the default values can add the following stanza to
their server's configuration file:

    [path_search_max]
    7
2021-11-17 20:52:18 -08:00
Nik Bougalis
d36024394d Report job queue data if a deadlock is detected 2021-11-17 20:52:12 -08:00
Nik Bougalis
35e0ab4280 Adjust verbosity of log messages 2021-11-17 20:52:12 -08:00
Nik Bougalis
ef60ac8348 Properly handle oversized ledger data replies 2021-11-17 20:52:11 -08:00
Nik Bougalis
0c47cfad6f Adjust priorities of jobs:
The priority of different types of jobs was set back in the early
days of development, based on insight and observations that don't
necessarily apply any longer.

Specifically, job types used by the server to sync to the network
were being treated as lower priority than client requests, making
it more difficult to regain sync.

This commit adjusts the priority of several jobs and should allow
servers to prioritize resynchronizing to the network over serving
clients.
2021-11-17 20:52:11 -08:00
Nik Bougalis
eb6b79bed7 Adjust default number of threads for JobQueue:
The existing calculation would limit the maximum number of threads
that would be created by default to at most 6; this may have been
reasonable a few years ago, but given both the load on the network
as of today and the increase in the number of CPU cores, the value
should be revisited.

This commit, if merged, changes the default calculation for nodes
that are configured as `large` or `huge` to allow for up to twelve
threads.
2021-11-17 20:52:11 -08:00
Nik Bougalis
eaff0d30fb Make the sweep_interval individually configurable:
The "sweep interval" is the amount of time between successive sweeps of
of various in-memory data structures to remove stale items.

Prior to this commit, the interval was automatically adjusted, based on
the value of the `[node_size]` option in a server's configuration file.

If merged, this commit introduces a new configuration option that makes
it possible for a server operator to adjust the sweep interval and make
a CPU/memory tradeoff:

    [sweep_interval]
    <integer>

The specified value represents the number of seconds between successive
sweeps. The range of valid values is between 10 and 600.

Important operator notes:

This is an advanced configuration option that should not be used unless
there is empirical data which suggests that the default sweep frequency
is either resulting in performance problems or is causing undue load to
the server.

Note that adjusting the sweep interval may not have the intended effect
on the server. Lower values will not always translate to a reduction of
memory usage and higher values will not always translate to a reduction
of CPU usage and/or load.
2021-11-17 20:52:10 -08:00
Nik Bougalis
fae9f9b24b Use unordered map for handling book_offers:
The performance characteristics of `std::unordered_map` are better
than `std::map` and the former should be preferred when the strict
ordering of the latter is not required.
2021-11-17 20:52:10 -08:00
manojsdoshi
5d44998368 Set version to 1.8.0-rc1 2021-10-28 15:07:30 -07:00
manojsdoshi
a145759d1e Set version to 1.8.0-b7 2021-10-21 12:07:50 -07:00
Peng Wang
e2a42184b9 Implement tx stream with history 2021-10-21 12:05:48 -07:00
Devon White
00a4c3a478 Implement node-to-shard RPC control 2021-10-19 16:27:30 -07:00
Michael Legleux
0320d2169e Remove old boost workaround 2021-10-18 15:37:04 -07:00
Richard Holland
da26d11593 make building tests optional:
* disable build specific commandline options when built without tests
2021-10-18 15:37:04 -07:00
Ikko Ashimine
90aa3c75a7 Fix typo in LedgerHandler.cpp
non-existant -> non-existent
2021-10-18 15:37:04 -07:00
manojsdoshi
1197e49068 Set version to 1.8.0-b6 2021-10-06 13:17:24 -07:00
seelabs
b6ed50eb03 Remove unused variable:
Clang 13 warns about `nbytes` being unused. This patch removes this unused variable.
2021-10-06 13:17:10 -07:00
CJ Cobb
8c78c83d05 Add modification type to GetLedger gRPC method
* Specify whether a ledger object was created, modified or deleted.
2021-10-06 13:17:10 -07:00
CJ Cobb
a5c4684273 Move TxMeta to xrpl_core
* add peekNodes() to TxMeta for const access to nodes
2021-10-06 13:17:10 -07:00
Michael Legleux
2bbf0eb588 Don't restart rippled during apt upgrade:
Resolves #3294
2021-10-06 13:17:10 -07:00
Mark Travis
6095f55bf1 Build packages with new LICENSE.md file. 2021-10-06 13:17:10 -07:00
Edward Hennis
f64bd54093 Disable Boost's deque forward declaration 2021-10-06 13:17:10 -07:00
Edward Hennis
bc91fd740f Fix out-of-bounds reserve, and some minor optimizations 2021-10-06 12:26:09 -07:00
Edward Hennis
4a9bd7ed6d Refactor Feature name management and creation:
* Only require adding the new feature names in one place. (Also need to
  increment a counter, but a check on startup will catch that.)
* Allows rippled to have the code to support a given amendment, but
  not vote for it by default. This allows the amendment to be enabled in
  a future version without necessarily amendment blocking these older
  versions.
* The default vote is carried with the amendment name in the list of
  supported amendments.
* The amendment table is constructed with the amendment and default
  vote.
2021-10-06 12:24:19 -07:00
Edward Hennis
1ca8898703 Set OpenSSL 1.1.1 as the minimum version, recommend 3.0.0 for Windows
* Also clean up some formatting in the Windows instructions
* Changed the recommended version for Windows to 1.1.1L after deeper
  checking uncovered some build issues.
2021-10-06 12:22:52 -07:00
Nik Bougalis
8a25f32824 Set version to 1.8.0-b5 2021-09-13 15:13:15 -07:00
Nathan Nichols
d9d001dffd Forward validations and manifests in reporting mode 2021-09-13 15:13:15 -07:00
Mark Travis
bdfafa0b58 Always persist ledgers in SQLite 2021-09-13 15:13:15 -07:00
Mark Travis
2266b04dd8 Narrow check for appropriate configurations 2021-09-13 15:13:15 -07:00
Scott Schurr
c50d166c23 Add ripple::Expected which simulates std::expected:
Also integrates use of ripple::Expected into the code base.
2021-09-13 15:13:15 -07:00
Edward Hennis
de43d43560 Health check endpoint ignores fee escalation:
* Also refactor to use `jss` labels instead of strings for JSON fields.
2021-09-13 15:13:15 -07:00
Guido Vranken
f954faada6 Patch soci only once:
* Patch the soci unsigned-types.h file. If no changes are made, delete
  the patched file and exit. If there are changes, backup the original
  and replace it with the patched file.
* Fixes #3885

Patch Rocksdb only once:

* The repeated patches do not appear to affect build times, but avoiding
  unnecessary copies is good for its own sake.
2021-09-13 15:13:15 -07:00
Howard Hinnant
9f75f2d522 Replace boost::clamp with std::clamp 2021-09-13 15:13:15 -07:00
Devon White
cf70ecbd6d Improve Relational DB Interface safety checks 2021-09-13 15:13:15 -07:00
cdy20
2a298469be Properly handle configurations where use_tx_tables = 0 2021-09-13 15:13:15 -07:00
Michael Legleux
3be668b343 Link statically against PostgreSQL 2021-09-13 15:13:15 -07:00
Michael Legleux
2027f642ec Use absolute paths for conffile in Debian package:
Recent changes to reject relative pathnames in DEBIAN/conffiles
cause deb packages to fail on Debian 11 (Bullseye).

For more details:
https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/?id=15416a18f7645dbb9a36edb1e52f30a9e83c6627
2021-09-13 15:13:15 -07:00
Ikko Ashimine
9376d81d0d Fix typo in rippled-example.cfg
availabe -> available
2021-09-13 15:13:15 -07:00
seelabs
54e5d5fc35 Add missing include 2021-09-13 15:13:15 -07:00
seelabs
b8552abcea Fix potential deadlock in Validator sites:
There are two mutexes in ValidatorSite: `site_mutex_` and `state_mutex_`. Some
function end up locking both mutexes. However, depending on the call, the
mutexes could be locked in different orders, resulting in deadlocks.

If both mutexes are locked, this patch always locks the `sites_mutex_` first.
2021-09-13 15:13:15 -07:00
CJ Cobb
3fb60a89a3 Validate data size before calling uint256::fromVoid 2021-09-13 15:13:15 -07:00
Elliot Lee
15b0ae5bf0 Link to macOS Build Instructions
The most up-to-date macOS build instructions are on xrpl.org
2021-09-13 15:13:15 -07:00
Gregory Tsipenyuk
33b396c7b4 Discard non-validator TMSquelch message 2021-09-13 15:13:15 -07:00
Gregory Tsipenyuk
ea145d12c7 Improve transaction relaying logic:
The existing logic involves every server sending every transaction
that it receives to all its peers (except the one that it received
a transaction from).

This commit instead uses a randomized algorithm, where a node will
randomly select peers to relay a given transaction to, caching the
list of transaction hashes that are not relayed and forwading them
to peers once every second. Peers can then determine whether there
are transactions that they have not seen and can request them from
the node which has them.

It is expected that this feature will further reduce the bandwidth
needed to operate a server.
2021-09-13 15:13:15 -07:00
Nik Bougalis
0d17dd8228 Update LICENSE file:
The existing license file contains copyright statements and license
snippets referencing code that has long since been removed from the
codebase and which are not necessary any longer.

The general copyright statement for the ISC License is also changed
from "Ripple Labs" which is one contributor to the codebase, to the
more general "XRP Ledger Developers".

Remaining code which was originally taken from Bitcoin includes the
relevant copyright statement(s) inline.

To aid in the automated detection of the license type by GitHub and
tools like `licensee`, the following statement which referenced the
ISC license, and which was listed at the bottom of the file, is now
incorporated at the top:

>The accompanying files incorporate work covered by the following copyright
>and previous license notice:
>
>Copyright (c) 2011 Arthur Britto, David Schwartz, Jed McCaleb,
>Vinnie Falco, Bob Way, Eric Lombrozo, Nikolaos D. Bougalis, Howard Hinnant
2021-09-13 15:13:15 -07:00
Nik Bougalis
af5f28cbf8 Cleanup LedgerEntryType & TxType:
This commit removes the `ltINVALID` pseudo-type identifier from
`LedgerEntryType` and the `ttINVALID` pseudo-type identifier from
`TxType` and includes several small additional improvements that
help to simplify the code base.

It also improves the documentation `LedgerEntryType` and `TxType`,
which was all over the place, and highlights some important caveats
associated with making changes to the ledger and transaction type
identifiers.

The commit also adds a safety check to the `KnownFormats<>` class,
that will catch the the accidental reuse of format identifiers.
Ideally, this should be done at compile time but C++ does not (yet?)
allow for the sort of introspection that would enable this.
2021-09-13 15:13:14 -07:00
Nik Bougalis
234b754038 Eliminate code duplication and improve documentation:
The legacy functions `cdirFirst` and `dirFirst` were mostly
identical; the differences were only type-related. The same
situation existed with `cdirNext` and `dirNext`.

This commit removes the duplicated code by introducing new
template functions that abstract away the differences that
are present between each pair of functions.

This commit also improves the naming of function arguments,
helping to elucidate their purpose & use and to make the
code self-documenting.
2021-09-13 15:13:13 -07:00
Nik Bougalis
c231adf324 Minor cleanups in offer processing code 2021-09-09 11:26:06 -07:00
Nik Bougalis
7a088a5280 Merge master (1.7.3) into develop (1.8.0-b4) 2021-09-09 11:22:18 -07:00
Nik Bougalis
96bbabbd2e Set version to 1.7.3 2021-08-11 12:13:30 -07:00
Peng Wang
e37c108195 Add 'NegativeUNL' as a supported amendment:
The Negative UNL is a feature of the XRP Ledger consensus protocol that
improves liveness (the network's ability to make forward progress) during
a partial outage. Using the Negative UNL, servers adjust their effective
UNLs based on which validators are currently online and operational, so
that a new ledger version can be declared validated even if several trusted
validators are offline.

The Negative UNL has no impact on how the network processes transactions
or what transactions' outcomes are, except that it improves the network's
ability to declare outcomes final during some types of partial outages.

The feature was originally introduced with version **1.6.0** but it was
only possible to manually enable this. If merged, this commit introduces
the amendment associated with the feature so that server operators can
vote on whether to enable this feature.

For more details, please see https://xrpl.org/negative-unl.html

This commit closes #3898.
2021-08-10 18:20:51 -07:00
Edward Hennis
53df35eef3 Address OOB read in the base58 decoder:
Under some circumstances, it is possible to induce an out-of-bounds
memory read in the base58 decoder.

This commit addresses this issue.

Acknowledgements:
    Guido Vranken for discovering and responsibly disclosing this issue.

Bug Bounties and Responsible Disclosures:
    We welcome reviews of the rippled code and urge researchers to
    responsibly disclose any issues they may find.

    Ripple is generously sponsoring a bug bounty program for the
    rippled project. For more information please visit:

        https://ripple.com/bug-bounty
2021-08-10 18:20:37 -07:00
Scott Schurr
1061b01ab3 Improve SLE usage in check cashing 2021-08-10 18:00:49 -07:00
manojsdoshi
aee422e819 Set version to 1.8.0-b4 2021-07-27 11:35:54 -07:00
Bharath Chari
324667b877 Add the XRP Ledger Foundation to the validator_list_sites 2021-07-27 11:35:50 -07:00
Fraser Waters
10d73655bc Move signer_list in JSON response for api_version 2:
See https://github.com/ripple/xrpl-dev-portal/issues/938 for context.
2021-07-27 11:35:50 -07:00
Miguel Portilla
96f11c786e Refactor GetLedger and LedgerData message handlers:
* Verify message fields first
* Break up GetLedger handler into several functions
2021-07-27 11:35:50 -07:00
Edward Hennis
9202197354 Fix nested locks in ValidatorSite:
* Modify relevant unit tests to recreate the problematic scenario.
2021-07-27 11:35:50 -07:00
CJ Cobb
d78a396525 Don't build gRPC benchmarks 2021-07-27 11:35:50 -07:00
seelabs
cd27b5f2bd Some code cleanups tagged by static analysis 2021-07-27 11:35:50 -07:00
seelabs
78bc2727f7 Disable false-alarm gcc warning 2021-07-27 11:35:50 -07:00
Ikko Ashimine
8b58e93a2e Fix typo in Quality.h:
alway -> always
2021-07-27 11:35:50 -07:00
Nik Bougalis
3752234161 Report additional fields in validation stream:
The HardenedValidations amendment introduces additional fields
in validations:

- `sfValidatedHash`, if present, is the hash the of last ledger that
  the validator considers to be fully validated.
- `sfCookie`, if present, is a 64-bit cookie (the default
  implementation selects it randomly at startup but other
  implementations are possible), which can be used to improve the
  detection and classification of duplicate validations.
- `sfServerVersion`, if present, reports the version of the software
  that the validator is running. By surfacing this information,
  server operators gain additional insight about variety of software
  on the network.

If merged, this commit fixes #3797 by adding the fields to the
`validations` stream as shown below:

- `sfValidateHash` as `validated_hash`: a 256-bit hex string;
- `sfCookie` as `cookie`: a 64-bit integer as a string; and
- `sfServerVersion` as `server_version`: a 64-bit integer as
  a string.
2021-07-27 11:35:50 -07:00
Scott Schurr
bf75094224 Introduce CheckCashMakesTrustLine amendment:
With this amendment, the CheckCash transaction creates a TrustLine
if needed.  The change is modeled after offer crossing.  And,
similar to offer crossing, cashing a check allows an account to
exceed its trust line limit.
2021-07-27 11:35:50 -07:00
Scott Schurr
8d59c7dd40 Redistribute unit test priorities based on current measurements 2021-07-27 11:35:50 -07:00
Scott Schurr
a1fd579756 Clean up Section in BasicConfig.h:
The following changes were made:
- Removed dependency on template defined in beast detail namespace.
- Removed Section::find() method which had an obsolete interface.
- Made Section::get<>() easier to use for the common case of
  retrieving a std::string.  The revised get() method replaces old
  calls to Section::find().
- Provided a default template parameter to free function
  get<>(Section config, std::string name) so it stays similar to
  Section::get<>().

Then the rest of the code was adapted to these changes.

- Calls to Section::find() were replaced with calls to Section::get.
- Unnecessary get<std::string>() arguments were reduced to get().

These changes dug up an interesting artifact in the SHAMap unit
tests.  I'm not sure why the tests were working before, but there
was a problem with the case of a Section key.  The unit test is
fixed.
2021-07-27 11:35:50 -07:00
Nicolai Helldorff
b9943d3746 Adding another UNL to the examples:
* Added the third well known UNL (https://vl.xrplf.org)
2021-07-27 11:35:28 -07:00
Michael Legleux
b5502a49c3 update boost URL 2021-07-21 16:14:00 -07:00
manojsdoshi
7bd5d51e4e Set version to 1.8.0-b3 2021-06-03 10:58:32 -07:00
plan-do-break-fix
d4d937c37b Correct typos in documentation 2021-06-03 10:58:24 -07:00
Robert Zhang
2f0231025f Future proof rippled by changing variable name 'requires' to 'require'
'requires' is a keyword in C++20.

https://en.cppreference.com/w/cpp/keyword/requires
2021-06-03 10:58:24 -07:00
Howard Hinnant
f1a9e8840f Add scope_exit utilities 2021-06-03 10:58:24 -07:00
Howard Hinnant
2c559116fb Remove unneeded header includes. 2021-06-03 10:58:24 -07:00
Nik Bougalis
4bedbd1d39 Remove unused code 2021-06-03 10:58:24 -07:00
Nik Bougalis
433feade5d Automatically determine the node size:
The `[node_size]` configuration parameter is used to tune various
parameters based on the hardware that the code is running on. The
parameter can take five distinct values: `tiny`, `small`, `medium`,
`large` and `huge`.

The default value in the code is `tiny` but the default configuration
file sets the value to `medium`. This commit attempts to detect the
amount of RAM on the system and adjusts the node size default value
based on the amount of RAM and the number of hardware execution
threads on the system.

The decision matrix currently used is:

|         |   1  | 2 or 3 |   ≥ 4  |
|:-------:|:----:|:------:|:------:|
|  > ~8GB | tiny |   tiny |   tiny |
| > ~12GB | tiny |  small |  small |
| > ~16GB | tiny |  small | medium |
| > ~24GB | tiny |  small |  large |
| > ~32GB | tiny |  small |   huge |

Some systems exclude memory reserved by the the hardware, the kernel
or the underlying hypervisor so the automatic detection code may end
up determining the node_size to be one less than "appropriate" given
the above table.

The detection algorithm is simplistic and does not take into account
other relevant factors. Therefore, for production-quality servers it
is recommended that server operators examine the system holistically
and determine what the appropriate size is instead of relying on the
automatic detection code.

To aid server operators, the node size will now be reported in the
`server_info` API as `node_size` when the command is invoked in
'admin' mode.
2021-06-03 10:58:24 -07:00
Scott Schurr
10e4608ce0 Improve unit test coverage of AccountDelete transactor 2021-06-03 10:58:24 -07:00
Scott Schurr
ff3d2e7c29 Use jtx::offer_cancel more places:
Replace explicit construction of OfferCancel transactions in unit
tests with calls to jtx::offer_cancel().
2021-06-03 10:58:24 -07:00
Scott Schurr
7822a28c87 Replace dirAdd() calls with dirAppend() or dirInsert(): 2021-06-03 10:58:24 -07:00
Scott Schurr
dcba79be48 Only warn on non-pseudo transactions without accounts:
Closes #3153
2021-06-03 10:58:24 -07:00
Scott Schurr
2a7c573dec Fix clang warnings about copies vs references:
A recent version of clang notes a number of places in range
for loops where the code base was making unnecessary copies
or using const lvalue references to extend lifetimes.  This
fixes the places that clang identified.
2021-06-03 10:58:24 -07:00
Scott Schurr
22cc9a254a Maintenance of Offer and PayStrand unit tests 2021-06-03 10:58:24 -07:00
Scott Schurr
09ae9168ca Remove some "Env::close" noise from unit test output 2021-06-03 10:58:24 -07:00
CJ Cobb
9eb9b8f631 Fix reporting mode build issue
* Add isCaughtUp() as a member function of RelationalDBInterfacePostgres
2021-06-03 10:58:24 -07:00
CJ Cobb
6298daba1a Support dns in gRPC connection for reporting ETL 2021-06-03 10:58:24 -07:00
Peng Wang
2eb1c6a396 Enable testing beta RPC API version with config 2021-06-02 13:37:40 -07:00
Peng Wang
7717056cf2 Add higher bound to validations to keep 2021-06-02 13:37:40 -07:00
seelabs
9fd5cd303d Do not use __func__ inside lambdas 2021-06-02 13:37:40 -07:00
seelabs
04ff6249d5 Support boost 1.76:
* remove unused `std::hash` specialization
* Use `std::chrono::floor` instead of `floor`
2021-06-02 13:37:40 -07:00
seelabs
fa9ecae2d6 Update doc specifying min gcc version:
* resolves https://github.com/ripple/rippled/issues/3782
* gcc 8 is required for the charconv include file
2021-06-02 13:37:40 -07:00
Devon White
62d2b76fa8 Advance shard store unittests 2021-06-02 13:37:30 -07:00
Devon White
d95aab1139 Add lookup DBs to the Relational DB Interface
* Create SQLite database for mapping transaction IDs to shard indexes
* Create SQLite database for mapping ledger hashes to shard indexes
* Create additional test cases for the shard database
2021-06-01 15:37:15 -07:00
Miguel Portilla
80c2302fd3 Extend peer shard info 2021-06-01 15:37:15 -07:00
Devon White
38f954fd46 Improve NodeStore to ShardStore imports
* Run the import process in a background thread
* Prevent online_delete from removing ledgers pending import
2021-06-01 15:37:15 -07:00
John Freeman
14b2f27c3e Remove dead code 2021-06-01 15:37:04 -07:00
John Freeman
a2a37a928a Redesign stoppable object pattern 2021-06-01 15:36:28 -07:00
manojsdoshi
c10c0be11b Merge master (1.7.2) into develop (1.8.0-b2) 2021-05-26 14:05:55 -07:00
manojsdoshi
34ee4ca0cb Set version to 1.7.2 2021-05-19 12:22:03 -07:00
Edward Hennis
30fd45890b Create comparator wrapper classes for MSVC 2019 build issues:
* The std::less and std::equal_to comparators have [[nodiscard]], which
  conflicts with boost::bimap validations.
2021-05-17 10:43:56 -07:00
Edward Hennis
36fe1966c3 Update version of vcpkg for Windows builds 2021-05-17 10:43:56 -07:00
seelabs
1bb99e5d3c Rm some offers where the quality is reduced:
Substantial reductions in an offer's effective quality from its
initial quality may clog offer books.
2021-05-07 15:02:23 -07:00
CJ Cobb
430802c1cf Add load_factor to server_info in reporting mode
* load_factor was missing from server_info when the server was running in
  reporting mode. Now, the reporting mode server calls server_info on the p2p
  node, and propagates the load_factor back to the client.
2021-05-07 14:58:29 -07:00
Nik Bougalis
9106a06579 Set version to 1.7.1 2021-04-01 13:40:41 -07:00
Nik Bougalis
79e69da364 Adjust OpenSSL defaults and mitigate CVE-2021-3499:
In order to effectively mitigate CVE-2021-3499 even when compiling
against versions of OpenSSL prior to 1.1.1k, this commit:

1) requires use of TLS 1.2 or later. Note that both TLS 1.0 and
   TLS 1.1 have been officially deprecated for over a year.
2) disables renegotiation support for TLS 1.2 connections.

Lastly, this commit also changes the default list of ciphers that
the server offers, limiting it only to ciphers that are part of
TLS 1.2.
2021-04-01 13:40:40 -07:00
CJ Cobb
73116297aa Properly encode results from the tx RPC command:
The `tx` command supports output in both "text" and "binary" modes,
controlled by the binary flag. For more details on the command and
the possible arguments, please see: https://xrpl.org/tx.html.

The existing handler would incorrectly deal with metadata when in
binary mode. This commit corrects this issue, ensuring that the
metadata is properly encoded, depending on the mode.
2021-04-01 13:21:02 -07:00
CJ Cobb
8579eb0c19 Maintain compatibility for forwarded RPC responses:
Typically, an RPC response contains a `result` field, which
contains details about the operation performed. For ease of
parsing, forwarded responses must look like a non-forwarded
response.

In some instances the response was incorrectly composed, so
that the actual `result` object would be encapsulated by an
outer `result` object, breaking existing code.

This commit, addresses this issue and correctly "folds" the
`result` field, ensuring a consistent schema for responses.
2021-04-01 10:40:05 -07:00
Mark Travis
9c8caddc5a Support HTTP health check in reporting mode. 2021-04-01 10:40:05 -07:00
manojsdoshi
2913847925 Set version to 1.8.0-b2 2021-04-01 10:39:22 -07:00
Crypto Brad Garlinghouse
cf8438fe1d Remove obsolete URLs and references to Ripple 2021-04-01 10:38:47 -07:00
cdy20
6d82fb83a0 Relational DB interface 2021-04-01 10:38:22 -07:00
CJ Cobb
207e1730e9 Serialize metadata as blob in tx RPC when binary flag is true 2021-04-01 10:37:46 -07:00
J. Scott Branson
f0424fe7dd Document parameters in example configuration file 2021-04-01 10:37:38 -07:00
Alloy Networks
2e456a835d Remove legacy protocol version (1.2) 2021-04-01 10:37:17 -07:00
Edward Hennis
ab9039e77d Create comparator wrapper classes for MSVC 2019 build issues:
* The std::less and std::equal_to comparators have [[nodiscard]], which
  conflicts with boost::bimap validations.
2021-03-18 14:12:41 -07:00
Howard Hinnant
9932a19139 Reduce coupling to date.h by calling C++17 chrono functions 2021-03-17 15:02:15 -07:00
Howard Hinnant
64e4a89470 Clarify the safety of NetClock::time_point arithmetic:
* NetClock::rep is uint32_t and can be error-prone when
  used with subtraction.
* Fixes #3656
2021-03-17 15:01:52 -07:00
manojsdoshi
9d89d4c188 Set version to 1.8.0-b1 2021-03-11 16:21:51 -08:00
Edward Hennis
b2bf2b6e6b Add instructions for resolving validation failures:
* Add instructions to the workflow at the point where the failure would
  occur, which is where someone experiencing a failure is most likely to
  look. To keep things simple, the instructions are always printed. The
  assumption is that if the job succeeds, nobody is likely to look
  anyway.
* Provides the diff of the failure as an artifact, so the user can apply
  it directly to their repo.
* Also update the levelization/README.md to clarify the levels a little
  bit.
2021-03-11 14:35:31 -08:00
Scott Schurr
3b33318dc8 Prefer std::optional over boost:optional:
Some of the boost::optionals must remain for now.  Both
boost::beast and SOCI have interfaces that require
boost::optional.
2021-03-11 14:35:31 -08:00
Scott Schurr
85307b29d0 Add constexpr constructor for base_uint 2021-03-11 14:35:31 -08:00
John Northrup
95426efb8a Update to use Ripple Artifactory for docker cache
This updates the build process to use the local Artifactory server as a docker image cache to avoid being rate limited by docker hub during the build process.
2021-03-11 14:35:31 -08:00
CJ Cobb
3e2b568ef9 Add list of dependencies for building reporting mode 2021-03-11 14:35:31 -08:00
Nik Bougalis
a06525649d Use libsecp256k1 instead of OpenSSL for key derivation:
While most of the code associated with secp256k1 operations had
been migrated to libsecp256k1, the deterministic key derivation
code was still using calls to OpenSSL.

If merged, this commit replaces the OpenSSL-based routines with
new libsecp256k1-based implementations. No functional change is
expected and the change should be transparent.

This commit also removes several support classes and utility
functions that wrapped or adapted various OpenSSL types that
are no longer needed.

A tip of the hat to the original author of this truly superb
library, Dr. Pieter Wuille, and to all other contributors.
2021-03-11 14:35:31 -08:00
Nik Bougalis
b4699c3b46 Improve the Byzantine validator detector:
This commit expands the detection capabilities of the Byzantine
validation detector. Prior to this commit, only validators that
were on a server's UNL were monitored. Now, all the validations
that a server receives are passed through the detector.
2021-03-11 14:35:31 -08:00
Nik Bougalis
27d978b891 Simplify SHAMapItem construction:
The existing class offered several constructors which were mostly
unnecessary. This commit eliminates all existing constructors and
introduces a single new one, taking a `Slice`.

The internal buffer is switched from `std::vector` to `Buffer` to
save a minimum of 8 bytes (plus the buffer slack that is inherent
in `std::vector`) per SHAMapItem instance.
2021-03-11 14:35:30 -08:00
cdy20
f91b568069 Add support for deterministic database shards (#2688):
Add support to allow multiple indepedent nodes to produce a binary identical
shard for a given range of ledgers. The advantage is that servers can use
content-addressable storage, and can more efficiently retrieve shards by
downloading from multiple peers at once and then verifying the integrity of
a shard by cross-checking its checksum with the checksum other servers report.
2021-03-10 16:53:36 -08:00
Howard Hinnant
06bd16c928 Simplify basic_seconds_clock:
* Remove unneeded vector of workers and associated mutex
* Remove unneeded generic worker
* Remove unneeded Clock template parameter
2021-03-10 16:53:18 -08:00
manojsdoshi
c0a0b79d2d Set version to 1.7.0 2021-02-23 12:51:24 -08:00
manojsdoshi
1d5d902d28 Set version to 1.7.0-rc4 2021-02-18 18:26:17 -08:00
Mark Travis
f284a19246 Don't delete from tx table if it's not in use 2021-02-18 18:24:39 -08:00
manojsdoshi
735b8b7d48 Set version to 1.7.0-rc3 2021-02-12 13:33:37 -08:00
Scott Schurr
a2e1a7a84d TicketSequence with non-zero Sequence is an error:
Before this change any non-zero Sequence field was handled as
a non-ticketed transaction, even if a TicketSequence was
present.  We learned that this could lead to user confusion.
So the rules are tightened up.

Now if any transaction contains both a non-zero Sequence
field and a TicketSequence field then that transaction
returns a temSEQ_AND_TICKET error code.

The (deprecated) "sign" and "submit" RPC commands are tuned
up so they auto-insert a Sequence field of zero if they see
a TicketSequence in the transaction.

No amendment is needed because this change is going into
the first release that supports the TicketBatch amendment.
2021-02-12 13:33:26 -08:00
John Freeman
c138338358 Fix declaration of CassandraBackend::counters_ 2021-02-12 13:32:33 -08:00
Ikko Ashimine
e7eba93666 Fix typo in aged_unordered_container.h 2021-02-12 12:42:22 -08:00
Edward Hennis
93ea4b2f4f Fix a typo in the [sqlite] section description 2021-02-12 12:38:01 -08:00
CJ Cobb
5776c2ebe5 Document secure_gateway for gRPC 2021-02-12 12:37:32 -08:00
manojsdoshi
8defb4cd28 Set version to 1.7.0-rc2 2021-02-04 19:13:14 -08:00
Miguel Portilla
d358495f02 Add database counters:
Fix bug where DatabaseRotateImp::getBackend and ::sync utilized the writable
backend without a lock. ::getBackend was replaced with ::getCounters.
2021-02-04 19:10:28 -08:00
Gregory Tsipenyuk
38dd2d6677 Fix idle peer timer:
This commit fixes  #3754
2021-02-04 19:10:01 -08:00
manojsdoshi
80bd107e57 Set version to 1.7.0-rc1 2021-01-29 12:01:56 -08:00
manojsdoshi
68286df23d Set version to 1.7.0-b12 2021-01-26 11:26:17 -08:00
Edward Hennis
d494d7a725 Use 64-bit Visual Studio build tools in documentation and Travis 2021-01-26 11:26:02 -08:00
seelabs
36be4856fd Fix path finding for XRP dst and sendmax:
* Fix bug where incorrect max amount was set for XRP
* Fix bug where incorrect source currencies were set when XRP was the dst and a
  sendmax amount was set
2021-01-25 18:49:49 -08:00
seelabs
c47b4f3667 Improve canonicalization of serialized amounts:
The existing code that deserialized an STAmount was sub-optimal and performed
poorly. In some rare cases the operation could result in otherwise valid
serialized amounts overflowing during deserialization. This commit will help
detect error conditions more quickly and eliminate the problematic corner cases.
2021-01-25 18:49:49 -08:00
seelabs
fe129e8e4f Optimize payment path exploration in flow:
* Use theoretical quality to order the strands
* Do not use strands below the user specified quality limit
* Stop exploring strands (at the current quality iteration) once any strand is non-dry
2021-01-25 18:49:49 -08:00
John Freeman
0a1fb4e6ca Reduce nesting and remove dead code 2021-01-25 18:49:49 -08:00
Peng Wang
7e97bfce10 Implement ledger forward replay 2021-01-25 18:49:49 -08:00
cdy20
49409dbf27 Added document describing why 16k-ledgers shards
are better than 8k- and 32k-ledgers
2021-01-25 18:49:43 -08:00
mDuo13
c3227a67ec Fix telBAD_PUBLIC_KEY message:
The previous error description was focused on keys that are too long,
but this error can occur if the key is too short or does not contain
the correct prefix.
2021-01-25 13:44:31 -08:00
Elliot Lee
a4a46a491f Update validators-example.txt 2021-01-25 13:44:16 -08:00
manojsdoshi
c11037fd27 Set version to 1.7.0-b11 2021-01-20 16:33:19 -08:00
Miguel Portilla
114981f774 Include missing header in json_reader 2021-01-20 16:32:59 -08:00
CJ Cobb
27543170d0 Add Reporting Mode
* Add a new operating mode to rippled called reporting mode
* Add ETL mechanism for a reporting node to extract data from a p2p node
* Add new gRPC methods to faciliate ETL
* Use Postgres in place of SQLite in reporting mode
* Add Cassandra as a nodestore option
* Update logic of RPC handlers when running in reporting mode
* Add ability to forward RPCs to a p2p node
2021-01-20 11:30:03 -08:00
Mark Travis
b0a39c5f86 Add Postgres functionality 2021-01-20 10:53:24 -08:00
John Freeman
4fbb1699be Add start-sync-stop Python script 2021-01-20 10:53:24 -08:00
Wietse Wind
319f29da8d Update ripple.com » xrpl.org links 2021-01-20 10:53:24 -08:00
Nik Bougalis
70bacb349e Improve manifest relaying and domain reporting:
- The changes to manifest relaying introduced with commit f74b469e68
  will cause newly accepted manifests to be sent back to the peer from
  which they were received. This no longer happens: a newly accepted
  manifest is never sent back to the peer we received it from.
- When encountering a manifest without a domain set, the `manifest` and
  `validator_info` commands would include an empty string as the domain
  associated with the manifest. This no longer happens: if a domain is
  not present, the `domain` field will not be.
2021-01-20 10:53:24 -08:00
Nik Bougalis
012bbcfe36 Fix parsing of node public keys in manifest CLI:
The existing code attempts to validate the provided node public key
using a function that assumes that the encoded public key is for an
account. This causes the parsing to fail.

This commit fixes #3317 by letting the caller specify the type of
the public key being checked.
2021-01-20 10:51:07 -08:00
Nik Bougalis
f74b469e68 Improve manifest relaying:
The manifest relay code would only ever relay manifests from validators
on a server's UNL which means that the manifests of validators that are
not broadly trusted can fail to propagate across the network, which can
make it difficult to detect and track such validators.

This commit, if merged, propagates all manifests on a best-effort basis
resulting in broader availability of manifests on the network and avoid
the need to introduce on-ledger manifest storage or to establish one or
more manifest repositories.
2021-01-20 10:51:04 -08:00
Nik Bougalis
55dc7a252e Set version to 1.7.0-b10 2021-01-09 13:50:07 -08:00
Nik Bougalis
b015623128 Add missing include headers 2021-01-09 13:50:07 -08:00
Nik Bougalis
c76a124d14 Remove unused Beast code 2021-01-09 13:50:06 -08:00
Gregory Tsipenyuk
74d96ff4bd Add experimental validation & proposal relay reduction support:
- Add validation/proposal reduce-relay feature negotiation to
  the handshake
- Make squelch duration proportional to a number of peers that
  can be squelched
- Refactor makeRequest()/makeResponse() to facilitate handshake
  unit-testing
- Fix compression enable flag for inbound peer
- Fix compression algorithm parsing in the header parser
- Fix squelch duration in onMessage(TMSquelch)

This commit fixes 3624, fixes 3639 and fixes 3641
2021-01-09 13:49:40 -08:00
Scott Schurr
44fe0e1fc4 Add support for the 'TicketBatch' amendment:
Support for 'out-of-sequence' transaction execution was introduced
in commit 7724cca384.

The changes in that commit were gated under a feature but there was
no corresponding amendment introduced that would allow the network
to vote on this amendment.

This commit introduces 'TicketBatch' amendment as the amendment
that is associated with the tickets feature. If the amendment is
enabled, it will activate support for tickets.

This commit also removes several workarounds that are no longer
needed in unit tests.
2021-01-08 14:43:06 -05:00
John Freeman
78245a072c Clean-up the Stoppable architecture 2021-01-08 14:43:06 -05:00
Miguel Portilla
1fd1c34112 Prevent deadlock in storeSQLite 2021-01-08 14:43:06 -05:00
Edward Hennis
0dae22adf2 Prevent simultaneous outgoing async peer messages:
* Also, only send validator list on incoming connection instead of both
  incoming and outgoing.
2021-01-08 14:43:06 -05:00
Edward Hennis
746181cb33 Analyze and document levelization:
* Markdown explanation of what levelization is, the intended levels, as
  well as the process used to determine dependencies
* Shell script finds all dependencies, groups them, and finds cyclic
  dependencies and maps out non-cyclic dependencies.
* Github job to run the script and fail if anything changes. Should
  catch introduction of new dependencies and new problems. Will also
  detect changes if problems or dependencies are removed.
2021-01-08 14:43:02 -05:00
Edward Hennis
4b9d3ca7de Support UNLs with future effective dates:
* Creates a version 2 of the UNL file format allowing publishers to
  pre-publish the next UNL while the current one is still valid.
* Version 1 of the UNL file format is still valid and backward
  compatible.
* Also causes rippled to lock down if it has no valid UNLs, similar to
  being amendment blocked, except reversible.
* Resolves #3548
* Resolves #3470
2021-01-08 12:35:08 -05:00
Nik Bougalis
54da532ace Set version to 1.7.0-b9 2020-12-18 15:23:31 -08:00
Gregory Tsipenyuk
4159b02753 Properly handle the "compression" flag:
- Fix compression enable flag in the inbound peer
- Fix compression algorithm fetching in protocol message
2020-12-18 15:23:26 -08:00
Edward Hennis
42a068ab5e Force wipe the travis caches with "travis_clean_cache" 2020-12-18 15:23:26 -08:00
Edward Hennis
6f7d413d88 Simplify travis config:
* Move all the vcpkg windows dependency installations into one step.
* Move the unmodified `before_install` step above the matrix to improve
  readability, because this step runs before any of the matrix steps.
2020-12-18 15:23:26 -08:00
Edward Hennis
1aaafeb57b Reset corrupt / inconsistent dependency source folders:
* Also display some more information about the build
2020-12-18 15:23:26 -08:00
JoelKatz
6dd3d825c8 Adjust time resolution of genesis ledger:
Due to some quirky emergent behavior, the server can't really begin
synching until twice the default close time resolution of the genesis
ledger, which is 30 seconds, has passed. In effect, this causes a one
minute delay.

This commit adjusts the default close time resolution down to the
minimum allowed resoluion of 10 seconds, so the corresponding delay
is reduced by 67% down to 20 seconds. This should be enough time to
ensure the server has reasonable connectivity without unduly delaying
initial synch times.
2020-12-18 15:23:26 -08:00
JoelKatz
02ccdeb94e Rework deferred node logic and async fetch behavior
This comment explains this patch and the associated patches
that should be folded into it. This paragraph should be removed
when the patches are folded after review.

This change significantly improves ledger sync and fetch
times while reducing memory consumption. The change affects
the code from that begins with SHAMap::getMissingNodes and runs
through to Database::threadEntry.

The existing code issues a number of async fetches which are then
handed off to the Database's pool of read threads to execute.
The results of each read are placed in the Database's positive
and negative caches. The caller waits for all reads to complete
and then retrieves the results out of these caches.

Among other issues, this means that the results of the first read
cannot be processed until the last read completes. Additionally,
all the results must sit in memory.

This patch changes the behavior so that each read operation has a
completion handler associated with it. The completion of the read
calls the handler, allowing the results of each read to be
processed as it completes. As this was the only reason the
negative and positive caches were needed, they can now be removed.

The read generation code is also no longer needed and is removed.
The batch fetch logic was never implemented or supported and is
removed.
2020-12-17 09:11:39 -08:00
seelabs
7b192945eb Replace std::pmr with boost::pmr:
gcc's implementation of `prm::synchronized_pool_resource` showed
extremely poor performance compared with
`boost::synchronized_pool_resouece`. Boost's implementation of pmr is
now used in all cases (previously it was only used when a standard
lib, like clang's, lacked an implementation of pmr).

This patch also makes a minor change where inner nodes are constructed
with sparse arrays, unless "dense" is explicitly requested.
2020-12-17 09:11:37 -08:00
Howard Hinnant
28ed2b9e69 Persist API-configured voting settings:
Prior to this commit, the amendments that a server would vote in support
of or against could be configured both via the configuration file and
via the command line "feature" command. Changes made in the configuration
file would only be loaded once at server startup and changes made via the
command line take effect immediately but are not persisted across
restarts.

This commit deprecates management of amendments via the configuration
file and stores the relevant information in the `wallet.db` database
file.

1. On startup, the new code parses the configuration file.
2. If the `[veto_amendments]` or `[amendments]` sections are present,
   we check if the `FeatureVotes` table is present in `wallet.db`.
3. If it is not, we create the `FeatureVotes` table and transfer the
   settings from the config file.
4. Proceed normally but only reference the `FeatureVotes` table instead
   of the config file.
5. Warns if the voting table already exists in `wallet.db` and there
   exists voting sections in the config file. The config file is ignored
   in this case.

This change addresses & closes #3366
2020-12-17 05:56:17 -08:00
Nik Bougalis
24b17c6de9 Set version to 1.7.0-b8 2020-12-12 04:51:02 -08:00
Edward Hennis
6eb93e9f8a Fix nested locks in ValidatorList:
* Found several functions called under lock that take a lock. Refactor
  to require a lock as a parameter instead.
* Found several functions called under lock that don't take a lock, but
  should. Refactored those as well to require a lock as a parameter.
2020-12-12 04:50:59 -08:00
Danil Nemirovsky
8173d1f643 Add pkg-config package to linux build guide 2020-12-12 04:50:59 -08:00
seelabs
b29812e40b Fix double counting unit test failures:
Unit tests are counting test failures, process crashes, and process exit code
failures in the count. Since a failing tests causes the process exit code to
return failure, we get extra counts. This patch removes process exit code
failures from the count.
2020-12-12 04:50:59 -08:00
seelabs
183dcd08d9 Remove ReadViewFwdRange end iterator cache:
ReadViewFwdRange was storing a cached `end_` iterator that was lazily
created in an iterators `end()` function. When the cache is empty, and
the range it iterated from multiple threads, this creates a race
condition.

This change has performance consequences for "old style" for loops.
For example:

```
 // don't do this
for(auto i = tx_range.begin(); i != tx_range.end(); ++i)

```

Can call the now expensive `end()` function more often than needed.
Range-based for loop (I.e. `for(auto const& t : tx_range)`) should be
used instead.
2020-12-12 04:50:59 -08:00
Nik Bougalis
deea16e14f Correct handling of corner-cases involving custom iterators:
- Under some conditions, comparing `ReadViewFwdRange::iterators`
  for equality could derefence an empty `std::unique_ptr` which
  will result in a crash.
- Misuse of the `equal` API could result in a `std::bad_cast`
  exception being thrown from when iterating transactions or
  SLEs from the `OpenView`, `RawStateTable` and `Ledger` classes.
2020-12-12 04:50:59 -08:00
Edward Hennis
e96a719724 Fix compiler warnings & linker errors:
Changes introduced with 1.7.0-b5 and 1.7.0-b6 could result in compiler
warnings and/or linker errors when compiling in certain configurations.
2020-12-11 15:45:12 -08:00
Edward Hennis
8e38d8e6f0 Add clang release builds to CI 2020-12-11 15:45:08 -08:00
Nik Bougalis
a0b862bbf7 Set version to 1.7.0-b7 2020-12-09 05:54:53 -08:00
Nik Bougalis
434e2f4cbf Set version to 1.7.0-b6 2020-12-09 08:50:42 -05:00
seelabs
cb0572d66e Store InnerNode children in sparse arrays:
A large percentage of inner nodes only store a small number of children. Memory
can be saved by storing the inner node's children in sparse arrays. Measurements
show that on average a typical SHAMap's inner nodes can be stored using only 25%
of the original space.
2020-12-09 08:50:42 -05:00
Nik Bougalis
5c8e072b7f Set version to 1.7.0-b5 2020-12-04 12:45:16 -08:00
Nik Bougalis
e39c452f31 Simplify processing of inbound ledger data 2020-12-04 12:45:14 -08:00
Nik Bougalis
1bb294afbc Refactor and improve the SHAMap code:
This commit combines a number of cleanups, targeting both the
code structure and the code logic. Large changes include:

 - Using more strongly-typed classes for SHAMap nodes, instead of relying
   on runtime-time detection of class types. This change saves 16 bytes
   of memory per node.
 - Improving the interface of SHAMap::addGiveItem and SHAMap::addItem to
   avoid the need for passing two bool arguments.
 - Documenting the "copy-on-write" semantics that SHAMap uses to
   efficiently track changes in individual nodes.
 - Removing unused code and simplifying several APIs.
 - Improving function naming.
2020-12-04 12:45:13 -08:00
Nik Bougalis
5def79e93c Track counts of SHAMap-related objects 2020-12-04 12:45:12 -08:00
Nik Bougalis
f072469409 Simplify & modernize code:
- Simplify and consolidate code for parsing hex input.
- Replace beast::endian::order with boost::endian::order.
- Simplify CountedObject code.
- Remove pre-C++17 workarounds in favor of C++17 based solutions.
- Improve `base_uint` and simplify its hex-parsing interface by
  consolidating the `SexHex` and `SetHexExact` methods into one
  API: `parseHex` which forces callers to verify the result of
  the operation; as a result some public-facing API endpoints
  may now return errors when passed values that were previously
  accepted.
- Remove the simple fallback implementations of SHA2 and RIPEMD
  introduced to reduce our dependency on OpenSSL. The code is
  slow and rarely, if ever, exercised and we rely on OpenSSL
  functionality for Boost.ASIO as well.
2020-12-04 12:45:12 -08:00
Nik Bougalis
4a444f7d60 Simplify SHAMapTreeNode APIs:
- Provide separate functions for serializing depending on whether
  one wants a "wire" version of a node, or one suitable for hashing.
- Remove unused functions
2020-12-04 12:45:11 -08:00
Nik Bougalis
ab77444fa3 Simplify SHAMapNodeID:
The existing SHAMapNodeID object has both a valid and an invalid state
and requirs callers to verify the state of an instance prior to using
it. A simple set of changes removes that restriction and ensures that
all instances are valid, making the code more robust.

This change also:

1. Introduces a new function to construct a SHAMapNodeID from a
   serialized blob; and
2. Reduces the amount of constructors the class exposes.
2020-12-04 12:45:11 -08:00
Nik Bougalis
57ffc58613 Optimize peer I/O:
- Limit the lifetime of a buffer that was only used in the early
  phases of peer connection establishment but which lived on as
  long as the peer was active.
- Cache the message used to transfer manifests, so it can be reused
  instead of recreated for every peer connection.
- Improve the reading of partial messages by passing a hint to the
  I/O layer if the number of bytes needed to complete the message
  is known.
2020-12-04 12:45:10 -08:00
Nik Bougalis
8c386ae07e Reduce PING frequency and simplify logic:
The existing code issues a PING to each peer every 8 seconds. While
frequent PINGs allow us to estimate a peer's latency with a high
degree of accuracy, this "inter-server polka dance" is inefficient
and not useful. This commit, if merged, reduces the PING frequency
to once every 60 seconds.

Additionally, this commit simplifies the PING handling logic and
merges the code used to check and disconnect peers which fail to
track the network directly into the timer callback.
2020-12-04 12:45:10 -08:00
Nik Bougalis
cba6b4a749 Improve handling of peers that aren't synced:
When evaluating the fitness and usefulness of an outbound peer, the code
would incorrectly calculate the amount of time that the peer spent in
a non-useful state.

This commit, if merged, corrects the calculation and makes the timeout
values configurable by server operators.

Two new options are introduced in the 'overlay' stanza of the config
file. The default values, in seconds, are:

[overlay]
max_unknown_time = 600
max_diverged_time = 300
2020-12-04 12:45:09 -08:00
Nik Bougalis
64191a4b13 Update Travis build matrix:
The Travis build matrix included builds for gcc 7 and clang 7 which
do not appear to have sufficient C++17 support to compile rippled
any longer.
2020-12-04 12:45:09 -08:00
Nik Bougalis
9bd6b249ce Remove a pre-C++17 workaround for std::is_invocable_r 2020-12-04 12:45:08 -08:00
Gregory Tsipenyuk
bec6c626d8 Add finer-grained control for incoming & outgoing peer limits:
This commit replaces the `peers_max` configuration element which had
a predetermined split between incoming and outgoing connections with
two new configuration options, `peers_in_max` and `peers_out_max`,
which server operators can use to explicitly control the number of
incoming and outgoing peer slots.
2020-12-04 12:44:19 -08:00
Devon White
7ddf856d58 Temper excessive warnings about historical shard
capacity
2020-12-04 12:44:19 -08:00
Howard Hinnant
a3915fa5c4 Switch to boost::shared_mutex:
* Avoid std::shared_mutex bug on ubuntu 18.04
* Fixes #3663
2020-12-04 12:44:19 -08:00
Scott Schurr
17abca1caa Use macros to instantiate most SField instances:
There have been cases in the past where SFields have been defined
in such a way that they did not follow our conventions.  In
particular, the string representation of an SField should match
the in-code name of the SField.

This change leverages the preprocessor to encourage SFields to
be properly constructed.

The suffixes of SField types are changed to be the same as
the suffixes of corresponding SerializedTypeIDs.  This allows
The preprocessor to match types using simple name pasting.

Since the string representation of the SField is part of our
stable API, the name of sfPayChannel was changed to sfChannel.
This change allows sfChannel to follow our conventions while
making no changes to our external API.
2020-12-04 12:44:19 -08:00
John Northrup
f239050054 Remove DinD Service from container build template (#3676)
* Remove DinD Service from container build template

DinD has changed how it works on GItLab due to recent docker changes such that the service no long needs to be called so long as the runner is being run on a `docker-X` tagged machine.

* refactor for docker service on normal node
2020-12-03 23:43:21 -06:00
manojsdoshi
76a6956138 Set version to 1.7.0-b4 2020-11-19 10:59:46 -08:00
Devon White
cf5ca9a5cf Download queued shards only if there is space for them all 2020-11-19 10:57:13 -08:00
Edward Hennis
77ec62e9c8 Always check the sequence when adding to the transaction queue:
* If multiple transactions are queued for the account, change the
  account's sequence number in a temporary view before processing the
  transaction.
* Adds a new "at()" interface to STObject which is identical to the
  operator[], but easier to write and read when dealing with ptrs.
* Split the TxQ tests into two suites to speed up parallel run times.
2020-11-18 13:25:27 -08:00
Edward Hennis
a3f2196d4e Test cancelling offers that are still in the TxQ 2020-11-18 13:25:27 -08:00
seelabs
d89c158a77 Replace qalloc with boost::pmr 2020-11-18 13:25:27 -08:00
Miguel Portilla
ef53197e1f Fix ledger sequence on copynode 2020-11-18 13:07:52 -08:00
Miguel Portilla
8707c15b9c Use NuDB burst size and use NuDB version 2.0.5 2020-11-18 13:07:26 -08:00
manojsdoshi
0b4e34b03b Set version to 1.7.0-b3 2020-10-14 11:19:09 -07:00
Miguel Portilla
6968da153c Fix improper promotion of bool on return 2020-10-14 11:17:44 -07:00
Miguel Portilla
03c809371a Add Shard pool management 2020-10-14 11:17:44 -07:00
Nik Bougalis
d282b0bf85 Report server domain to other servers:
This commit introduces a new configuration option that server
operators can set. The value is communicated to other servers
and is also reported via the `server_info` API.

The value is meant to allow third-party applications or tools
to group servers together. For example, a tool that visualizes
the network's topology can group servers together.

Similar to the "Domain" field in validator manifests, an operator
can claim any domain. Prior to relying on the value returned, the
domain should be verified by retrieving the xrp-ledger.toml file
from the domain and looking for the server's public key in the
`nodes` array.
2020-10-14 11:17:44 -07:00
Edward Hennis
efa615a5e3 Improve Windows and MacOS test performance in Travis CI:
* Increases hard-coded number of parallel unit test processes for
  Windows and MacOS builds from 1 to 2.
* Reduces Travis job time to well under the timeout value of 1.5 hours.
* Continue using a hard-coded value rather than `nprocs` because higher
  values cause some jobs to run out of memory.
2020-10-14 11:17:44 -07:00
Edward Hennis
068db1f48b Generalize unit test that depends on number of amendments 2020-10-13 11:31:54 -07:00
Devon White
afacbe2a3a Update shard db sections in the example config file 2020-10-13 11:31:54 -07:00
Devon White
f8a0ef8f87 Delay HTTPDownloader dtor until current session finishes 2020-10-13 11:31:54 -07:00
Carl Hua
e1a2939f89 Set version to 1.7.0-b2 2020-09-21 08:43:37 -04:00
Carl Hua
95da398e7d Update manifest RPC to be public 2020-09-21 08:43:16 -04:00
Scott Schurr
52adcc73d9 Add unit tests to check synchronization of KnownFormats and gRPC 2020-09-17 15:05:59 -07:00
Edward Hennis
dbde686a97 Improve reporting of unusual unit test failures:
* Jobs with no unit tests are counted as failures. Resolves #3474
* Crashed processes are counted as failures. Resolves #3600
* Any tests specified on the command line test do not have matching
  suites are counted as failures.
* Remove unused CI manual test.
2020-09-17 15:05:36 -07:00
seelabs
1129110be3 Document unused SHAMap constructor parameter 2020-09-17 15:05:15 -07:00
Nathan Nichols
a26a175957 Add GRPCServer to Stoppable Hierarchy 2020-09-17 15:05:06 -07:00
Nathan Nichols
795de3a75a Load transaction metadata from SQLite:
When processing the `tx` command, we will now load both the transaction
and its metadata directly from SQLite.

Previously the `tx` RPC call was querying SQLite for the transaction
and then separately querying the key-value store for the metadata.
2020-09-17 15:01:37 -07:00
Nathan Nichols
8116b569c7 Refactor AccountLinesRPC no_ripple_default logic 2020-09-17 15:01:37 -07:00
Devon White
2bba79138f Support shard downloading via HTTP or HTTPS 2020-09-17 15:01:22 -07:00
Nik Bougalis
a8d481c2a5 Set version to 1.7.0-b1 2020-09-01 16:39:07 -07:00
Nik Bougalis
85fc1e8235 Remove legacy support for endpoint dissemination:
Support for IPv6 messages was added with commit 08382d866b
and version 1.1.0. No peer presently connected to the network in a useful capacity fails
to understand v2 messages.

This commit removes the code that generates and processes v1 messages and deletes legacy
messages from the protocol buffer definition file.
2020-09-01 16:39:00 -07:00
Miguel Portilla
ab4102a632 Improve shard finalize memory utilization 2020-09-01 16:39:00 -07:00
Howard Hinnant
4ae2f06be4 Remove unused nullConstSHAMapItem
This commit fixes #3550.
2020-09-01 16:39:00 -07:00
Crypto Brad Garlinghouse
831e03ad2a Update Base58 codec and remove Bitcoin support:
Use C++17 constant expressions to calculate the inverse
alphabet map at compile time instead of at runtime.

Remove support for encoding & decoding tokens using the
Bitcoin alphabet.
2020-09-01 16:39:00 -07:00
seelabs
ab9f3fa42a Remove assert from Keylet:
Some transactions specify a keylet directly (Checks, for example). In those
cases, the assert will trigger even though there is no error.
2020-09-01 16:39:00 -07:00
Richard Holland
13b8359de6 Reserve transaction type and error codes for event hooking:
Event hooks will allow accounts to introduce configurable behavior
in response to send and receive events.
2020-09-01 16:39:00 -07:00
Nathan Nichols
660d9c1602 Make the transaction job queue limit adjustable:
The job queue can impose limits of how many jobs of a particular
type can be queued.

This commit makes the previously hard-coded limit associated with
transactions configurable by the server's operator. Servers that
have increased memory capacity or which expect to see an influx
of transactions can increase the number of transactions their
server will be able to queue.

This commit fixes #3556.
2020-09-01 16:39:00 -07:00
Nathan Nichols
b1d47c65d4 Add public_key to the "vl" method response:
The "/vl" HTTP endpoint can be used to request a particular
UNL from a rippled instance.

This commit, if merged, includes the public key of the requested
list in the response.

This commit fixes #3392
2020-09-01 16:39:00 -07:00
Devon White
271e79095b Consolidate programmer documentation on sharding 2020-09-01 16:39:00 -07:00
Devon White
6c268a3e9c Allow multiple paths for shard storage:
* Distinguish between recent and historical shards
* Allow multiple storage paths for historical shards
* Add documentation for this feature
* Add unit tests
2020-09-01 16:39:00 -07:00
Elliot Lee
e5ff70f606 Add pull request template
Based on the PR template used by xpring-eng:

https://raw.githubusercontent.com/xpring-eng/.github/master/.github/pull_request_template.md
2020-09-01 16:39:00 -07:00
Gregory Tsipenyuk
801b1580f5 Improve handling of RPC ledger_index argument:
Some RPC commands return `ledger_index` as a quoted numeric
string. This change allows the returned value to be directly
copied and used for follow-on RPC commands.

This commit fixes #3533
2020-09-01 16:39:00 -07:00
Gregory Tsipenyuk
707868be33 Implement sticky DNS resolution for validator list retrieval:
When attempting to load a validator list from a configured
site, attempt to reuse the last IP that was successfully
used if that IP is still present in the DNS response.

Otherwise, randomly select an IP address from the list of
IPs provided by the DNS system.

This commit fixes #3494.
2020-09-01 10:37:19 -07:00
Gregory Tsipenyuk
9b9f34f881 Optimize relaying of validation and proposal messages:
With few exceptions, servers will typically receive multiple copies
of any given message from its directly connected peers. For servers
with several peers this can impact the processing latency and force
it to do redundant work. Proposal and validation messages are often
relayed with extremely high redundancy.

This commit, if merged, introduces experimental code that attempts
to optimize the relaying of proposals and validations by allowing
servers to instruct their peers to "squelch" delivery of selected
proposals and validations. Servers making squelching decisions by
a process that evaluates the fitness and performance of a given
server and randomly selecting a subset of the best candidates.

The experimental code is presently disabled and must be explicitly
enabled by server operators that wish to test it.
2020-09-01 09:07:32 -07:00
Scott Schurr
7724cca384 Implement enhanced Ticket support:
Tickets are a mechanism to allow for the "out-of-order" execution of
transactions on the XRP Ledger.

This commit, if merged, reworks the existing support for tickets and
introduces support for 'ticket batching', completing the feature set
needed for tickets.

The code is gated under the newly-introduced `TicketBatch` amendment
and the `Tickets` amendment, which is not presently active on the
network, is being removed.

The specification for this change can be found at:
https://github.com/xrp-community/standards-drafts/issues/16
2020-09-01 08:58:57 -07:00
manojsdoshi
01bd5a2646 Set version to 1.6.0 2020-08-18 15:33:02 -07:00
manojsdoshi
2b44868373 Set version to 1.6.0-rc3 2020-08-07 13:45:44 -07:00
seelabs
f79a4a8cdb Revert "Remove CryptoConditionsSuite stub amendment:"
This reverts commit d3798f6290.
2020-08-07 13:45:22 -07:00
seelabs
7bb6b75f3c Use the correct root hash for the tx tree 2020-08-07 11:17:48 -07:00
Nik Bougalis
e5d17a9452 Set version to 1.6.0-rc2 2020-08-06 10:14:03 -07:00
manojsdoshi
dbd5f0073e Revert support for deterministic shards:
Commit 4dc08f8202 introduced support for
deterministic shards, which makes the sharding functionality provided
by rippled more useful.

After merging, several opportunities for further improvements to the
deterministic sharding implementation were identified and a significant
increase int memory usage during shard finalization was detected.

Because of these issues, the commit is being reverted and the feature is
being rolled back. It will be reintroduced in a future release.
2020-08-06 10:11:55 -07:00
Peng Wang
12c0e8148b Improve naming of fields associated with the NegativeUNL 2020-08-06 10:05:43 -07:00
Edward Hennis
72a9a2bdbb Reorder the Travis build:
* Builds Windows dependencies first.
* Builds ALL OSs in the last stage.
* Fix the MacOS builds.
* Windows dependency stages are allowed to fail so ALL configurations will
  attempt to build. Windows builds will probably fail if dependencies fail
  (caching may allow them to succeed), but they will at least be attempted.
* Remove broken AppVeyor config file, so it stops trying.
2020-08-06 10:05:43 -07:00
seelabs
80860fa8f5 Add preliminary support for Boost 1.74 2020-08-06 10:05:43 -07:00
seelabs
8cf542abb0 Fix memory management issues with checkpointers:
The checkpointer class had assumed that the database would exist for the
lifetime of the application. This is no long true. These changes resolve bugs
involving dangling pointers.
2020-08-06 10:05:43 -07:00
seelabs
a931b020b5 Switch to updated date library exception handling:
Without defining `HAS_UNCAUGHT_EXCEPTIONS` the date library would use the
deprecated `uncaught_exception` which would generate a warning.
2020-08-06 09:56:04 -07:00
seelabs
d317060ae4 Fix a race condition with TrustedPublisherServer:
There was a race condition in `on_accept` where the object's destructor
could run while `on_accept` was called.

This patch ensures that if `on_accept` is called then the object remains
valid for the duration of the call.
2020-08-06 09:56:04 -07:00
seelabs
eee07a4f96 Add missing cstdint include 2020-08-06 09:55:37 -07:00
manojsdoshi
7b048b423e Set version to 1.6.0-rc1 2020-07-08 13:40:03 -07:00
manojsdoshi
a459206848 Set version to 1.6.0-b9 2020-07-08 13:38:27 -07:00
Howard Hinnant
3a3b0b4c14 Modify health check API
* Fixes #3486
* load factor computation normalized by load_base.
* last validated ledger age set to -1 while syncing.
* Return status changed:
*    healthy  -> ok
*    warning  -> service_unavailable
*    critical -> internal_server_error
2020-07-08 13:35:07 -07:00
manojsdoshi
de0c527387 Set version to 1.6.0-b8 2020-06-30 12:40:51 -07:00
manojsdoshi
b54453d1a7 Fix a build issue caused by pip3 being unavailable 2020-06-30 09:15:37 -07:00
Peng Wang
706ca874b0 Implement negative UNL functionality:
This change can help improve the liveness of the network during periods of network
instability, by allowing the network to track which validators are presently not online
and to disregard them for the purposes of quorum calculations.
2020-06-30 09:15:37 -07:00
Peng Wang
51bd4626b1 Implement version upgrade warning:
If the 'HardenedValidations' amendment is enabled, this commit will
track the version of the software that validators embed in their
validations.

If a server notices that at least 60% of the validators on its UNL
are running a newer version than it is running, it will periodically
print an informational message, reminding the operator to check for
update.
2020-06-30 09:15:37 -07:00
mDuo13
cf6f40ea8f Deprecate unused/obsolete error codes:
The tecUNFUNDED code is actively used when attempting to create payment
channels; the messages incorrectly list it as deprecated.

Meanwhile, the tecUNFUNDED_ADD code actually is an unused legacy code,
dating back to when there was a WalletAdd transactor. The terLAST and
terFUNDS_SPENT codes are also unused legacy codes.

Engine result messages are not part of the binary format and are
documented as subject to change without notice, so this should not
require an amendment nor a new API version.

Align error code table for human readability.
2020-06-30 08:52:18 -07:00
seelabs
d3798f6290 Remove CryptoConditionsSuite stub amendment:
The amendment was partially complete, included no functional code
and, even if activated, it would result in no changes to transaction
proessing. Despite this, removing the amendment is the prudent course
of action and avoids the possibility of an accidental activation.

If additional cryptoconditions are implemented, they will be each
assigned a new, unique amendment code.
2020-06-30 08:52:18 -07:00
Rome Reginelli
4e33a1abf7 crawl_shards comment fix
Corrects the public_key parameter name in the comment.
See https://github.com/ripple/xrpl-dev-portal/pull/854 for context.
2020-06-30 08:52:18 -07:00
Miguel Portilla
86e8f2e232 Add Shard Family 2020-06-30 08:52:18 -07:00
Miguel Portilla
91e857874f Improve LedgerMaster shard acquisition 2020-06-30 08:52:17 -07:00
Miguel Portilla
8f50fd051e Use NuDB version 2.0.3 2020-06-30 08:52:17 -07:00
p2peer
94e8e94750 Add support for deterministic database shards (#2688):
This commit, if merged, adds support to allow multiple indepedent nodes to
produce a binary identical shard for a given range of ledgers. The advantage
is that servers can use content-addressable storage, and can more efficiently
retrieve shards by downloading from multiple peers at once and then verifying
the integrity of a shard by cross-checking its checksum with the checksum
other servers report.
2020-06-30 08:52:17 -07:00
Edward Hennis
54ece72b62 Update cmake so that rippled can build as a submodule 2020-06-25 19:47:20 -07:00
Edward Hennis
4702c8b591 Improve online_delete configuration and DB tuning:
* Document delete_batch, back_off_milliseconds, age_threshold_seconds.
* Convert those time values to chrono types.
* Fix bug that ignored age_threshold_seconds.
* Add a "recovery buffer" to the config that gives the node a chance to
  recover before aborting online delete.
* Add begin/end log messages around the SQL queries.
* Add a new configuration section: [sqlite] to allow tuning the sqlite
  database operations. Ignored on full/large history servers.
* Update documentation of [node_db] and [sqlite] in the
  rippled-example.cfg file.

Resolves #3321
2020-06-25 19:46:43 -07:00
Mark Travis
00702f28c2 Improve reporting of ledger age in server_info 2020-06-25 19:46:43 -07:00
Gregory Tsipenyuk
df29e98ea5 Improve amendment processing and activation logic:
* The amendment ballot counting code contained a minor technical
  flaw, caused by the use of integer arithmetic and rounding
  semantics, that could allow amendments to reach majority with
  slightly less than 80% support. This commit introduces an
  amendment which, if enabled, will ensure that activation
  requires at least 80% support.
* This commit also introduces a configuration option to adjust
  the amendment activation hysteresis. This option is useful on
  test networks, but should not be used on the main network as
  is a network-wide consensus parameter that should not be
  changed on a per-server basis; doing so can result in a
  hard-fork.

Fixes #3396
2020-06-25 19:46:43 -07:00
Nik Bougalis
fe9922d654 Improve compression support:
* Optimize parsing of compressed message headers
* Enforce protocol-defined message size maxima
* Update comments
2020-06-25 19:46:43 -07:00
Nik Bougalis
362a017eee Cleanup SHAMap and simplify interfaces:
* Improve error reporting (more readable exception messages)
* Reduce function complexity (split oversized function to smaller pieces)
* Reduce code duplication
* Reduce buffer copying
2020-06-25 19:46:43 -07:00
Nik Bougalis
e8f3525226 Improve Slice:
Slice should, eventually, be replaced by std::string_view<std::uint8_t>
so begin adding some helpful functions to align its interface.
2020-06-25 19:46:43 -07:00
Howard Hinnant
1067086f71 Consolidate "Not Synced" error messages:
Work on a version 2 of the XRP Network API has begun. The new
API returns:

* `notSynced` in place of `noClosed`, `noCurrent`, and `noNetwork`;
* `invalidParams` in place of `lgrIdxInvalid`.

The new version 2 API cannot be selected yet, as it remains a work
in progress.

Fixes #3269
2020-06-25 19:45:59 -07:00
Howard Hinnant
0214d83aa5 Improve handling of empty buffer in varint parsing (RIPD-1683) 2020-06-25 19:45:59 -07:00
CJ Cobb
d69a902876 Add comments to protobuf files 2020-06-25 19:09:19 -07:00
manojsdoshi
f43aeda49c Set version to 1.6.0-b7 2020-05-29 17:55:01 -07:00
Yusuf Sahin HAMZA
27484f78a9 Use libarchive version 3.4.3 2020-05-29 17:54:19 -07:00
p2peer
93bf77bdec Unit tests for database shards 2020-05-29 17:54:19 -07:00
Miguel Portilla
728651b5d5 Use LZ4_decompress_safe 2020-05-29 17:54:19 -07:00
Miguel Portilla
fb74eefa9e Update LZ4 library to 1.9.2 2020-05-29 17:54:19 -07:00
manojsdoshi
853c964194 Fix a build issue caused by the latest Docker version:
* The latest docker version is not supported by
   artifactory causing the package build to fail.
   Setting the docker version to 19.03.8 to fix the build.
2020-05-29 17:54:19 -07:00
Gregory Tsipenyuk
645c06764b Update the default port for [ips] and [ips_fixed]:
If a port number is not specified in the [ips] or [ips_fixed]
blocks, automatically add the new default peer port which was
registered with IANA: 2459. Also use 2459 if no port is specified
with manually using the `connect` command; previously it was
using 6561, which could have resulted in spurious failures.
This commit, if merged, fixes #2861.
2020-05-29 17:52:49 -07:00
Howard Hinnant
2d23e7bd18 Use std::size in place of std::extent
* Fixes RIPD-269
2020-05-29 14:03:08 -07:00
Howard Hinnant
0290d0b82c Create health_check rpc
* Gives a summary of the health of the node:
  Healthy, Warning, or Critical

* Last validated ledger age:
  <7s is Healthy,
  7s to 20s is Warning
  > 20s is Critcal

* If amendment blocked, Critical

* Number of peers:
  > 7 is Healthy
  1 to 7 is Warning
  0 is Critical

* server state:
  One of full, validating or proposing is Healthy
  One of syncing, tracking or connected is Warning
  All other states are Critical

* load factor:
  <= 100 is Healthy
  101 to 999 is Warning
  >= 1000 is Critical

* If not Healthy, info field contains data that is considered not
  Healthy.

Fixes: #2809
2020-05-29 14:02:08 -07:00
manojsdoshi
3d86b49dae Set version to 1.6.0-b6 2020-05-26 18:48:13 -07:00
John Freeman
0b9e935806 Find date::date in CMake package configuration file 2020-05-26 18:41:30 -07:00
Mark Travis
17412d17e2 Fix historical ledger acquisition when using online deletion:
Commit e257a22 introduced changes in the logic used to acquire historical
ledgers. The logic could cause historical ledgers to be acquired only since
the last online deletion interval instead of the configured value to allow
deletion.
2020-05-26 18:39:48 -07:00
seelabs
99f5193699 Disable formatting operator&:
Clang 10.0.0 and 10.0.1 disagree on formatting operator&
Disable until we upgrade to clang 11.
2020-05-26 18:39:36 -07:00
seelabs
328e42ad42 Minor cleanups:
* Make sure variables are always initialized
* Use lround instead of adding .5 and casting
* Remove some unneeded vars
* Check for null before calling strcmp
* Remove redundant if conditions
* Remove make_TxQ factory function
2020-05-26 18:39:24 -07:00
seelabs
6d28f2a8d9 Cleanup code using move semantics 2020-05-26 18:39:16 -07:00
seelabs
421417ab07 Fixes for gcc 10 2020-05-26 18:39:06 -07:00
Scott Schurr
68494a308e AmendmentTable improvements 2020-05-26 18:38:58 -07:00
CJ Cobb
16f79d160a Add delivered amount to GetAccountTransactionHistory responses 2020-05-26 18:38:51 -07:00
John Freeman
8f984042f4 Try to fix usage of pkg-config for static linking 2020-05-26 18:37:52 -07:00
Edward Hennis
eb1a699c5a Fix typo in error message 2020-05-26 18:37:45 -07:00
Devon White
ac766ec0eb Introduce ShardArchiveHandler improvements:
* Improve documentation
* Make the ShardArchiveHandler rather than the DatabaseShardImp perform
  LastLedgerHash verification for downloaded shards
* Remove ShardArchiveHandler's singleton implementation and make it an
  Application member
* Have the Application invoke ShardArchiveHandler initialization
  instead of clients
* Add RecoveryHandler as a ShardArchiveHandler derived class
* Improve commenting
2020-05-26 18:37:01 -07:00
Devon White
21340a1c1e Validate LastLedgerHash for downloaded shards:
* Add documentation for shard validation
* Retrieve last ledger hash for imported shards
* Verify the last ledger hash in Shard::finalize
* Limit last ledger hash retrieval attempts for imported shards
* Use a common function for removing failed shards
* Add new ShardInfo::State for imported shards
2020-05-26 18:36:54 -07:00
Nik Bougalis
c594f3b0cf Reduce visibility of retired amendment identifiers:
Identifiers for retired amendments should not generally be used
in the codebase.

This commit reduces their visibility down to one translation
unit and marks them as unused and deprecated to prevent
accidental reuse.
2020-05-26 18:36:15 -07:00
Nik Bougalis
268e28a278 Tune relaying of untrusted proposals & validations:
In deciding whether to relay a proposal or validation, a server would
consider whether it was issued by a validator on that server's UNL.

While both trusted proposals and validations were always relayed,
the code prioritized relaying of untrusted proposals over untrusted
validations. While not technically incorrect, validations are
generally more "valuable" because they are required during the
consensus process, whereas proposals are not, strictly, required.

The commit introduces two new configuration options, allowing server
operators to fine-tune the relaying behavior:

The `[relay_proposals]` option controls the relaying behavior for
proposals received by this server. It has two settings: "trusted"
and "all" and the default is "trusted".

The `[relay_validations]` options controls the relaying behavior for
validations received by this server. It has two settings: "trusted"
and "all" and the default is "all".

This change does not require an amendment as it does not affect
transaction processing.
2020-05-26 18:36:06 -07:00
Nik Bougalis
ca664b17d3 Improve handling of sfLedgerSequence field:
The sfLedgerSequence field is designated as optional in the object
template but it is effectively required and validations which do not
include it were, correctly, rejected.

This commit migrates the check outside of the peer code and into the
constructor used for validations being deserialized for the network.

Furthermore, the code will generate an error if a validation that is
generated by a server does not include the field.
2020-05-26 18:35:58 -07:00
Nik Bougalis
3936110c8d Use boost::circular_buffer:
The existing code used std::deque along with a size check to constrain the
size of a buffer and, effectively, "hand rolled" a circular buffer. This
change simply migrates directly to boost::circular_buffer.
2020-05-26 18:35:49 -07:00
Nik Bougalis
9f91870b1c Adjust frequency of mtENDPOINTS messages 2020-05-26 18:35:39 -07:00
Nik Bougalis
97712107b7 Set version to 1.6.0-b5 2020-05-05 16:05:24 -07:00
Carl Hua
d9fa148688 Revert "Add PR automation for project boards"
This reverts commit cd78ce3118.
2020-05-05 16:05:23 -07:00
Miguel Portilla
d88a7c73b4 Improve online delete backend locking 2020-05-05 16:05:23 -07:00
Scott Schurr
894d3463ce Extend unit testing of account_tx with deleted account:
The unit test now verifies that if an account is not present in the
starting account_tx ledger, account_tx still iterates down and finds
the transaction that deletes the account (and earlier transactions).
2020-05-05 16:05:23 -07:00
John Freeman
5b5226d518 Cleanup the 'PeerSet' hierarchy:
This commit introduces no functional changes but cleans up the
code and shrinks the surface area by removing dead and unused
code, leveraging std:: alternatives to hand-rolled code and
improving comments and documentation.
2020-05-05 16:05:23 -07:00
Alloy Networks
d025f3fb28 Add descriptive comments to 'getRippledInfo.sh':
The script, when invoked by a server operator can collect information
useful for debugging, while attempting to redact potentially sensitive
data.

It contained no explanation or other exposition to allow people who
look at the file but aren't familiar with shell scripts to understand
its purpose.
2020-05-05 16:05:23 -07:00
Markus Alvila
11be30dd40 Correct typo in comment 2020-05-05 16:05:23 -07:00
Elliot Lee
4fad421c8a Correct typos in SECURITY.md 2020-05-05 16:05:23 -07:00
seelabs
57b3543e7b Support boost 1.73 2020-05-05 16:05:23 -07:00
Kirill Fomichev
a00543b6bc Fix docs about misconfigured neighbor port 2020-05-05 16:05:23 -07:00
Kirill Fomichev
dbd25f0e32 Remove excessive redirect call on PeerManager 2020-05-05 16:05:23 -07:00
Nik Bougalis
62a3f33d72 Remove the built-in "sustain" watchdog:
The built-in watchdog is simplistic and can, sometimes, cause problems
especially on systems that have the ability to automatically start and
monitor processes.

This commit removes the sustain system entirely, changes the handling
of the SIGTERM signal to properly terminate the process and improves
the error message reported to the user when the command line used to
start `rippled` is incorrect and malformed.
2020-05-05 16:05:23 -07:00
Nik Bougalis
74f9edef07 Prefer keylets instead of naked hashes:
Entries in the ledger are located using 256-bit locators. The locators
are calculated using a wide range of parameters specific to the entry
whose locator we are calculating (e.g. an account's locator is derived
from the account's address, whereas the locator for an offer is derived
from the account and the offer sequence.)

Keylets enhance type safety during lookup and make the code more robust,
so this commit removes most of the earlier code, which used naked
uint256 values.
2020-05-05 16:05:22 -07:00
Nik Bougalis
dbee3f01b7 Clean up and modernize code:
This commit removes obsolete comments, dead or no longer useful
code, and workarounds for several issues that were present in older
compilers that we no longer support.

Specifically:

- It improves the transaction metadata handling class, simplifying
  its use and making it less error-prone.
- It reduces the footprint of the Serializer class by consolidating
  code and leveraging templates.
- It cleanups the ST* class hierarchy, removing dead code, improving
  and consolidating code to reduce complexity and code duplication.
- It shores up the handling of currency codes and the conversation
  between 160-bit currency codes and their string representation.
- It migrates beast::secure_erase to the ripple namespace and uses
  a call to OpenSSL_cleanse instead of the custom implementation.
2020-05-05 16:05:22 -07:00
Nik Bougalis
6c72d5cf7e Improve loading of validator tokens (RIPD-1687):
A deliberately malformed token can cause the server to crash during
startup. This is not remotely exploitable and would require someone
with access to the configuration file of the server to make changes
and then restart the server.

Acknowledgements:
Guido Vranken for responsibly disclosing this issue.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.

Ripple is generously sponsoring a bug bounty program for the
rippled project. For more information please visit:

    https://ripple.com/bug-bounty
2020-05-01 12:55:12 -07:00
Nik Bougalis
2827de4d63 Report the server version in published validations:
Currently there is no mechanism for a validator to report the
version of the software it is currently running. Such reports
can be useful for those who are developing network monitoring
dashboards and server operators in general.

This commit, if merged, defines an encoding scheme to encode
a version string into a 64-bit unsigned integer and adds an
additional optional field to validations.

This commit piggybacks on "HardenedValidations" amendment to
determine whether version information should be propagated
or not.

The general encoding scheme is:

XXXXXXXX-XXXXXXXX-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY

X: 16 bits identifying the particular implementation
Y: 48 bits of data specific to the implementation

The rippled-specific format (implementation ID is: 0x18 0x3B) is:

00011000-00111011-MMMMMMMM-mmmmmmmm-pppppppp-TTNNNNNN-00000000-00000000

    M: 8-bit major version (0-255)
    m: 8-bit minor version (0-255)
    p: 8-bit patch version (0-255)
    T: 11 if neither an RC nor a beta
       10 if an RC
       01 if a beta
    N: 6-bit rc/beta number (1-63)
2020-05-01 12:55:12 -07:00
Nik Bougalis
381606aba2 Harden validations:
This commit introduces the "HardenedValidations" amendment which,
if enabled, allows validators to include additional information in
their validations that can increase the robustness of consensus.

Specifically, the commit introduces a new optional field that can
be set in validation messages can be used to attest to the hash of
the latest ledger that a validator considers to be fully validated.

Additionally, the commit leverages the previously introduced "cookie"
field to improve the robustness of the network by making it possible
for servers to automatically detect accidental misconfiguration which
results in two or more validators using the same validation key.
2020-05-01 12:55:11 -07:00
Nik Bougalis
567e42e071 Deprecate 'Time to Live' fields 2020-04-30 20:05:09 -07:00
manojsdoshi
2bf3b194fa Set version to 1.6.0-b4 2020-04-27 16:17:15 -07:00
manojsdoshi
444ea56182 Adding support to build rippled packages for ubuntu 20.04 2020-04-27 16:16:56 -07:00
CJ Cobb
d797589164 Add README for gRPC protobuf folder 2020-04-23 18:11:12 -07:00
Edward Hennis
1577c775b3 Close download socket before result is passed to the callback:
* Specifically relevent to the processing of UNL downloads via
  ValidatorSite.
2020-04-23 18:10:21 -07:00
Edward Hennis
3bf0b724a3 Adjust timeouts in Validator Site tests:
* Prevents spurious failures due to DNS resolution.
2020-04-23 18:10:12 -07:00
Miguel Portilla
bd8dbb87b6 Update to RocksBD 6.7.3 2020-04-23 18:09:45 -07:00
Carl Hua
cd78ce3118 Add PR automation for project boards 2020-04-23 18:09:08 -07:00
manojsdoshi
023f5704d0 Set version to 1.6.0-b3 2020-04-23 14:27:29 -07:00
manojsdoshi
123e94c603 Fix a build issue involving Ubuntu Docker containers:
Upgrade the build pipeline to use ubuntu 19.10 container
in order for the build pipeline to successfully complete builds.
2020-04-23 14:27:12 -07:00
John Freeman
156dc2ec4c Add GitHub Action for checking w/ clang-format-10 2020-04-23 10:15:36 -07:00
John Freeman
b7b7e098be Add .git-blame-ignore-revs (requires Git >= 2.24) 2020-04-23 10:14:29 -07:00
Pretty Printer
50760c6935 Format first-party source according to .clang-format 2020-04-23 10:02:04 -07:00
John Freeman
65dfc5d19e Prepare code for formatting
- Add missing `#include` in `ripple/core/JobTypeInfo.h`
- Protect version string from clang-format in
  `ripple/protocol/impl/BuildInfo.cpp`.
  `Builds/CMake/RippledVersion.cmake` searches for this line by pattern.
2020-04-23 09:53:49 -07:00
Nik Bougalis
020b285808 Set version to 1.6.0-b2 2020-04-14 19:42:50 -07:00
Nik Bougalis
bdd22e4d51 Improve reporting of missing node exceptions 2020-04-14 19:42:50 -07:00
Nik Bougalis
b7631d2a28 Correct a typo that could result in a nullptr dereference 2020-04-14 19:42:50 -07:00
Nik Bougalis
284ed38471 Reduce calls to std::random_device:
Existing per-thread PRNGs are individually initialized using calls
to std::random_device.

If merged, this commit will use a single PRNG, initialized from
std::random_device on startup, to seed the thread-specific PRNGs.

Acknowledgements:
    Thomas Snider, who reported this issue to Ripple on April 8, 2020.
2020-04-14 19:42:49 -07:00
Nik Bougalis
b96ef6adb8 Add SECURITY.md to the repository 2020-04-14 19:42:49 -07:00
manojsdoshi
ce6b427202 Fix a build issue involving Ubuntu Docker containers:
The package 'apt-transport-https' needs to be downloaded and installed
in order for the build pipeline to successfully complete builds.
2020-04-14 19:42:42 -07:00
John Freeman
858e93c7f8 Improve the Doxygen Workflow
* Use official build environment Docker image
* Update documentation and Doxygen website URLs
2020-04-14 19:42:42 -07:00
Miguel Portilla
6477bdf3e8 Fix division by zero with shards file stats 2020-04-14 19:42:42 -07:00
Miguel Portilla
ce5f240551 Fix invalid shard removal 2020-04-14 19:42:42 -07:00
Gábor Lipták
1c3c69f8b5 Update Travis to bionic 2020-04-14 19:17:13 -07:00
Mo Morsi
be2652544b Add ledger_cleaner command to rippled cmd line help 2020-04-14 19:17:13 -07:00
Mo Morsi
f155eaff4b Unit test for memo 2020-04-14 19:17:13 -07:00
Scott Schurr
67981f002f Reduce strand re-execute log message severity to warning:
Historically strand re-execute log messages have been treated as
errors.  However in the vast majority of cases these log messages
are caused by well understood mechanics in the payment engine.
So usually these log messages should be treated as warnings.
2020-04-14 19:17:13 -07:00
Scott Schurr
0d83223445 Remove conditionals for fix1201 enabled 14Nov2017 2020-04-09 11:42:34 -07:00
Scott Schurr
9f8d648514 Remove conditionals for fix1512 enabled 14Nov2017 2020-04-09 11:42:34 -07:00
Scott Schurr
3d3b6d85cd Remove conditionals for fix1523 enabled 14Nov2017 2020-04-09 11:42:34 -07:00
Scott Schurr
8cf7c9548a Remove conditionals for fix1528 enabled 14Nov2017 2020-04-09 11:42:34 -07:00
Scott Schurr
323dbc7962 Remove conditionals for featureSortedDirectories enabled 14Nov2017 2020-04-09 11:42:34 -07:00
Scott Schurr
46a76fb318 Remove conditionals for featureEnforceInvariants enabled 07Jul2017 2020-04-09 11:42:34 -07:00
Scott Schurr
a6246b0baa Remove conditionals for fix1373 enabled 07Jul2017 2020-04-09 11:42:34 -07:00
Scott Schurr
c8282795ef Remove conditionals for featureEscrow enabled 31Mar2017 2020-04-09 09:31:06 -07:00
Scott Schurr
e93a44fe9b Remove conditionals for fix1368 enabled 31Mar2017 2020-04-09 09:31:06 -07:00
Scott Schurr
3e870866e0 Remove conditionals for featurePayChan enabled 31Mar2017 2020-04-09 09:31:06 -07:00
Scott Schurr
78d771af36 Remove conditionals for featureTickSize enabled 21Feb2017 2020-04-09 09:31:06 -07:00
Scott Schurr
6bb9dd22e0 Remove conditionals for featureCryptoConditions enabled 03Jan2017 2020-04-09 09:31:06 -07:00
Scott Schurr
1661c84af6 Remove unused featureCompareFlowV1V2 2020-04-09 09:31:06 -07:00
manojsdoshi
4f422f6f39 Setting version to 1.6.0-b1 2020-04-07 16:26:40 -07:00
manojsdoshi
393ca87572 Change the location for the project signer public keys:
The automated build system only builds packages signed with a list of
approved keys. This is a security measure to prevent someone who gains
push access to the repository from producing potentially malicious
packages that are signed by Ripple's trusted private keys.

Moving this list to the new location makes it easy to add and delete
new keys to the list.
2020-04-07 16:25:19 -07:00
Howard Hinnant
f4c56cbd53 Update SHAMap Documentation 2020-04-07 16:25:09 -07:00
Howard Hinnant
9470558ecc Remove all uses of the name scoped_lock
*  scoped_lock is now a std name with subtly different semantics
   compared to lock_guard.  Namely it can be used to lock 0 or
   more mutexes.  This is valuable, but can also be accidentally
   used to lock 0 mutexes when 1 was intended, creating a
   run-time error.

   Therefore, if and when we use scoped_lock, extra care needs to
   be taken in reviewing that code to ensure it doesn't
   accidentally lock 0 mutexes when 1 was intended.  To aid in
   such careful reviewing, the use of the name scoped_lock should
   be limited to those cases where the number of mutexes is not
   exactly one.
2020-04-07 16:25:09 -07:00
Howard Hinnant
f22fcb3b2a Rename canonicalize into two functions:
* canonicalize_replace_cache
* canonicalize_replace_client

Now it is clear at the call site that if there are
duplicate copies of the data between the cache and
the caller, which copy gets replaced.

Additionally data parameter is now const-correct.
If it is not going to be replaced (canonicalize_replace_cache),
then the shared_ptr to the client data is const.
2020-04-07 16:25:09 -07:00
Mark Travis
e257a226f3 Maintain history back to the earliest persisted ledger:
This makes behavior consistent with configurations both with and
without online delete.
2020-04-07 16:25:09 -07:00
rabbit
a4e9878790 Document the 'devnet' network identifier setting:
* The [network_id] option allows three string values:
- main: the XRP Ledger
- testnet: the Testnet operated by Ripple.
- devnet: the development test network operated by Ripple.
2020-04-07 16:23:34 -07:00
John Freeman
25b13978e7 Fix unity build 2020-04-07 11:31:23 -07:00
John Freeman
3e9cff9287 Fix Doxygen build 2020-04-06 17:28:53 -07:00
Gregory Tsipenyuk
758a3792eb Add protocol message compression support:
* Peers negotiate compression via HTTP Header "X-Offer-Compression: lz4"
* Messages greater than 70 bytes and protocol type messages MANIFESTS,
  ENDPOINTS, TRANSACTION, GET_LEDGER, LEDGER_DATA, GET_OBJECT,
  and VALIDATORLIST are compressed
* If the compressed message is larger than the uncompressed message
  then the uncompressed message is sent
* Compression flag and the compression algorithm type are included
  in the message header
* Only LZ4 block compression is currently supported
2020-04-06 17:22:59 -07:00
seelabs
ade5eb71cf Fix unneeded copies in range some range for loops:
clang 10 warns about an unneeded copy for these range for
loops (range-loop-construct warnings)
2020-04-06 17:22:59 -07:00
seelabs
d097819c52 Check XRP endpoints for circular paths (RIPD-1781):
The payment engine restricts payment paths so two steps do not input the
same Currency/Issuer or output the same Currency/Issuer. This check was
skipped when the path started or ended with XRP. An example of a path
that was incorrectly accepted was: XRP -> //USD -> //XRP -> EUR

This patch enables the path loop check for paths that start or end with
XRP.
2020-04-06 17:22:59 -07:00
Devon White
905a97e0aa Make ShardArchiveHandler downloads more resilient:
* Make ShardArchiveHandler a singleton.
* Add state database for ShardArchiveHandler.
* Use temporary database for SSLHTTPDownloader downloads.
* Make ShardArchiveHandler a Stoppable class.
* Automatically resume interrupted downloads at server start.
2020-04-06 17:22:47 -07:00
Miguel Portilla
cc452dfa9b Improve shard concurrency:
* Reduce lock scope on all public functions
* Use TaskQueue to process shard finalization in separate thread
* Store shard last ledger hash and other info in backend
* Use temp SQLite DB versus control file when acquiring
* Remove boost serialization from cmake files
2020-04-06 17:22:23 -07:00
Carl Hua
f00f263852 Set version to 1.5.0 2020-03-30 18:57:35 -04:00
Carl Hua
b54d672710 Set version to 1.5.0-rc3 2020-03-24 14:42:45 -05:00
Mark Travis
5047b89922 Allow non-root users to build packages. 2020-03-24 11:36:00 -07:00
seelabs
78ea75b116 Remove incorrect STAmount assert:
When computing rates for offers, an STAmount's value can be out of can be out of
range (before canonicalizing). There was an assert that could incorrectly fire
in some cases. This patch removes that assert.
2020-03-23 18:17:00 -07:00
Carl Hua
c0964832f5 Fix failing Travis CI Windows VS2019 build
The newest MSVC 19.25.28610.4 does not build rocksdb. During the
Travis CI Windows job, the vs_BuildTools.exe automatically
downloads the newest version of the compiler. This fix forces the
install of MSVC 19.24.28314.0 to build rocksdb.
2020-03-23 15:43:07 -07:00
Manoj doshi
4e3dc0e820 Set version to 1.5.0-rc2 2020-03-10 16:08:42 -07:00
CJ Cobb
2a2ad898b1 Set applied field to true even if engine result is not tesSUCCESS 2020-03-10 16:07:10 -07:00
Manoj doshi
b26ed948c8 Set version to 1.5.0-rc1 2020-03-06 15:12:51 -08:00
Manoj doshi
ba53722425 Set version to 1.5.0-b8 2020-03-06 15:10:49 -08:00
seelabs
4852daf695 Remove entry for fix1781 amendment:
The fix1781 amendment was incorrectly introduced during conflict
resolution and support for it is not included at this time. This commit
removes the definition of the amendment identifier.
2020-03-06 14:59:39 -08:00
Carl Hua
2d1be70ba2 Set version to 1.5.0-b7 2020-03-05 14:12:18 -05:00
seelabs
b126641f13 Clarify the code a comment applies to:
The `seq & 0xff` does not depend of `diff`, so does not belong in the block that
defines `diff`.
2020-03-05 14:10:31 -05:00
Nik Bougalis
6529d3e6f7 Use ephemeral validator public keys in lag ratchet:
A review of the lag ratchet code revealed that we were using
the long-term master public keys of trusted validators, when
we should have been using the ephemeral public keys instead.

As a result, the lag ratchet code would be effectively
inoperable.
2020-03-05 14:10:25 -05:00
Nik Bougalis
16c659b22c Simplify HashPrefix by converting to an enum 2020-03-05 14:10:20 -05:00
Mo Morsi
ec137044a0 Protocol Amendment: Always Require Fully-Canonical Signatures 2020-03-05 14:10:06 -05:00
John Freeman
053b6d9fd3 Include field name when diagnosing field not found:
Closes #3263
2020-03-05 14:08:07 -05:00
Mike Ellery
f76a5a3183 Set version to 1.5.0-b6
closes: #3226
closes: #3249
closes: #3260
closes: #3254
closes: #3258
closes: #3267
2020-02-25 19:22:00 -08:00
Elliot Lee
cfe59ee028 Fix typos in comments 2020-02-25 19:21:55 -08:00
John Freeman
20d4a39b63 Cooperate with CMAKE_MODULE_PATH set by Conan 2020-02-25 19:21:55 -08:00
CJ Cobb
e7ce3909d2 gRPC support for account_tx and tx
- Add support for all transaction types and ledger object types to gRPC
  implementation of tx and account_tx.

- Create common handlers for tx and account_tx.

- Remove mutex and abort() from gRPC server. JobQueue is stopped before
  gRPC server, with all coroutines executed to completion, so no need for
  synchronization.
2020-02-25 19:21:55 -08:00
Mike Ellery
acf4b78892 Add getRippledInfo to packages:
* include info gathering script in rippled linux pkgs
* modify script to output results to single markdown file
* rename info script
2020-02-25 19:21:55 -08:00
Mike Ellery
e9b3c58043 Use native cmake unity support:
* Remove hand-rolled unity sources
* unity unavailable for cmake 3.15 and earlier
* remove deprecated target configure option
2020-02-25 19:21:50 -08:00
seelabs
455105d3dc Limit account_channels to channels owned by the source account 2020-02-24 15:30:58 -08:00
seelabs
2dac108c57 Fix account_channels marker functionality 2020-02-24 15:30:58 -08:00
Nik Bougalis
0c6d380780 Set version to 1.5.0-b5 2020-02-12 11:30:45 -08:00
Edward Hennis
2c71802e38 Propagate validator lists (VLs or UNLs) over the peer network:
* Whenever a node downloads a new VL, send it to all peers that
  haven't already sent or received it. It also saves it to the
  database_dir as a Json text file named "cache." plus the public key of
  the list signer. Any files that exist for public keys provided in
  [validator_list_keys] will be loaded and processed if any download
  from [validator_list_sites] fails or no [validator_list_sites] are
  configured.
* Whenever a node receives a broadcast VL message, it treats it as if
  it had downloaded it on it's own, broadcasting to other peers as
  described above.
* Because nodes normally download the VL once every 5 minutes, a single
  node downloading a VL with an updated sequence number could
  potentially propagate across a large part of a well-connected network
  before any other nodes attempt to download, decreasing the amount of
  time that different parts of the network are using different VLs.
* Send all of our current valid VLs to new peers on connection.
  This is probably the "noisiest" part of this change, but will give
  poorly connected or poorly networked nodes the best chance of syncing
  quickly. Nodes which have no http(s) access configured or available
  can get a VL with no extra effort.
* Requests on the peer port to the /vl/<pubkey> endpoint will return
  that VL in the same JSON format as is used to download now, IF the
  node trusts and has a valid instance of that VL.
* Upgrade protocol version to 2.1. VLs will only be sent to 2.1 and
  higher nodes.
* Resolves #2953
2020-02-12 10:19:23 -08:00
Edward Hennis
3753840de5 Handle build directories for specific versions of gcc/clang:
* Example: gcc.Debug will use the the default version of gcc installed on the
  system. gcc-9.Debug will use version 9, regardless of the default. This will
  be most useful when the default is older than required or desired.
2020-02-12 10:19:23 -08:00
Edward Hennis
5ff23f8f31 Warn operators about upcoming unknown amendments:
* When an unknown amendment reaches majority, log an error-level
  message, and return a `warnings` array on all successful
  admin-level RPC calls to `server_info` and `server_state` with
  a message describing the problem, and the expected deadline.
* In addition to the `amendment_blocked` flag returned by
  `server_info` and `server_state`, return a warning with a more
  verbose description when the server is amendment blocked.
* Check on every flag ledger to see if the amendment(s) lose majority.
  Logs again if they don't, resumes normal operations if they did.

The intention is to give operators earlier warning that their
instances are in danger of being amendment blocked, which will
hopefully motivate them to update ahead of time.
2020-02-12 10:19:23 -08:00
Mike Ellery
4315913a5d Update RocksDB to 6.5:
* update EP and find package requirements
* minor protobuf/libarchive build fixes
* change travis release builds to nounity to
  ameliorate vm memory exhaustion.

FIXES: #3223, #3232
2020-02-12 10:19:23 -08:00
mbhandary
facb627786 Improve reporting of StateAccounting metrics:
* Metrics are now exported over insight.
* Fixes a minor bug that affected the reporting of gauges
2020-02-12 10:19:23 -08:00
mbhandary
b784988caf Added support for statsD Traffic Counts reporting 2020-02-12 10:19:23 -08:00
Nik Bougalis
172ead8221 Remove unused 'old Beast' code (RIPD-1652) 2020-02-11 19:14:24 -08:00
p2peer
d224d7e404 Switch to Boost.Beast for SSL detection (#3166) 2020-02-11 19:14:24 -08:00
p2peer
7ea78c8517 Remove workaround for waitable timers (#3166) 2020-02-11 19:14:24 -08:00
Howard Hinnant
ce589225dd Add time zone abbreviation to all time point streaming 2020-02-11 19:14:24 -08:00
Howard Hinnant
79896af275 Qualify tolower with std:: and remove obsolete comments
* Fixes RIPD-1759
2020-02-11 19:14:24 -08:00
Scott Schurr
2831ed0538 Add unittests for peer_reservations command line interface 2020-02-11 19:14:24 -08:00
Mo Morsi
e24dd2f954 Fix broken tests (missing api_version parameter) 2020-02-11 19:14:24 -08:00
Mo Morsi
60f0f5224d Accept 'strict' param in certain CLI options
account_info, owner_info, account_currencies
2020-02-10 23:36:31 -08:00
Mo Morsi
3578acaf0b Add new validator_info and manifest rpc methods
Returns local validator details and specified manifest information
respectively. Folded and rebased on latest develop
2020-02-10 23:36:31 -08:00
Carl Hua
2f9edf495e Add issue templates 2020-02-10 23:36:31 -08:00
Manoj doshi
cc4cefaa4b Set version to 1.5.0-b4 2020-01-30 15:56:43 -08:00
seelabs
81326a6d08 Remove old payment code 2020-01-30 15:38:14 -08:00
seelabs
9d3626fec5 Fix bug in qualityUpperBound:
* In and Out parameters were swapped when calculating the rate
* In and out qualities were not calculated correctly; use existing functions
  to get the qualities
* Added tests to check that theoretical quality matches actual computed quality
* Remove in/out parameter from qualityUpperBound
* Rename an overload of qualityUpperBound to adjustQualityWithFees
* Add fix amendment
2020-01-30 13:35:52 -08:00
Scott Schurr
ae707b814f Always enable fix1449 dated March 30, 2017 20:00:00 UTC 2020-01-30 13:22:54 -08:00
Scott Schurr
9c580b20a3 Always enable fix1443 dated March 12, 2017 01:00:00 UTC 2020-01-30 13:22:49 -08:00
Scott Schurr
70c21f90b9 Always enable fix1298 dated December 21, 2016 18:00:00 UTC 2020-01-30 13:22:43 -08:00
Scott Schurr
0964379a66 Always enable fix1274 dated September 30, 2016 17:00:00 UTC 2020-01-30 13:22:34 -08:00
Scott Schurr
a176f58a92 Always enable fix1141 dated July 1, 2016 17:00:00 UTC 2020-01-30 13:22:26 -08:00
Scott Schurr
fc0a082700 Remove STAmountSO::soTime and soTime2:
STAmount::soTime and soTime2 were time based "amendment like"
switches to control small changes in behavior for STAmount.
soTime2, which was the most recent, was dated Feb 27, 2016.
That's over 3 years ago.

The main reason to retain these soTimes would be to replay
old transactions.  The likelihood of needing to replay a
transaction from over three years ago is pretty low.  So it
makes sense to remove these soTime values.

In Flow_test the testZeroOutputStep() test is removed.  That
test started to fail when the STAmount::soTimes were removed.
I checked with the original author of the test.  He said
that the code being tested by that unit test has been removed,
so it makes sense to remove the test.  That test is removed.
2020-01-30 13:22:19 -08:00
Scott Schurr
f7fffee28d Warn on replay of ledgers from before Jan 1 2018 2020-01-30 13:22:08 -08:00
Scott Schurr
51ed7db002 Remove conditionals for featureTrustSetAuth enabled 19Jul2016 2020-01-30 13:20:01 -08:00
Scott Schurr
6e4945c56b Remove conditionals for featureMultiSign enabled 27Jun2016 2020-01-30 13:19:55 -08:00
Scott Schurr
c48be14f4f Remove comments about featureFeeEscalation enabled 19May2016 2020-01-30 13:19:48 -08:00
Jeroen Meulemeester
ecbbabfc3c Reduce log level of xrp/xrp offer payment step
This fixes #3181
2020-01-30 13:18:44 -08:00
Mark Travis
687cdbb78b Set start/stop logging for online_delete to appropriate verbosity level. 2020-01-30 13:18:35 -08:00
Mike Ellery
098086592b Resolve minor build issues:
- allow SOCI EP to build with vcpkg
 - fix container tagging in pkg build
 - correct libarchive name for win
2020-01-30 13:18:18 -08:00
Carl Hua
90d9ca901d Set version to 1.5.0-b3 2020-01-12 13:02:17 -05:00
Mike Ellery
eb016456a1 Streamline pkg and travis CI:
* use tagged containers for pkg build
* update build images
* continue to build container images in pipeline, but allow
  failure (non-block)
* limit travis macos cache
* add vs2019 windows to travis
* remove xcode 9 travis build
* remove clang5/6 from CI and update min version of Clang required in
  cmake
* break windows CI build into stages to reduce timeouts
* update datelib
* add if condition to travis builds to allow commit message to limit
  builds by platform
2020-01-12 07:26:19 -08:00
Devon White
cd9732b47a Change how fail_hard transactions are handled.
FIXES: #2847

* Transactions that are submitted with the fail_hard flag
  and that result in any TER code besides tesSUCCESS shall
  be neither queued nor held.

[FOLD] Keep tec results out of the open ledger when fail_hard:

* Improve TransactionStatus const correctness, and remove redundant
  `local` check
* Check open ledger tx count in fail_hard tests
* Fix some wrapping
* Remove duplicate test
2020-01-10 12:40:31 -08:00
CJ Cobb
7d867b806d Add gRPC support (#3127):
* add support for AccountInfo, Fee and Submit RPCs

* add partial support for Tx RPC (only supports Payments)
2020-01-10 12:31:24 -08:00
seelabs
761bb5744e Make XRPAmount constructor explicit:
Remove the implicit conversion from int64 to XRPAmount. The motivation for this
was noticing that many calls to `to_string` with an integer parameter type were
calling the wrong `to_string` function. Since the calls were not prefixed with
`std::`, and there is no ADL to call `std::to_string`, this was converting the
int to an `XRPAmount` and calling `to_string(XRPAmount)`.

Since `to_string(XRPAmount)` did the same thing as `to_string(int)` this error
went undetected.
2020-01-08 16:10:25 -08:00
Edward Hennis
e3b5b808c5 Add units to all fee calculations:
* Uses existing XRPAmount with units for drops, and a new TaggedFee for
  fee units (LoadFeeTrack), and fee levels (TxQ).
* Resolves #2451
2020-01-08 18:44:01 -05:00
Edward Hennis
1901b981f3 Convert tuning params to constexpr 2020-01-08 17:58:47 -05:00
Nik Bougalis
c624fc9b17 Set version to 1.5.0-b2 2020-01-02 13:29:22 -08:00
mbhandary
9294a07362 Report the validated and publish ledger ages over insight 2020-01-01 18:12:55 -08:00
mbhandary
9fcc30df89 Improve insight reporting of job queue timing:
Prior to this commit, the queue and execution times for individual jobs
were reported indepedently and could, potentially, be out of sync. This
change reports both values when either one of the exceeds the reporting
threshold.
2020-01-01 18:12:55 -08:00
Elliot Lee
1f5d9404d0 Update comments in TER.h 2020-01-01 18:12:55 -08:00
seelabs
5096cc1f5f Increase the deadlock detection timeout:
It's possible an overloaded job queue is causing false alarms on the deadlock
detector. Log a fatal message after 90s, declare a logic error after 600s.
2020-01-01 18:12:55 -08:00
Peng Wang
2aa11fa41d Support API versioning 2020-01-01 18:12:55 -08:00
p2peer
79e9085dd1 Augment "submit" command response:
If merged, this commit will report additional information in the
response to the submit command; this will make it easier for developers
to accurately track the status of transaction submission.

Fixes #2851
2020-01-01 18:12:38 -08:00
Mike Ellery
14f0234a26 Allow trailing comments in config file:
Treat all `#` characters in config files as comments (and remove)
*unless* the `#` is immediately preceded by `\`. Write a warning
to log file when trailing comments are found/ignored in the config
to let operators know that the treatment of trailing `#` has changed.

Fixes #3121
2020-01-01 18:12:38 -08:00
Howard Hinnant
a65c91a676 Backwards compatible workaround for boost 1.72 2019-12-30 20:20:34 -08:00
Nik Bougalis
607328e1a0 Improve the 'network_id' configuration option:
The 'network_id' option allows an administrator to specify to which
network they intend a server to connect. Servers can leverage this
information to optimize routing and prune automatically discovered
cross-network connections.

This commit will, if merged:

- add support for the devnet keyword, which corresponds to network ID #2;
- report the network ID, if one is configured, in server_info
2019-12-30 20:20:34 -08:00
Nik Bougalis
a96dc2ecea Remove unused configuration option 2019-12-30 20:20:33 -08:00
Nik Bougalis
3a77990781 Improve automatic I/O thread tuning algorithm 2019-12-30 20:20:33 -08:00
Nik Bougalis
4bb951d48e Fix node auto-configuration code:
The `node_size` configuration option is used to automatically
configure various parameters (cache sizes, timeouts, etc) for
the server.

A previous commit included changes that caused incorrect values
to be returned which can result in sub-optimal performance that
can manifest as difficulty syncing to the network, or increased
disk I/O and/or memory usage. The problem was introduced with
commit 66fad62e66.

This commit, if merged, fixes the code to ensure that the correct
values are returned and introduces a compile-time check to prevent
this issue from reoccurring.
2019-12-30 20:20:32 -08:00
Nik Bougalis
63503ee8f0 Improve platform detection and reduce includes:
The existing platform detection code was derived from the old Beast
library, which was, itself, derived from JUCE.

This commit removes that code and replaces it with the Boost.Predef
library which defines a consistent set of compiler, architecture,
operating system, library, and other version numbers.

For more on Boost.Predef, please see the Boost documentation. The
documentation for the current version as of this writing is at:
https://www.boost.org/doc/libs/1_71_0/doc/html/predef.html
2019-12-30 20:20:31 -08:00
Nik Bougalis
4a1148eb28 Set version to 1.5.0-b1 2019-11-28 09:46:20 -08:00
Nik Bougalis
f6916bfd42 Improve protocol-level handshaking protocol:
This commit restructures the HTTP based protocol negotiation that `rippled`
executes and introduces support for negotiation of compression for peer
links which, if implemented, should result in significant bandwidth savings
for some server roles.

This commit also introduces the new `[network_id]` configuration option
that administrators can use to specify which network the server is part of
and intends to join. This makes it possible for servers from different
networks to drop the link early.

The changeset also improves the log messages generated when negotiation
of a peer link upgrade fails. In the past, no useful information would
be logged, making it more difficult for admins to troubleshoot errors.

This commit also fixes RIPD-237 and RIPD-451
2019-11-28 09:46:17 -08:00
Scott Schurr
3ea525430e Improve Json::Value:
o Increase test coverage.
o Remove unused and/or broken interfaces.
2019-11-27 17:28:15 -08:00
John Northrup
fb0d065723 Move signing of deb to use Ubuntu 18.04 LTS 2019-11-27 17:28:15 -08:00
Devon White
47501b7f99 Provide additional info with txnNotFound errors.
* The `tx` command now supports min_ledger and max_ledger fields.
* If the requested transaction isn't found and these fields are
  provided, the error response indicates whether or not every
  ledger in the the provided range was searched.

This fixes #2924
2019-11-27 17:28:15 -08:00
ShangyanLi
11cf27e006 Skip validity check for tx command:
* historical tx retrieval no longer needs to pass current validity check;
* introduced additional RPC error code for database deserialization error.

This fixes issue #2597
2019-11-27 17:28:15 -08:00
Mark Travis
ade1afe1b0 Support multiple proxies in X-Forwarded-For header 2019-11-27 17:28:15 -08:00
Mo Morsi
6cda070fe0 Search for soci before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
9bd470f2c7 Search for sqlite3 before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
530f1939d6 Search for system snappy before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
e4ea3752ac Search for system secp256k1 before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
b728bf0d09 Search for system libarchive before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
194fb2b86d Use find_package to search for system lz4 before vendoring 2019-11-27 16:58:56 -08:00
Mike Ellery
ce5d901e6e Switch to official date lib repo for date.h 2019-11-27 16:58:56 -08:00
Howard Hinnant
14075630d4 Remove per-process hash seeding for gcc.
* gcc now seeds hardened_hash per instance.

* Other platforms were already doing this.
2019-11-27 16:58:56 -08:00
Miguel Portilla
5c1dd87fab Make class members journal const 2019-11-27 16:58:56 -08:00
Nik Bougalis
06c371544a Set version to 1.4.0 2019-11-26 10:35:54 -08:00
Nik Bougalis
ffa0f048f3 Set version to 1.4.0-rc2 2019-11-25 21:51:54 -08:00
Mike Ellery
22ca0041b8 Use signed artifacts in pkg pipeline 2019-11-25 21:51:12 -08:00
Manoj doshi
232975bfdb Set version to 1.4.0-rc1 2019-11-04 11:45:31 -08:00
Howard Hinnant
4e84868747 Document the minimum GCC version as 7 2019-11-04 11:39:19 -08:00
John Northrup
9e1ccb900e GPG Sign DEB and RPM packages generated by build pipeline (#3144)
* adding package signing steps for rpm and deb

* first spike at GPG signing with CI and containers

* refine ubuntu portion

* get correct gpg package version

* adding CentOS support

* fixing errors in installing gpg on ubuntu

* base64 decode the GPG key

* fixing line continuations

* revised package signing, looking for package artifacts

* add dpkg-sig to ubuntu image

* sign all deb packges

* add passphrase to GPG process

* repeat yo slef on dpkg

* sign all the rpm packages too

* install rpm-sign in the CentOS docker image

* loop through rpm files

* no need for PIN on GPG signing
2019-11-03 12:08:40 -06:00
Manoj doshi
98d55b988f Set version to 1.4.0-b8 2019-10-30 12:24:25 -07:00
Mike Ellery
e720a66076 Ensure permissions for initial config 2019-10-30 12:23:58 -07:00
Joseph Busch
7be7343e05 Make the HTTP response size const 2019-10-30 12:23:58 -07:00
seelabs
e26dd7bdfe Reject overlong encodings earlier 2019-10-30 12:23:58 -07:00
seelabs
906b9ae00b Don't use set in AccountObjects test:
Collecting the returned and expected values in sets only works if there are no
duplicates. The implementation is changed to use sorted vectors to fix this case.
2019-10-30 12:23:58 -07:00
Mo Morsi
15c5f9c111 Report consensus phase changes in the server subscription stream 2019-10-30 12:23:57 -07:00
Howard Hinnant
726dd69ab9 Make Env::AppBundle constructor exception safe
When the Env::AppBundle constructor throws an exception
it still needs to run ~AppBundle(), otherwise the JobQueue
isn't properly shut down.  Specifically the  JobQueue
can destruct without waiting on outstanding jobs in the
queue.

This change ensures that if Env::AppBundle constructor
throws, Env::AppBundle::~AppBundle() runs.

This fixes the unit test crash exposed by PR #3047.
2019-10-30 12:23:57 -07:00
Manoj doshi
41b2c80dde Set version to 1.4.0-b7 2019-10-18 16:45:06 -07:00
Mike Ellery
cd01502d17 Relax STTx test failure criterion:
FIXES: #3106

Different versions of protobuf produce subtly different
results when given invalid message payloads. This leads to
subtly different behavior when we try to deserialize these
invalid messages. As such, we can't tie success to a
particular exception.
2019-10-18 16:44:16 -07:00
Mike Ellery
b2317f8b41 Omit downloader resolve test when it won't fail
Fixes: #3108
2019-10-18 16:44:16 -07:00
Nik Bougalis
113167acf4 Allow channel_authorize to use Ed25519 keys 2019-10-18 16:44:16 -07:00
Nik Bougalis
a3a9dc26b4 Introduce support for deletable accounts:
The XRP Ledger utilizes an account model. Unlike systems based on a UTXO
model, XRP Ledger accounts are first-class objects. This design choice
allows the XRP Ledger to offer rich functionality, including the ability
to own objects (offers, escrows, checks, signer lists) as well as other
advanced features, such as key rotation and configurable multi-signing
without needing to change a destination address.

The trade-off is that accounts must be stored on ledger. The XRP Ledger
applies reserve requirements, in XRP, to protect the shared global ledger
from growing excessively large as the result of spam or malicious usage.

Prior to this commit, accounts had been permanent objects; once created,
they could never be deleted.

This commit introduces a new amendment "DeletableAccounts" which, if
enabled, will allow account objects to be deleted by executing the new
"AccountDelete" transaction. Any funds remaining in the account will
be transferred to an account specified in the deletion transaction.

The amendment changes the mechanics of account creation; previously
a new account would have an initial sequence number of 1. Accounts
created after the amendment will have an initial sequence number that
is equal to the ledger in which the account was created.

Accounts can only be deleted if they are not associated with any
obligations (like RippleStates, Escrows, or PayChannels) and if the
current ledger sequence number exceeds the account's sequence number
by at least 256 so that, if recreated, the account can be protected
from transaction replay.
2019-10-18 16:44:16 -07:00
Joseph Busch
7e7664c29a Add deletion_blockers_only param to account_objects RPC command 2019-10-18 14:18:38 -07:00
Manoj doshi
fccb7e1c70 Set version to 1.4.0-b6 2019-10-15 12:02:12 -07:00
Howard Hinnant
3f45b8c3bd Switch deadlock detector to steady_clock
* Changes to system time should not trigger the deadlock detector.
* Fixes #3101
2019-10-15 12:01:37 -07:00
seelabs
ca6d5798e9 Support for boost 1.71:
* replace boost::beast::detail::iequals with boost::iequals
* replace deprecated `buffers` function with `make_printable`
* replace boost::beast::detail::ascii_tolower with lambda
* add missing includes
2019-10-15 12:01:37 -07:00
Mike Ellery
2110b24090 Add omitted unit tests, cleanup old files 2019-10-15 12:01:37 -07:00
Mike Ellery
82484e26f5 Add option to enable -Wextra for gcc/clang. 2019-10-15 12:01:37 -07:00
Scott Schurr
ca47583a3b Small bug fixes in BuildLedger.cpp 2019-10-15 12:01:37 -07:00
Joseph Busch
f4d6b0e1c4 Add metrics for PeerImp to track bandwidth usage 2019-10-15 12:01:37 -07:00
Devon White
9196d9541a Include validator's master public key in validation stream:
The validation stream only reported the ephemeral signing key for validators
which use manifests. This made tracking unnecessarily difficult for clients
processing the data stream.

With this change, the validator's long-term master public key is also
included.

This commit fixes #3005

* Provide proposing validator's master key in the validation stream
  subscription JSON responses.

Implement code review changes.

FIXES: #3005
2019-10-15 12:01:37 -07:00
Manoj doshi
b53fda1e1a Set version to 1.4.0-b5 2019-09-27 12:25:21 -07:00
Mike Ellery
9a5911f94c CI: remove boost from macos/hombrew 2019-09-27 12:24:19 -07:00
Mike Ellery
80acc85e59 Fix startup error with --import 2019-09-27 12:24:19 -07:00
Joseph Busch
8ce1c189f7 Fix VS2019 debug build 2019-09-27 12:24:19 -07:00
Howard Hinnant
7228b2e068 Remove SHAMap V2 2019-09-27 12:24:19 -07:00
Yusuf Sahin HAMZA
33ab0cd7bd Automatically update NodeDB path if changed 2019-09-27 12:24:19 -07:00
seelabs
e33ac1d450 Add PayChan to recipient's owner directory 2019-09-27 11:35:22 -07:00
Rome Reginelli
826cbbc3bf Clarify Linux build instructions for configuring
* Explain that Arch/Manjaro/etc. need `-Dstatic=OFF` during the configure step
* move configuration options closer to that step
* separate sub-headers for configuration and build
2019-09-27 11:35:22 -07:00
Mike Ellery
38f82115f7 Set version to 1.4.0-b4 2019-09-09 18:46:06 -07:00
Mike Ellery
d9bb78c8b8 Cleanup usage of ScopedUnlock 2019-09-09 18:45:56 -07:00
Mo Morsi
905f631b71 Modularize cmake build config 2019-09-09 18:45:53 -07:00
Mike Ellery
9213c49ca1 Honor SSL config settings for ValidatorSites:
FIXES: #2990

* refactor common SSL client setup
* enable SSL in unit-test http server
* add tests for SSLHTTPDownloader
* misc test refactoring
2019-09-09 10:55:31 -07:00
Nik Bougalis
fc7ecd672a Set version to 1.4.0-b3 2019-09-07 12:08:47 -07:00
Mark Travis
a6944be5cf Clarify online delete data error message. 2019-09-07 11:44:00 -07:00
Mark Travis
e5b61c9ac9 Update operating mode upon network disagreement. 2019-09-07 11:44:00 -07:00
Scott Schurr
a9a4e2c8fb Add Destination to Check threading 2019-09-07 11:39:02 -07:00
Miguel Portilla
56eac5c9a1 Fix rand_int assert in shard store 2019-09-07 11:39:02 -07:00
Miguel Portilla
4b1970afa9 Log database connection error 2019-09-07 11:39:02 -07:00
Miguel Portilla
22c9de487a Add shard thread safety 2019-09-07 11:39:02 -07:00
Miguel Portilla
66fad62e66 Implement Shard SQLite support 2019-09-07 11:39:02 -07:00
Mike Ellery
008fc5155a Improve CI and packaging:
* add travis build for min cmake supported
* add travis build for validator keys (uses xrpl_core)
* add travis build for ipv6 (mac only)
* add cmake target for validator keys via FetchContent
* use validator keys target in package build
2019-09-07 11:39:02 -07:00
seelabs
5834fbbc5d Set version to 1.4.0-b2 2019-08-23 11:33:59 -07:00
seelabs
06219f1151 Disable shadow warning:
Different compilers and handling the shadow warning differently. In particular,
some are warning about types being shadowed by variables. Until these can be
resolved the shadow warning is being disabled.

Note: the shadow warning was originally enabled to help with the structured
bindings patch. As that is now complete, it's less important to keep this
warning.
2019-08-23 11:33:59 -07:00
seelabs
c2d2ba9f45 Simplify code using if constexpr:
Also simplify msig construction
2019-08-23 11:33:59 -07:00
seelabs
1eb3753f26 Replace from_string_checked pair return type with optional<Endpoint> 2019-08-23 11:33:59 -07:00
seelabs
0a256247a0 Replace strUnHex pair return type with optional<Blob> 2019-08-23 11:33:59 -07:00
seelabs
7912ee6f7b Use structured bindings in some places:
Most of the new uses either:
* Replace some uses of `tie`
* bind to pairs when iterating through maps
2019-08-23 11:33:59 -07:00
seelabs
9c58f23cf8 Replace unordered_map::emplace with insert_or_assign 2019-08-23 08:47:43 -07:00
seelabs
9245b0b666 Use std::gcd to implement lowestTerms 2019-08-23 08:47:43 -07:00
seelabs
92925a0af6 Use [[fallthrough]] in some switch statements 2019-08-23 08:47:43 -07:00
seelabs
70c2e1b419 remove make_Amounts:
* c++-17's class template type deduction can replace this function
2019-08-23 08:47:43 -07:00
seelabs
5d1728cc96 Use class template argument deduction for locks 2019-08-23 08:47:43 -07:00
seelabs
4076b6d92e Replace for_each_arg trick with fold expressions 2019-08-23 08:47:42 -07:00
seelabs
b9e73b4852 Fix shadowing variables 2019-08-23 08:47:42 -07:00
seelabs
014df67fed Remove unused member variable 2019-08-23 08:47:35 -07:00
Mike Ellery
014ec021bb Build packages with gcc-8 2019-08-23 08:42:14 -07:00
Mike Ellery
7fa9b91d23 Fix jenkins/travis CI:
* remove clang builds from jenkins
* disable windows travis cache
* limit make parallelism
* update linux CI image
2019-08-23 08:41:49 -07:00
Nik Bougalis
98c4a7a2b1 Set version to 1.4.0-b1 2019-08-19 06:58:55 -07:00
Vishwas Patil
c04c00d279 Add "sahyadri.isrdc.in" to list of bootstrap nodes 2019-08-19 06:58:50 -07:00
Elliot Lee
ef139e8b3c Update README.md
- This fixes #3023; thanks @crypto-deus.
- This fixes #3026; thanks @William-Gomez.
2019-08-19 06:58:50 -07:00
Alloy Networks
45c1c38993 Warn if the update script is not executed as root 2019-08-19 06:58:50 -07:00
Mo Morsi
1942fee581 Modernize code and clean up out-of-date or obsolete comments:
- Remove references to nodestore ledger index. This was removed
  in f946d7b447.
2019-08-19 06:58:50 -07:00
seelabs
b3c85e2709 Remove unused TER code from StrandResult 2019-08-19 06:58:50 -07:00
seelabs
561942da23 Use enums for some parameters in payments:
* Use enums for StrandDirection, DebtDirection, and QualityDirection
2019-08-19 06:58:50 -07:00
seelabs
c217baa367 Enable C++17 2019-08-19 06:58:50 -07:00
Howard Hinnant
c699864c85 Fix incorrect snapShot unsharing
This fixes #3020.
2019-08-19 06:58:50 -07:00
Howard Hinnant
4ff0f482c3 Remove unneeded and unused base classes in insight 2019-08-19 06:58:50 -07:00
Scott Schurr
28b942b186 Enhance AccountTx unit test 2019-08-19 06:58:50 -07:00
Edward Hennis
4900e3081d Update appveyor dependencies for boost 1.70 2019-08-19 06:58:50 -07:00
Mike Ellery
145326c00c Add policy check to FindBoost 2019-08-19 06:58:50 -07:00
Mike Ellery
c7d90bfddd Update to current SOCI HEAD 2019-08-16 10:33:08 -07:00
Mike Ellery
bfa84cfca5 Fix io_latency_probe test on CI environments 2019-08-16 10:33:08 -07:00
Mike Ellery
cbc6e500b6 Set minimum versions for gcc/clang 2019-08-16 10:33:08 -07:00
Mike Ellery
13a4fefe34 Travis CI improvements:
FIXES: #2527

* define custom docker image for travis-linux builds based on
  package build image
* add macos builds
* add windows builds (currently allowed to fail)
* improve build and shell scripts as required for the CI envs
* add asio timer latency workaround
* omit several manual tests from TravisCI which cause memory exhaustion
2019-08-16 10:33:08 -07:00
John Freeman
87e9ee5ce9 Add support for reserved peer slots:
This commit allows server operators to reserve slots for specific
peers (identified by the peer's public node identity) and to make
changes to the reservations while the server is operating.

This commit closes #2938
2019-08-05 17:46:24 -07:00
John Freeman
20cc5df5fe Fix GitLab CI
- Update Docker image to Boost 1.70
- Bust dependency cache
- Pass `Boost_NO_BOOST_CMAKE` to CMake
2019-08-05 09:49:43 -07:00
Miguel Portilla
9e117b7b38 Fix shard path detection 2019-08-04 20:48:08 -07:00
Miguel Portilla
a02d914093 Move shard store init to Application 2019-08-04 20:48:08 -07:00
Miguel Portilla
c5a95f1eb5 Remove SQLite Validations table 2019-08-04 20:01:34 -07:00
Manoj doshi
e1adbd7ddd Set version to 1.3.1 2019-07-24 15:22:24 -07:00
Joseph Busch
355a7b04a8 Add a LogicError when a deadlock is detected 2019-07-24 14:08:47 -07:00
Lazaridis
d3ee0df93a Add links to in-repo build-instructions 2019-07-24 14:03:44 -07:00
Mark Travis
7c24f7b170 Improve logging during process startup. 2019-07-24 13:55:38 -07:00
Mike Ellery
a3060516c6 Improve package build pipeline:
- Add docker container tags for "latest_BRANCH"
- Prevent different branches from overwriting deb repo artifacts
- Manual approval always required before pushing to prod
2019-07-24 13:53:09 -07:00
Nik Bougalis
caa5c9e223 Set version to 1.3.0 2019-07-09 13:56:37 -07:00
Manoj doshi
846538304f Set version to 1.3.0-rc2 2019-07-09 13:50:12 -07:00
Mo Morsi
7b7e3b6750 Return WS error on closure when balance threshold exceeds 2019-07-09 13:50:12 -07:00
Miguel Portilla
a988b3224f Use NuDB context with backends 2019-07-09 13:50:12 -07:00
Scott Schurr
89b3bf0796 Stabilize RPC error code values:
The original intent was that RPC error codes were not stable.
But those codes were made available through the API, so some
users came to depend on the code values.  This change adapts
to the current state of affairs.
2019-07-09 13:50:12 -07:00
seelabs
6d8988b78a Improve handling of revoked manifests:
Manifests which are revoked can include ephemeral keys although doing
so does not make sense: a revoked manifest isn't used for signing and
so don't need to define an ephemeral key.
2019-07-09 13:38:59 -07:00
Mike Ellery
3acbd84f1d Set proper system openssldir in package build 2019-07-03 11:22:53 -07:00
Manoj doshi
45403b877f Set version to 1.3.0-rc1 2019-06-25 12:03:18 -07:00
Manoj doshi
f17d9bc421 Set version to 1.3.0-b6 2019-06-21 14:59:08 -07:00
Nik Bougalis
ba2714fa22 Make protocol message counters more granular:
A running instance of the server tracks the number of protocol messages
and the number of bytes it sends and receives.

This commit makes the counters more granular, allowing server operators
to better track and understand bandwidth usage.
2019-06-21 14:53:50 -07:00
Mike Ellery
2c4b3d515d Trim whitespace for all config lines
FIXES: 2979
2019-06-21 14:48:45 -07:00
Mike Ellery
59973a435e Add beast/cxx17 to install set 2019-06-19 11:32:31 -07:00
Mike Ellery
93232ec7df Add logrotate config to rpm/deb pkgs 2019-06-19 11:31:19 -07:00
Manoj doshi
efa926676c Set version to 1.3.0-b5 2019-06-13 21:06:25 -07:00
Scott Schurr
dc24748c24 Improve locking of PeerImp member variables 2019-06-13 20:59:28 -07:00
Mo Morsi
f8365f5009 Add JsonOptions enum class to contain options passed to getJSON methods 2019-06-13 20:40:33 -07:00
Mo Morsi
c2138c4e88 Fix Docker error about "FROM" macro usage 2019-06-13 20:39:47 -07:00
Mike Ellery
bfad96dbb9 Force snappy compression for RocksDB (remove option):
FIXES: https://github.com/ripple/rippled/issues/2860

 * Also remove RocksDBQuick backend which is non-functional.
2019-06-13 20:38:42 -07:00
Mike Ellery
0ef6d9f9a0 Add slack notify for approvals, remove old RPM build 2019-06-13 20:38:22 -07:00
Mike Ellery
c1a1cfe550 Pkgbld - Make approval blocking, add slack summary message 2019-06-13 20:38:05 -07:00
Howard Hinnant
773dcd1d48 Modernize base_uint:
*  Add construction and assignment from a generic
   contiguous container.  Both compile-time and run time
   safety checks are made to ensure the safety of this
   conversion.

*  Remove base_uint::copyFrom.  The generic copy assignment
   operator now does this functionality with enhanced
   safety and better syntax.

*  Remove construction from and dedendence on Blob.
   The generic constructor and assignment now handle this
   functionality.

*  Fix client code to adhere to this new API.

*  Removed the use of fromVoid in PeerImp.cpp as it was
   an inappropriate use of this dangerous API.  The
   generic container constructors do it with enhanced
   safety and better syntax.

*  Rename data member pn to data_ and make it private.

*  Remove constraint from hash_append

*  Remove array_type alias
2019-06-13 20:37:29 -07:00
Miguel Portilla
de99e79bf1 Fix SNTPClock shutdown
This PR addresses a problem where the server could hang indefinitely
on shutdown. The cause of the problem is the SNTPClock class was not
binding the socket to an endpoint on initialization. This can create
an error sent to the read handler. Unfortunately, the handler ignores
the error, reads again and enters into a loop preventing the
io_service from ever completing.
2019-06-13 20:36:45 -07:00
Manoj doshi
e83d367f49 Set version to 1.3.0-b4 2019-05-22 14:44:15 -07:00
invalidator
aa76b382af Document IPv6 usage in sample config:
- Explain how to bind to both IPv4 and IPv6 interfaces
- Provide a hint in the default [port_peer] section
- Do not enable it by default

Note that on Linux, use of '::' and IPv4-mapped IPv6 depends on a sysctl value
setting 'net.ipv6.bindv6only = 0' which seems to be the default on most Linux
distributions.
2019-05-22 13:15:43 -07:00
Howard Hinnant
595b7b194c Improve locking:
- Use `std::lock` when grabbing multiple mutexes to ensure consistent
locking order and avoid deadlocks.
- Reduce the scope of the master mutex lock by relesing it prior to
calling setHeartbeatTimer
2019-05-22 13:15:43 -07:00
JoelKatz
5f908ba870 Make some locks more granular:
- Overlay
- Cluster
- Validator List
- Consensus

The overlay also has its own lock and manages its own thread safety.
2019-05-22 13:15:43 -07:00
Mike Ellery
adc1b8a36b Update package build env to boost 1.70 2019-05-22 13:15:43 -07:00
Mike Ellery
73c6e47e8a Use local rippled core lib during pkg build 2019-05-22 13:15:43 -07:00
Mike Ellery
3a780f80f1 Remove repo package check from update script 2019-05-22 13:15:43 -07:00
mtrippled
c78404e233 Pause for lagging validators. 2019-05-22 13:15:43 -07:00
seelabs
79a0cb096b Payment paths with a zero output step are dry (RIPD-1749):
A tiny input amount to a payment step can cause this step to output zero. For
example, if a previous steps outputs a dust amount of 10^-80, and this step is a
IOU -> XRP offer, the offer may output zero drops. In this case, call the strand
dry. Before this patch, an error would be logged, the strand would be called
dry; in debug mode an assert triggered.

Note, this patch is not transaction breaking, as the caller did not user the ter
code. The caller only checked for success or failuer.

This patch addresses github issue issue reported here:
https://github.com/ripple/rippled/issues/2929
2019-05-20 15:58:54 -07:00
seelabs
6f9e8dc720 Support Boost 1.70:
This patch removes calls to several deprecated asio functions.

* `io_service::post` becomes `post` (free function)
* `io_service::work` becomes `executor_work_guard`
* `io_service::wrap` becomes `bind_executor`
* `get_io_context`   becomes `get_executor` or `get_executor().context()`

This patch was tested with boost 1.69 and 1.70. The functions
`ripple::get_lowest_layer` and `beast::create_waitable_timer` are required to
handle a breaking difference between these versions. When rippled no longer
needs to support pre 1.70 boost versions, both of these functions may be
removed, and the waitable timer injections may also be removed.
2019-05-20 15:58:54 -07:00
Scott Schurr
b39b0fef39 Get names of transactions and ledger types from jss 2019-05-20 15:58:54 -07:00
Edward Hennis
be139d9bde Add some missing items to help command list:
* validators
* validator_list_sites
* Put "version" in the right place
2019-05-20 15:58:54 -07:00
John Freeman
c6d82c722b Configure build+test matrix for GitLab CI:
* Disable parallel tests for address sanitizer
* Improve caching
* Prefer Ninja builds because they are faster
2019-05-20 15:58:54 -07:00
John Freeman
0c20e2eb8b Refine parseUrl regular expression (RIPD-1708):
The new parse logic is more strict but handles more cases. If an exception
is thrown, just bail.

* Allow parsing unenclosed IPv6 addresses without port
* Improve string construction
* Reduce nesting levels of code
2019-05-20 15:58:54 -07:00
James Fryman
63eeb8d734 Use recursive remove and clean for apt (OPS-508) 2019-05-20 15:58:34 -07:00
seelabs
5214b3c1b0 Set version to 1.3.0-b3 2019-04-29 08:18:48 -04:00
Jesper Wallin
5f7a61f040 Report a peer's public key and IP address in log messages (fixes #2675) 2019-04-29 08:17:24 -04:00
John Freeman
c5a938de55 Disallow using the master key as the regular key:
The XRP Ledger allows an account to authorize a secondary key pair,
called a regular key pair, to sign future transactions, while keeping
the master key pair offline.

The regular key pair can be changed as often as desired, without
requiring other changes on the account.

If merged, this commit corrects a minor technical flaw which would
allow an account holder to specify the master key as the account's
new regular key.

The change is controlled by the `fixMasterKeyAsRegularKey` amendment
which, if enabled, will:

1. Prevent specifying an account's master key as the account's
   regular key.
2. Prevent the "Disable Master Key" flag from incorrectly affecting
   regular keys.
2019-04-29 08:17:24 -04:00
Mike Ellery
9372a587e4 Request RocksDB PORTABLE build option 2019-04-29 08:17:24 -04:00
Mike Ellery
948e724dff Improvements to pkg CI pipeline:
* add manual approval option before push to prod
* Use new public repo DNS name
* add distros to smoketest
2019-04-29 08:17:24 -04:00
Mike Ellery
06faf2bd5b Improve exit and test failure handling in CI 2019-04-29 08:17:24 -04:00
Mike Ellery
1dd81c04f3 Improve jemalloc build config:
* fix include order for macos/homebrew
* use static jemalloc for static builds
* set CMP0074 for using <pkgname>_ROOT variables
2019-04-29 08:17:24 -04:00
Mike Ellery
56dbf70c3c Improve windows build README 2019-04-29 08:17:24 -04:00
Mike Ellery
f8a4ac6ad7 Use optimized OpenSSL implementations when possible 2019-04-29 08:17:24 -04:00
seelabs
61bd06177f Reserve memory before inserting into a flat set 2019-04-29 08:17:24 -04:00
seelabs
80e535a13c Arguments passed to jtx Env::operator() must be invocable:
Before this patch, jtx allowed non-invocable functions to be passed to
operator(). However, these arguments are ignored. This caused erronious code
code such as:

```
env (offer (account_to_test, BTC (250), XRP (1000)),
         offers (account_to_test, 1));
```

While it looks like the number of offers are checked, they are not. The `offers`
funclet is never run. While we could modify jtx to make the above code correct,
a cleaner solution is to run post conditions in a `require` statement after a
transasction runs.
2019-04-26 11:22:36 -07:00
Scott Schurr
64b55c0f88 Rename JsonFields.h to jss.h:
At this point all of the jss::* names are defined in the same
file.  That file has been named JsonFields.h.  That file name
has little to do with either JsonStaticStrings (which is what
jss is short for) or with jss.  The file is renamed to jss.h
so the file name better reflects what the file contains.

All includes of that file are fixed.  A few include order
issues are tidied up along the way.
2019-04-26 11:21:52 -07:00
Scott Schurr
afcc4ff296 Reduce likelihood of malformed SOTemplate:
Formerly an SOTemplate was default constructed and its elements
added using push_back().  This left open the possibility of a
malformed SOTemplate if adding one of the elements caused a throw.

With this commit the SOTemplate requires an initializer_list of
its elements at construction.  Elements may not be added after
construction.  With this approach either the SOTemplate is fully
constructed with all of its elements or the constructor throws,
which prevents an invalid SOTemplate from even existing.

This change requires all SOTemplate construction to be adjusted
at the call site.  Those changes are also in this commit.

The SOE_Flags enum is also renamed to SOEStyle, which harmonizes
the name with other uses in the code base.  SOEStyle elements
are renamed (slightly) to have an "soe" prefix rather than "SOE_".
This heads toward reserving identifiers with all upper case for
macros.  The new style also aligns with other prominent enums in
the code base like the collection of TER identifiers.

SOElement is adjusted so it can be stored directly in an STL
container, rather than requiring storage in a unique_ptr.
Correspondingly, unique_ptr usage is removed from both
SOTemplate and KnownFormats.
2019-04-26 11:17:45 -07:00
Scott Schurr
57fe197d3e Remove runtime inference of unrecognized SFields 2019-04-26 11:17:45 -07:00
Edward Hennis
9279a3fee7 Refactor SField construction:
* Use a private_access_tag_t to prevent other files from
  instantiating an SField.
* Delete SField move constructor and make helper.
2019-04-26 11:17:45 -07:00
JoelKatz
b6363289bf Use Json::StaticString for field names
Clean up some code relating to unknown fields and avoid
allocate/copy/free cycles for Json objects containing
serialized field names.
2019-04-26 11:17:45 -07:00
Nik Bougalis
8c1123edc6 Merge master (1.2.4) into develop (1.3.0-b2) 2019-04-26 10:42:51 -07:00
Nik Bougalis
834f545498 Set version to 1.2.4 2019-04-15 12:39:27 -07:00
Mike Ellery
dd99bf479f Enforce a 20s timeout when making validator list requests (RIPD-1737) 2019-04-15 12:39:16 -07:00
Miguel Portilla
2e26377e7c Use public key when routing shard crawl requests 2019-04-15 12:39:08 -07:00
seelabs
0329ee236f Set version to 1.2.3 2019-03-28 17:47:14 -04:00
seelabs
b347afcc5b Better error checking in CachedViewImpl::read:
* Prevent null pointer dereferences
* Alway check for correct sle type before returning sle
* Reformat code
2019-03-28 17:47:14 -04:00
Nik Bougalis
fa57859477 Set version to 1.3.0-b2 2019-03-19 15:31:21 -07:00
Nik Bougalis
88cb0e5928 Allow manifests to include an optional 'domain' field:
The new 'Domain' field allows validator operators to associate a domain
name with their manifest in a transparent and independently verifiable
fashion.

It is important to point out that while this system can cryptographically
prove that a particular validator claims to be associated with a domain
it does *NOT* prove that the validator is, actually, associated with that
domain.

Domain owners will have to cryptographically attest to operating particular
validators that claim to be associated with that domain. One option for
doing so would be by making available a file over HTTPS under the domain
being claimed, which is verified separately (e.g. by ensuring that the
certificate used to serve the file matches the domain being claimed) and
which contains the long-term master public keys of validator(s) associated
with that domain.

Credit for an early prototype of this idea goes to GitHub user @cryptobrad
who introduced a PR that would allow a validator list publisher to attest
that a particular validator was associated with a domain. The idea may be
worth revisiting as a way of verifying the domain name claimed by the
validator's operator.
2019-03-19 15:31:21 -07:00
Nik Bougalis
e239eed6de Remove obsolete code 2019-03-19 15:31:20 -07:00
Mark Travis
504b3441dd Apply resource limits to proxied clients:
Resource limits were not properly applied to connections with
known IP addresses but no corresponding users.

Add unit tests for unlimited vs. limited ports.
2019-03-19 08:00:17 -07:00
Scott Schurr
872478d965 Construct ErrorCodes lookup table at compile time 2019-03-19 08:00:17 -07:00
Scott Schurr
185f2baf76 Remove unused RPC error codes:
An audit showed that a number of the RPC error codes in
ErrorCodes.h are no longer used in the code base.  The unused
codes were removed from the file along with their support code
in ErrorCodes.cpp.
2019-03-19 08:00:09 -07:00
Scott Schurr
36d6758945 Disallow both single- and multi-signing in RPC (RIPD-1713):
The ledger already declared a transaction that is both single-
and multi-signing malformed.  This just adds some checking in
the signing RPC commands (like submit and sign_for) which allows
that sort of error to be identified a bit closer to the user.

In the process of adding this code a bug was found in the
RPCCall unit test.  That bug is fixed as well.
2019-03-18 17:08:36 -07:00
seelabs
d8c450d272 Remove incorrectly defaulted functions:
* The functions removed in this commit were explicitly defaulted
  but implicitly deleted
2019-03-18 17:08:36 -07:00
Mike Ellery
8ef5b9bab4 Make LedgerTrie remove work for truncated history 2019-03-18 17:08:36 -07:00
Mike Ellery
e6370a6482 Add dpkg/rpm building capability:
* docker container definitions for package building
* cmake targets for building packages
* initial gitlab CI + artifactory integration
2019-03-18 16:44:54 -07:00
Mike Ellery
b2170d016a Update travis dist/tools 2019-03-18 16:36:57 -07:00
Mike Ellery
5c124f11c2 Remove the 'rocksdb' subtree 2019-03-18 16:19:24 -07:00
Mike Ellery
2aed24a552 Build RocksDB by ExternalProject 2019-03-18 16:19:24 -07:00
Howard Hinnant
296469f5fe Reduce memory allocations for RCLCensorshipDetector 2019-03-18 16:19:24 -07:00
Miguel Portilla
08371ba2c4 Improve shard downloader status reporting 2019-03-18 16:19:24 -07:00
Miguel Portilla
56bc2a2ade Improve SSLHTTPDownloader:
* Use TLS 1.2
* Make certificate verification configurable
2019-03-18 16:19:23 -07:00
Nik Bougalis
1084dc6dd3 Set version to 1.3.0-b1 2019-03-06 19:37:54 -08:00
Elliot Lee
8023caaa97 Correct example configuration file:
Trailing comments are not permitted in the crawl section
and can cause the lines containing them to be ignored.
2019-03-06 19:37:48 -08:00
Howard Hinnant
8b97466285 Always use UTC to be timezone-neutral (RIPD-1659) 2019-03-06 19:37:48 -08:00
Mike Ellery
de1d102535 Allow build to support XCode 10.2 2019-03-06 19:14:52 -08:00
Scott Schurr
1e1e8c2547 Remove assert that accesses object post-dtor (RIPD-1704) 2019-03-06 19:14:52 -08:00
Crypto Brad Garlinghouse
aa49be65a1 Remove conditional check for feature introduced in 0.28.1-b7 2019-03-06 19:14:52 -08:00
Crypto Brad Garlinghouse
cd820b3777 Improve the server's PING/PONG logic 2019-03-06 19:14:52 -08:00
Crypto Brad Garlinghouse
8d59ed5b2a Remove STValidation::isValid overload 2019-03-06 19:14:52 -08:00
seelabs
e03efdbe0b Remove use of beast's detail::sec_ws_key_type 2019-03-06 19:14:52 -08:00
seelabs
4f52c2989c Remove use of beast::detail::is_invocable trait 2019-03-06 19:14:52 -08:00
Mike Ellery
3fb13233a9 Provide patch for FindBoost and apply it 2019-03-06 19:14:52 -08:00
seelabs
9695fd44ba Support boost 1.69 2019-03-06 19:14:52 -08:00
ChronusZ
1bb32134f8 Remove censorshipMaxWarnings 2019-03-06 19:14:52 -08:00
seelabs
0ebed96142 Set version to 1.2.2 2019-03-05 18:21:39 -05:00
Edward Hennis
4c06b3f86f Validate TxQ config and expected transactions range 2019-03-04 11:45:56 -05:00
Nik Bougalis
a3470c225b Set version to 1.2.1 2019-02-25 13:01:32 -08:00
seelabs
c5d215d901 Add delivered amount to the ledger RPC command 2019-02-25 13:01:12 -08:00
JoelKatz
9dbf8495ee Avoid a race condition during peer status change 2019-02-25 12:59:35 -08:00
Nik Bougalis
2529edd2b6 Properly transition state to disconnected:
If the number of peers a server has is below the configured
minimum peer limit, this commit will properly transition the
server's state to "disconnected".

The default limit for the minimum number of peers required was
0 meaning that a server that was connected but lost all its
peers would never transition to disconnected, since it could
never drop below zero peers.

This commit redefines the default minimum number of peers to 1
and produces a warning if the server is configured in a way
that will prevent it from ever achieving sufficient connectivity.
2019-02-25 12:59:35 -08:00
Nik Bougalis
e974c7d8a4 Avoid directly using memcpy to deserialize data 2019-02-25 12:59:34 -08:00
Nik Bougalis
b335adb674 Make validators opt out of crawl:
If a server is configured to support crawl, it will report the
IP addresses of all peers it is connected to, unless those peers
have explicitly opted out by setting the `peer_private` option
in their config file.

This commit makes servers that are configured as validators
opt out of crawling.
2019-02-25 12:59:34 -08:00
Nik Bougalis
c6ab880c03 Display validator status only to admin requests:
Several commands allow a user to retrieve a server's status. Commands
will typically limit disclosure of information that can reveal that a
particular server is a validator to connections that are not verified
to make it more difficult to determine validators via fingerprinting.

Prior to this commit, servers configured to operate as validators
would, instead of simply reporting their server state as 'full',
augment their state information to indicate whether they are
'proposing' or 'validating'.

Servers will only provide this enhanced state information for
connections that have elevated privileges.

Acknowledgements:
Ripple thanks Markus Teufelberger for responsibly disclosing this issue.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
2019-02-25 12:59:31 -08:00
Mike Ellery
7779dcdda0 Set version to 1.2.0 2019-02-12 16:41:03 -08:00
Mike Ellery
132f1b218c Set version to 1.2.0-rc2 2019-01-30 15:37:56 -08:00
Mike Ellery
e5d6f16f19 Remove [ips] section from sample config 2019-01-30 15:33:39 -08:00
Mike Ellery
8f973621fc Set version to 1.2.0-rc1 2019-01-28 12:02:33 -08:00
Mike Ellery
b75c2d71a5 Make sample config comment consistent with code 2019-01-28 11:53:30 -08:00
Nik Bougalis
eed210bb67 Set version to 1.2.0-b11 2019-01-18 12:13:22 -08:00
Mike Ellery
eab2a0d668 Improve debug information generated for the LedgerTrie 2019-01-18 12:13:21 -08:00
Howard Hinnant
148bbf4e8f Add safe_cast (RIPD-1702):
This change ensures that no overflow can occur when casting
between enums and integral types.
2019-01-18 12:13:21 -08:00
Joseph Busch
494724578a Enchance /crawl API endpoint with local server information (RIPD-1644):
The /crawl API endpoint allows developers to examine the structure of
the XRP Ledger's overlay network.

This commit adds additional information about the local server to the
/crawl endpoint, making it possible for developers to create data-rich
network-wide status dashboards.

Related:
 - https://developers.ripple.com/peer-protocol.html
 - https://github.com/ripple/rippled-network-crawler
2019-01-18 12:13:21 -08:00
Nik Bougalis
ea76103d5f Detect malformed data earlier during deserialization (RIPD-1695):
When deserializing specially crafted data, the code would ignore certain
types of errors. Reserializing objects created from such data results in
failures or generates a different serialization, which is not ideal.

Also addresses: RIPD-1677, RIPD-1682, RIPD-1686 and RIPD-1689.

Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these issues.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
2019-01-18 12:13:21 -08:00
Nik Bougalis
2151110976 Improve message buffering (RIPD-1699):
Specially crafted messages could cause the server to buffer large
amounts of memory which could increase memory pressure.

This commit changes how messages are buffered and imposes a limit
on the amount of data that the server is willing to buffer.

Acknowledgements:
Aaron Hook for responsibly disclosing this issue.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find. For information
on Ripple's Bug Bounty program, please visit:

    https://ripple.com/bug-bounty
2019-01-17 18:39:04 -08:00
Nik Bougalis
dfb45baa93 Set version to 1.2.0-b10 2018-12-28 13:32:27 -08:00
f443439f1f Add zaphod.alloy.ee to default hub configuration 2018-12-28 13:31:19 -08:00
Howard Hinnant
6d0b108ec1 Upgrade sqlite to 3.26 (fix #2810) 2018-12-28 13:31:19 -08:00
Howard Hinnant
710f9ee1ac Relax overly-strict assert in Serializer constructor (RIPD-1701):
The constructor would previously assert that the specified buffer pointer
was non-null, even if the buffer size is specified as 0. While reasonable,
this also makes it more difficult to use this API.
2018-12-28 13:31:19 -08:00
Howard Hinnant
76d5ecb595 Verify invariants when calling SHAMapInnerNodeV2::addRaw (RIPD-1700) 2018-12-28 13:32:09 -08:00
Joseph Busch
ba9ca1378e Strict input validation against expected schema (RIPD-1709, RIPD-1710) 2018-12-28 13:31:19 -08:00
Miguel Portilla
1be8094ee2 Improve crawl shard resource usage 2018-12-28 13:31:19 -08:00
Nik Bougalis
96c949a997 Set version to 1.2.0-b9 2018-12-11 13:01:05 -08:00
Mike Ellery
9121e26708 Update libarchive to 3.3.3 from official repo 2018-12-11 12:52:29 -08:00
Edward Hennis
2432f13903 Reserve correct vector size for fee calculations:
* Using txnsExpected_, which is influenced by both the config
  and network behavior, can reserve far too much or far too
  little memory, wasting time and resources.
* Not an issue during normal operation, but a user could
  cause problems on their local node with extreme configuration
  settings.
2018-12-11 12:51:46 -08:00
Edward Hennis
259fb1c32e Fix unit test with incorrectly hard-coded parameter:
* initFee was using a lot of logic that could look unclear. Add
  some documentation explaining why certain values were used.
* Because initFee had side effects, callers needed to repeat the
  max queue size computation, making the initial problem more
  likely. Instead, return the max queue size value, so the caller
  can reuse it.
* A newer test (testInFlightBalance()) was incorrectly using a
  hard-coded queue limit. Fix it to use initFee's new return
  value.
2018-12-11 12:51:46 -08:00
Rome Reginelli
e0515b0015 Correct amount serialization comments 2018-12-11 12:51:46 -08:00
John Freeman
412a3ec710 Fix the --rpc_port command-line argument
The --rpc_port command-line option is effectively ignored. We construct
an `Endpoint` with the given port, but then drop it on the floor.
(Perhaps the author thought the `Endpoint::at_port` method is a mutation
instead of a transformation.) This small change adds the missing
assignment to hold on to the new endpoint.

Fixes #2764
2018-12-11 12:50:05 -08:00
Nik Bougalis
30bba29da2 Merge master (1.1.2) into develop (1.2.0-b8) 2018-12-11 12:48:32 -08:00
Nik Bougalis
4f3a76dec0 Set version to 1.1.2 2018-11-29 21:49:10 -08:00
Nik Bougalis
61f443e3bb Properly bypass connection limits for cluster peers (fix #2795) 2018-11-29 21:38:35 -08:00
Brad Chase
bd2a38f584 Improve preferred ledger calculation:
This changeset ensures the preferred ledger calculation
properly distinguishes the absence of trusted validations
from a preferred ledger which is the genesis ledger.
2018-11-29 21:38:12 -08:00
Nik Bougalis
4cff94f7a4 Set version to 1.2.0-b8 2018-11-25 17:39:49 -08:00
Mark Travis
fbdbffed67 Report duration in current state. 2018-11-25 17:37:31 -08:00
Scott Schurr
ad5c5f1969 STObject::applyTemplate() throws with description of error:
The `STObject` member function `setType()` has been renamed to
applyTemplate() and modified to throw if there is a template
mismatch.

The error description in the exception is, in certain cases,
used, to better indicate why a particular transaction was
considered ill formed.

Fixes #2585.
2018-11-25 17:37:31 -08:00
John Freeman
c354809e1c Implement missing string conversions for JSON
`Json::Value::isConvertibleTo` indicates that unsigned integers and
reals are convertible to string, but trying to do so (with
`Json::Value::asString`) throws an exception because its internal switch
is missing these cases. This change fills them in (and adds tests).

Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing this issue.

Closes #2778
2018-11-25 17:37:14 -08:00
John Freeman
dc4d76f626 Prefer regex to manual parsing in parseURL:
Although `parseURL` used a regex to pull the authority out of the URL
being parsed, it performed manual parsing of the hostname and port.

This commit rolls the parsing of the username and password, if any,
directly into the regex. The hostname can be a name, an IPv4 or an
IPv6 address.

Fixes #2751
2018-11-21 17:08:21 -08:00
Edward Hennis
c1a02440dc Load validator list from file:
* Adds local file:// URL support to the [validator_list_sites] stanza.
  The file:// URL must not contain a hostname. Allows a rippled node
  operator to "sideload" a new list if their node is unable to reach
  a validator list's web site before an old list expires. Lists
  loaded from a file will be validated in the same way a downloaded
  list is validated.
* Generalize file/dir "guards" from Config test so they can be reused
  in other tests.
* Check for error when reading validators.txt. Saves some parsing and
  checking of an empty string, and will give a more meaningful error.
* Completes RIPD-1674.
2018-11-20 19:49:39 -08:00
Edward Hennis
e7a69cce65 Account for minimum reserve in potential spend:
* Relevant when deciding whether an account can queue multiple
  transactions. If the potential spend of the already queued
  transactions would dip into the reserve, the reserve is
  preserved for fees.
* Also change several direct modifications of the owner count to
  call adjustOwnerCount to preserve overflow checking.
* Update related unit testcase
* Resolves #2251
2018-11-20 19:49:39 -08:00
Howard Hinnant
60dc949314 Remove custom terminate handler
* Reduce the amount of code we have to maintain.
* Remove the potential for degrading stack dumps.
2018-11-20 19:45:02 -08:00
Nik Bougalis
cc824685e7 Set version to 1.2.0-b7 2018-11-09 07:40:46 -08:00
JoelKatz
be70d81bd7 Perform some extra checks on ledger changes
Perform some extra checks on the close time and sequence number
of a candidate for network consensus ledger. This tightens
defenses against some "insane/hostile supermajority" attacks.
2018-11-09 07:40:41 -08:00
JoelKatz
6df96f08df Ensure websocket PING/PONG token has length 8 (RIPD-1670) 2018-11-09 07:40:41 -08:00
JoelKatz
9ad2b9be45 Fix a rare race condition on shutdown:
If we happen to get very unlucky and close the door when no
accept operation is pending, the do_accept loop would never
terminate.
2018-11-09 07:40:41 -08:00
JoelKatz
0d2b2923da Control memory growth from slow writes
* Don't allow a write batch to grow without bound
* Don't fetch history if write load is high
2018-11-09 07:40:41 -08:00
Mike Ellery
265f5f1fb1 Delete old protobuf subtree 2018-11-08 18:58:13 -08:00
Mike Ellery
a2ab6c4b02 Build protobuf as ExternalProject when not found 2018-11-08 18:58:13 -08:00
Mike Ellery
6bdc9e7b30 Use correct manifest cache when loading ValidatorList 2018-11-08 18:58:13 -08:00
Nik Bougalis
c71eb45240 Eliminate potential undefined behavior (RIPD-1685):
Under certain conditions, we could call `memcpy` or `memcmp` with a null
source pointer. Even when specifying 0 as the amount of data to copy this
could result in undefined behavior under the C and C++ standards.

Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these issues.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
2018-11-08 18:58:13 -08:00
Nik Bougalis
753600a2a0 Reset the validator list fetch timer if an error occurs 2018-11-08 18:58:12 -08:00
Nik Bougalis
945493d9cf Allow servers to detect transaction censorship attempts (RIPD-1626):
The XRP Ledger is designed to be censorship resistant. Any attempt to
censor transactions would require coordinated action by a majority of
the system's validators.

Importantly, the design of the system is such that such an attempt is
detectable and can be easily proven since every validators must sign
the validations it publishes.

This commit adds an automated censorship detector. While the server is
in sync, the detector tracks all transactions that, in the view of the
server, should have been included and issues warnings of increasing
severity for any transactions which, have not after several rounds.
2018-11-08 18:58:11 -08:00
Nik Bougalis
2a8b0e4b88 Set version to 1.2.0-b6 2018-11-06 10:27:29 -08:00
Nik Bougalis
513b1dd194 Add support for Ed25519 seeds encoded using ripple-lib:
When Ed25519 support was added to ripple-lib, a way to specify
whether a seed should be used to derive a "classic" secp256k1
keypair or a "new" Ed25519 keypair was needed, and the
requirements were that:

1. previously seeds would, correctly, generate a secp256k1
   keypair.
2. users would not have to know about whether the seed was
   used to generate a secp256k1 or an Ed25519 keypair.

To address these requirements, the decision was made to encode
the type of key within the seed and a custom encoding was
designed.

The encoding uses a token type of 1 and prefixes the actual
seed with a 2 byte header, selected to ensure that all such
keypairs will, when encoded, begin with the string "sEd".

This custom encoding is non-standard and was not previously
documented; as a result, it is not widely supported and other
sofware may treat such keys as invalid. This can make it
difficult for users that have stored such a seed to use
wallets or other tooling that is not based on ripple-lib.

This commit adds support to rippled for automatically
detecting and properly handling such seeds.
2018-11-06 10:27:13 -08:00
Nik Bougalis
77462b8f72 Remove deprecated 'validation_seed' RPC command:
The 'validation_seed' RPC command was used to change the validation
key used by a validator at runtime.

Its implementation was commented out with commit fa796a2eb5
which has been included in the codebase since the 0.30.0 release
and there are no plans to reintroduce the functionality at this
point.

Validator operators should migrate to using validator manifests
instead.

This fixes #2748.
2018-11-06 10:27:12 -08:00
Nik Bougalis
1682fe3a39 Cleanup unused Beast bits and pieces:
This cleanup does not remove Boost.Beast code, but old-style Beast
which is no longer relevant or helpful.
2018-11-06 10:27:10 -08:00
Edward Hennis
58f786cbb4 Make the FeeEscalation amendment permanent (RIPD-1654):
The FeeEscalation amendment has been enabled on the XRP Ledger network
since May 19, 2016. The transaction which activated this amendment is:
5B1F1E8E791A9C243DD728680F108FEF1F28F21BA3B202B8F66E7833CA71D3C3.

This change removes all conditional code based around the FeeEscalation
amendment, but leaves the amendment definition itself since removing the
definition would cause nodes to think an unknown amendment was activate
causing them to become amendment blocked.

The commit also removes the redundant precomputed hashes from the
supportedAmendments vector.
2018-11-06 10:26:29 -08:00
Edward Hennis
a96cb8fc1c Remove undocumented experimental options from RPC sign (RIPD-1653):
The `x_assume_tx` and `x_queue_okay` experimental options were
associated with the transaction queue that were not officially
supported.
2018-11-06 10:26:29 -08:00
Joe Loser
c587012e5c Inline calls to cachedRead:
Problem:
- There are only a few call sites to cachedRead, and all of them
  currently do more work than is required since we know the type in each
  case.

Solution:
- "Inline" the codepath to cachedRead, but do not check if the type is
  valid. In all such call sites, we know the keylet to read directly.

This fixes #2550
2018-11-06 10:26:29 -08:00
Mike Ellery
ad4bbd8dff Add source filtering for coverage with option to disable 2018-11-06 10:26:29 -08:00
Mike Ellery
202d91c9f0 Remove unused json_batchallocator.h 2018-11-06 10:26:29 -08:00
Howard Hinnant
146ea5d44e Remove a use after std::move
Fixes: #2538
Fixes: #2536
2018-11-06 10:26:29 -08:00
Howard Hinnant
157c066f2b Fix memory leak in Json move assignment operator
*  When move assignment is creates a cyclic ownership pattern
   memory was being leaked.  This patch breaks the cycle.

*  Fixes: #2572
2018-11-06 10:26:29 -08:00
Howard Hinnant
156e8dae83 Replace WaitableEvent with portable std primitives:
The WaitableEvent class was a leftover from the pre-Boost
version of Beast and used Windows- and pthread-specific
APIs.

This refactor replaces that functionality by using only
interfaces provided by the C++ standard, making the code
more portable.

Closes #2402.
2018-11-06 10:26:29 -08:00
Markus Teufelberger
5e96da51f9 Remove the state file for the random number generator 2018-11-06 10:26:29 -08:00
Nik Bougalis
cb71d493a0 Set version to 1.2.0-b5 2018-10-23 08:33:18 -07:00
MarkusTeufelberger
8124c1f51f remove duplicated include
The errno.h header is already included for both Linux and Android above
2018-10-23 08:24:11 -07:00
Nik Bougalis
6ed2270bc9 Merge master (1.1.1) into develop (1.2.0-b4) 2018-10-23 08:21:43 -07:00
Mike Ellery
4e7c038520 Set version to 1.2.0-b4 2018-10-19 12:24:51 -07:00
1535239824@qq.com
7b48dc36f5 Add fixTakerDryOfferRemoval amendment 2018-10-19 12:23:25 -07:00
Miguel Portilla
d5c0e1216d Change conflicting example websocket port 2018-10-19 12:22:47 -07:00
Scott Schurr
a999894dae Allow rippled to compile with C++17:
Many of the warnings on Windows were not resolved, just
silenced with _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS.
They need to be resolved in a future commit.
2018-10-19 12:21:57 -07:00
Scott Schurr
63e167b7a3 ledger_entry RPC by index matches other forms [RIPD-1538] 2018-10-19 12:21:10 -07:00
MarkusTeufelberger
8fc6a8175b Remove unused execinfo.h header
Fixes #2671 and #2159
2018-10-19 12:20:11 -07:00
Edward Hennis
af1697cc6a Improve RPC error message for fee command:
* If rippled is not synced to the network, `fee` will return a
  "no network" error instead of the possibly confusing "not enabled"
  error.
* Resolves RIPD-1588
2018-10-19 12:19:20 -07:00
Mark Travis
e98c76110a Remove outdated example configs. 2018-10-19 12:18:29 -07:00
Mike Ellery
7fe1d4b9c2 Accept redirects from validator list sites:
Honor location header/redirect from validator sites. Limit retries per
refresh interval to 3. Shorten refresh interval after HTTP/network errors.

Fixes: RIPD-1669
2018-10-19 12:16:57 -07:00
Nik Bougalis
b36e11bc49 Properly handle expired validator lists when validating (RIPD-1661):
A validator that was configured to use a published validator list could
exhibit aberrent behavior if that validator list expired.

This commit introduces additional logic that makes validators operating
with an expired validator list bow out of the consensus process instead
of continuing to publish validations. Normal operation will resume once
a non-expired validator list becomes available.

This commit also enhances status reporting when using the `server_info`
and `validators` commands. Before, only the expiration time of the list
would be returned; now, its current status is also reported in a format
that is clearer.
2018-10-19 12:15:36 -07:00
seelabs
72e6005f56 Set version to 1.1.1 2018-10-19 13:12:40 -04:00
Nik Bougalis
152d698957 Properly handle expired validator lists when validating (RIPD-1661):
A validator that was configured to use a published validator list could
exhibit aberrent behavior if that validator list expired.

This commit introduces additional logic that makes validators operating
with an expired validator list bow out of the consensus process instead
of continuing to publish validations. Normal operation will resume once
a non-expired validator list becomes available.

This commit also enhances status reporting when using the `server_info`
and `validators` commands. Before, only the expiration time of the list
would be returned; now, its current status is also reported in a format
that is clearer.
2018-10-19 13:08:56 -04:00
Mike Ellery
7c96bbafbd CI rpm build fix 2018-10-11 11:08:55 -07:00
Mike Ellery
bdaad19e70 Accept redirects from validator list sites:
Honor location header/redirect from validator sites. Limit retries per
refresh interval to 3. Shorten refresh interval after HTTP/network errors.

Fixes: RIPD-1669
2018-10-11 11:08:27 -07:00
seelabs
63c3fc30d8 Set version to 1.2.0-b3 2018-10-10 13:09:25 -04:00
Joe Loser
1ac9694dbc Simplify strHex:
Problem:
- There are several specific overloads with some custom code that can be
  easily replaced using Boost.Hex.

Solution:
- Introduce `strHex(itr, itr)` to return a string given a begin and end
  iterator.
- Remove `strHex(itr, size)` in favor of the `strHex(T)` where T is
  something that has a `begin()` member function. This allows us to
  remove the strHex overloads for `std::string`, Blob, and Slice.
2018-10-10 13:09:22 -04:00
Miguel Portilla
3661dc88fe Add RPC command shard crawl (RIPD-1663) 2018-10-10 12:16:01 -04:00
Edward Hennis
86c066cd7e Include entire src tree in multiconfig projects:
* For example Visual Studio, XCode. This will allow easily working with
  any file in the IDE.
* Also ignore the file created by Visual Studio when using cmake
  integration.
* Use conditional for unity/nounity sources (h/t @mellery451)
2018-10-10 10:25:25 -04:00
Mike Ellery
d70464032c Add dependency for NuDB ExternalProject 2018-10-10 10:19:00 -04:00
Scott Schurr
0bbe6e226c Remove beast::Journal default constructor 2018-10-10 10:18:03 -04:00
Mike Ellery
49e61cc0a6 Improve codecov builds:
- allow private token for jenkins/codecov
- add custom targets for gcc/clang to generate codecov reports
- use CMake coverage target in jenkins build
- optional coverage_test argument when configuring the build
2018-10-10 10:15:10 -04:00
Scott Schurr
6572fc8e95 Implement MultiSignReserve amendment [RIPD-1647]:
Reduces the account reserve for a multisigning SignerList from
(conditionally) 3 to 10 OwnerCounts to (unconditionally) 1
OwnerCount.  Includes a transition process.
2018-10-01 18:17:33 -07:00
Nik Bougalis
3ce4dda5cb Set version to 1.2.0-b2 2018-10-01 11:26:31 -07:00
Edward Hennis
7295cf979b Grow the open ledger expected transactions quickly (RIPD-1630):
* When increasing the expected ledger size, add on an extra 20%.
* When decreasing the expected ledger size, take the minimum of the
  validated ledger size or the old expected size, and subract another 50%.
* Update fee escalation documentation.
* Refactor the FeeMetrics object to use values from Setup
2018-10-01 11:26:22 -07:00
Edward Hennis
e14f913244 Update TxQ developer docs:
* Rename a couple of member variables for clarity.
2018-10-01 11:26:22 -07:00
Joe Loser
cd1c5a30dd Add user defined literals for megabytes and kilobytes 2018-10-01 11:26:22 -07:00
Joe Loser
8dd8433bb6 Remove unused function in AutoSocket.h 2018-10-01 07:40:56 -07:00
Scott Schurr
eeb9d92fb0 Add RPCCall unit tests (RIPD-1634) 2018-10-01 07:40:56 -07:00
Scott Schurr
4104778067 Improve transaction error condition handling (RIPD-1578, RIPD-1593):
As described in #2314, when an offer executed with `Fill or Kill`
semantics, the server would return `tesSUCCESS` even if the order
couldn't be filled and was aborted. This would require additional
processing of metadata by users to determine the effects of the
transaction.

This commit introduces the `fix1578` amendment which, if enabled,
will cause the server to return the new `tecKILLED` error code
instead of `tesSUCCESS` for `Fill or Kill` orders that could not
be filled.

Additionally, the `fix1578` amendment will prevent the setting of
the `No Ripple` flag on trust lines with negative balance; trying
to set the flag on such a trust line will fail with the new error
code `tecNEGATIVE_BALANCE`.
2018-09-30 14:10:40 -07:00
Spec
4dcb3c9199 Avoid dispatching multiple fetch pack threads 2018-09-30 13:54:59 -07:00
Nik Bougalis
b0092aee24 Set version to 1.2.0-b1 2018-09-28 09:15:12 -07:00
Mike Ellery
bb52b04c25 Remove subtrees for soci, sqlite, lz4, snappy, nudb 2018-09-28 09:15:06 -07:00
Mike Ellery
83dac8b382 Use ExternalProject for NIH dependencies
Fixes: RIPD-1648

 - use ExternalProject for snappy, lz4, SOCI, and sqlite3
 - use FetchContent for NuDB
 - update SOCI from 79e222e3c2278e6108137a2d26d3689418b37544 to
   3a1f602b3021b925d38828e3ff95f9e7f8887ff7
 - update lz4 from c10863b98e1503af90616ae99725ecd120265dfb to v1.8.2
 - update sqlite3 from 3.21 to 3.24
 - update snappy from b02bfa754ebf27921d8da3bd2517eab445b84ff9 to 1.1.7
 - update NuDB from 00adc6a4f16679a376f40c967f77dfa544c179c1 to 1.0.0
2018-09-28 09:15:06 -07:00
Mike Ellery
8a4951947d Improve ssl and nih in cmake:
- provide better override handling for ssl dir
- include build type in nih cache for single config
  to avoid cmake cache collision
2018-09-28 09:15:06 -07:00
Mike Ellery
ab6163e989 Remove test sensitivity to error text from OpenSSL 2018-09-28 09:15:06 -07:00
Mike Ellery
5741a8356f Refine json object test for NDEBUG case 2018-09-28 09:15:06 -07:00
seelabs
b2f2d89a08 Support boost 1.68 2018-09-28 09:15:06 -07:00
seelabs
c946043280 Suppress clang warning on intentional self assignment 2018-09-28 09:15:06 -07:00
Miguel Portilla
820546c873 Report fetch pack errors with shards 2018-09-28 09:15:06 -07:00
Scott Schurr
b36e9dd1b4 Remove noisy log write from Stoppable.cpp 2018-09-28 09:15:06 -07:00
Scott Schurr
582d1691a9 Improve error descriptions in JSONRPC unit test 2018-09-28 09:15:06 -07:00
Nik Bougalis
3e22a1e9e8 Set version to 1.1.0 2018-09-14 12:53:38 -07:00
wilsonianb
7b0367730c Set version to 1.1.0-rc3 2018-08-21 13:56:28 -05:00
wilsonianb
8c14002c25 Do not use beast base64 encoding without fix:
Boost 1.67 and 1.68 are missing this fix
0439dcfa7a
2018-08-21 10:05:45 -05:00
Nik Bougalis
c0d396fb3c Set version to 1.1.0-rc2 2018-08-15 20:02:19 -07:00
Nik Bougalis
65d517d0df Don't filter proposals by close time at the wire protocol level:
When validators publish a proposal, they include the close time that they
believe the new ledger should have, and the network attempts to reach
consensus on that.

Instead of delaying consensus if no close time has the required majority
the servers can "agree to disagree"; if this happens, they switch to
proposing a close time of 0, and the network avalanches to that value.

If that occurs, determinstic rules record the new ledger's close time as
being one second later than its parent, and set a flag indicating that
no consensus on the close time was reached.

The wire protocol decoder would incorrectly filter such proposals, so
that they would not be seen by the higher level consensus engine.

This commit removes the low-level filtering, and allows higher level
code to filter out stale proposals instead.
2018-08-15 19:59:55 -07:00
Nik Bougalis
38c3a46a33 Deprecate commands that perform remote tx signing (RIPD-1649):
In order to facilitate transaction signing, `rippled` offers the `sign` and
`sign_for` and `submit` commands, which, given a seed, can be used to sign or
sign-and-submit transactions. These commands are accessible from the command
line, as well as over the WebSocket and RPC interfaces that `rippled` can be
configured to provide.

These commands, unfortunately, have significant security implications:

  1. They require divulging an account's seed (commonly known as a "secret
     key") to the server.
  2. When executing these commands against remote servers, the seeds can be
     transported over clear-text links.
  3. When executing these commands over the command line, the account
     seed may be visible using common tools that show running processes
     and may potentially be inadvertently stored by system monitoring
     tools or facilities designed to maintain a history of previously
     typed commands.

While this commit cannot prevent users from issuing these commands to a
server, whether locally or remotely, it restricts the `sign` and `sign_for`
commands, as well as the `submit` command when used to sign-and-submit,
so that they require administrative privileges on the server.

Server operators that want to allow unrestricted signing can do so by
adding the following stanza to their configuration file:

    [signing_support]
    true

Ripple discourages server operators from doing so and advises against using
these commands, which will be removed in a future release. If you rely on
these commands for signing, please migrate to a standalone signing solution
as soon as possible. One option is to use `ripple-lib`; documentation is
available at https://developers.ripple.com/rippleapi-reference.html#sign.

If the commands are administratively enabled, the server includes a warning
on startup and adds a new field in the resulting JSON, informing the caller
that the commands are deprecated and may become unavailable at any time.

Acknowledgements:
Jesper Wallin for reporting this issue to Ripple.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
2018-08-15 19:59:52 -07:00
Scott Schurr
d3258c7f1f deposit_authorized gives error if source not in ledger (#2640) 2018-08-14 08:46:59 -07:00
seelabs
8a02903fa5 Set version to 1.1.0-rc1 2018-08-08 21:07:57 -04:00
Mike Ellery
dbc8f147c9 Improve subproj handling and deprecated target options:
Exclude several libraries from build when we are included in a
super-project (this is the case when someone only wants to use
xrpl_core). Force several target (deprecated) params to be cache
variables since they are now exposed as options.
2018-08-08 21:07:54 -04:00
Mike Ellery
7a547b8cf2 Add missing sources to build 2018-08-08 21:07:54 -04:00
Mike Ellery
2c13ca0109 Define DEBUG preprocessor symbol for debug builds 2018-08-08 21:07:54 -04:00
Miguel Portilla
a7ed5bfbee Improve shards file exception handling 2018-08-08 21:07:54 -04:00
Miguel Portilla
a73372cb9d Add RPC shard download 2018-08-08 21:07:54 -04:00
Miguel Portilla
658f904ce0 Add shard import support to shard database 2018-08-08 21:07:54 -04:00
Miguel Portilla
9212c28ef8 Add HTTPS file downloader client 2018-08-08 21:07:54 -04:00
Miguel Portilla
5336e3715a Add archive and lz4 extracting 2018-08-08 21:07:54 -04:00
Mike Ellery
c12dbc4386 Add libarchive 2018-08-08 21:07:54 -04:00
Scott Schurr
2901577be7 Remove using namespace declarations at namespace scope in headers 2018-08-08 21:07:54 -04:00
seelabs
4aa0bc37c0 Add delimiter when appending to CMAKE_CXX_FLAGS 2018-08-08 21:07:54 -04:00
Mike Ellery
24d2687f2d Remove empty source file 2018-08-08 21:07:54 -04:00
Mike Ellery
ed5a0bdc3c Correct werr flag for jenkins build 2018-08-08 21:07:54 -04:00
Mark Travis
04745b11a8 Expand SQLite potential storage capacity:
Increase page size for SQLite transaction database upon creation
Provide diagnostics for transaction db page usage.
Shut down rippled gracefullly if transaction db is running out of pages.
Add new rippled maintenance command line option to cause new page size
to take effect.
2018-08-08 21:07:54 -04:00
wilsonianb
9b63f4fb53 Remove executable bit from source files 2018-08-07 14:38:27 -04:00
Joe Loser
8ac6799149 Remove unused SNTP_DEBUG define in SNTPClock.cpp 2018-08-07 14:36:19 -04:00
Nik Bougalis
09050a860b Set version to 1.1.0-b5 2018-07-26 16:06:25 -07:00
mDuo13
32ca1dd6ed Update README with XRP Ledger branding 2018-07-26 16:06:16 -07:00
Mike Ellery
37d9544ef7 Refactor/modernize our cmake:
Switch to target-oriented dependencies. Use imported targets for
dependencies (openssl, boost). Localize FindBoost to remove cmake
version dependence for latest boost support. Logically separate
"ripple-libpp" core sources and add install targets.
Add ninja build for msvc. Add two clang sanitizer builds. Misc script
changes to work with latest modernized cmake.
2018-07-20 08:58:04 -07:00
Mike Ellery
63370b4441 Default to ipv4 for unit tests, add ipv6 option 2018-07-20 08:58:04 -07:00
Mike Ellery
49bcdda418 Improve charge handling in NoRippleCheckLimits test (RIPD-1641) 2018-07-20 08:58:04 -07:00
Miguel Portilla
d89ff1b63d Handle websocket construction exceptions:
Certain versions of the Beast HTTP & WebSocket library can
generate exceptions, which unless caught, will result in
unexpected behavior.

Acknowledgements:
Ripple thanks Thomas Snider for originally noticing this
issue and responsibly disclosing it to Ripple.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
2018-07-20 08:58:04 -07:00
Miguel Portilla
d289512aeb Improve SSLHTTPPeer asynchronous shutdown 2018-07-20 08:58:04 -07:00
Howard Hinnant
d98c4992dd Supply ConsensusTimer with milliseconds or finer precision 2018-07-20 08:58:04 -07:00
Howard Hinnant
d257d1b2c9 Migrate more code into the chrono type system:
Changes include:

  *  Database::tune and all tune overrides
  *  TaggedCache TargetAge
  *  KeyCache TargetAge
2018-07-20 08:58:04 -07:00
Scott Schurr
574ea2c14d Minor optimization of STObject::add 2018-07-20 08:58:04 -07:00
Joe Loser
70d9d88cda Remove using namespace beast in base_uint.h 2018-07-20 08:58:04 -07:00
Joe Loser
79d819584f Replace boost locks and mutexes with std-equivalent 2018-07-16 17:49:42 -07:00
Joe Loser
e222ff5868 Rename data members of ConsensusParms:
Based on a TODO comment in DisputedTX.h, it seems at one point the
data members of ConsensusParms were macros. Now that they are not,
we should spell them like other data members (without all uppercase).
2018-07-16 17:49:42 -07:00
Joe Loser
f58916a2e4 Remove comment about passing allocator to KeyCache:
After some discussion on https://github.com/ripple/rippled/pull/2595
we have decided that the allocator should not be plumbed through
the KeyCache class template. As such, remove the comment suggesting
to push the allocator through.
2018-07-16 17:49:42 -07:00
wilsonianb
7e30897ef4 Increase validation quorum to 80%
All listed validators are trusted and quorum is 80% of trusted
validators regardless of the number of:
* configured published lists
* listed or trusted validators
* recently seen validators

Exceptions:
* A listed validator whose master key has been revoked is not trusted
* Custom minimum quorum (specified with --quorum in the command line)
  is used if the normal quorum appears unreachable based on the number
  of recently received validators.

RIPD-1640
2018-07-16 17:49:42 -07:00
seelabs
cff1abba5d Add .clang-format rules and update code style 2018-07-16 17:49:42 -07:00
MarkusTeufelberger
aa4e3a98f7 Remove cmake conditional that could never be true
Boost >= 1.67 is required, the check was for boost versions <= 1.66
2018-07-03 02:09:33 +02:00
Nik Bougalis
381a1b948b Set version to 1.1.0-b4 2018-06-25 17:12:08 -07:00
Nik Bougalis
873ba1ba9b Merge master (1.0.1) into develop (1.1.0-b3) 2018-06-25 13:53:15 -07:00
Edward Hennis
16b9bbb517 Retried transactions that tec move from TxQ to open ledger:
* Unit test of tec code handling.
* Extra TxQ debug logging
2018-06-25 13:52:16 -07:00
Ian Roskam
7427cf7506 Beast was accepted into Boost:
Link to Beast repository was outdated.  Updated to the boostorg/beast repository.
2018-06-25 13:52:16 -07:00
Scott Schurr
b14bdb068a Standardize on default_prng() for non-crypto shuffling 2018-06-25 13:52:15 -07:00
Mike Ellery
8098cba4c2 Trim space in Endpoint::from_string
Fixes: RIPD-1643
2018-06-25 13:38:05 -07:00
Mike Ellery
68bebc472a only IPv4 allowed with travis 2018-06-25 13:38:05 -07:00
Joe Loser
243e181c08 Replace uses of dirDelete with ApplyView::dirRemove 2018-06-25 13:38:05 -07:00
Joe Loser
b0a1aef43d Replace deprecated usages of std::random_shuffle
std::random_shuffle is deprecated in C++14 and removed completely
in C++17. The two-iterator version of std::random_shuffle usually
depends on std::rand and also on a global state. The preferred
replacement is to use std::shuffle with a pseudo-random number
generator.
2018-06-25 13:38:05 -07:00
Joe Loser
aab47e09b6 Remove static_assert for Boost version 2018-06-25 13:38:05 -07:00
Joe Loser
fc3a3d8267 Remove BEAST_NO_ZERO_AUTO_RETURN in Zero.h 2018-06-25 13:38:05 -07:00
Joe Loser
73fb3f0bfa Mark some move and move-assignment ctors noexcept 2018-06-25 13:38:05 -07:00
Joe Loser
5f8037c55b Apply clang-tidy modernize-use-equals-default check 2018-06-25 13:38:05 -07:00
Nikolaos D. Bougalis
3aaf6d7857 Use Boost.Endian instead of custom wrappers 2018-06-25 13:38:00 -07:00
Mike Ellery
11ab98cced Set version to 1.1.0-b3 2018-06-19 11:56:08 -07:00
Joe Loser
06d0ff6e52 Remove conditional check for using Boost.Process:
- Since we require a min Boost version of 1.67 as of recently (for
  Beast), we also remove the conditional checks that existed for us
  to know whether Boost.Process is available or not. We can
  always assume it is available now.
- Remove runtime checks for minimum Boost and OpenSSL versions
  since they are checked at CMake configure time.
2018-06-19 11:56:08 -07:00
Mike Ellery
5a830b63e9 RPM dev build fixes:
skip signature checks and allow for all branches (not PRs).
2018-06-19 11:25:20 -07:00
Joe Loser
f658656b82 Mark some single-argument constructors explicit 2018-06-19 11:25:20 -07:00
wilsonianb
31e511afcf Fix duplicate validation and manifest suppression
RIPD-1636
RIPD-1638
RIPD-1632
2018-06-19 11:25:20 -07:00
Joe Loser
f0cc7c4c8d Replace beast::SharedPtr with std::shared_ptr 2018-06-19 11:25:20 -07:00
Scott Schurr
6a74d771ee Reduce occurrences of sporadic PerfLog unit test failures 2018-06-19 11:25:20 -07:00
seelabs
833fae57db Use liquidity from strands that consume too many offers (RIPD-1515):
This changes the rules for payments in two ways:

1) It sets the maximum number of offers any book step can consume from
2000 to 1000.

2) When a strand contains a step that consumes too many offers,
currently the liquidity is not used at all and the strand will
be considered dry. This changes things so the liquidity is used,
however the strand will still be considered dry.
2018-06-19 11:25:20 -07:00
Scott Schurr
5097656c83 Add xrpRoundToZero logging for FlowCross compareSandboxes 2018-06-19 11:25:20 -07:00
Edward Hennis
5b733fb485 Remove Transactor::mFeeDue member variable
* mFeeDue is only used in one place by one derived class, so
  only compute it as a local in that function.
* The baseFee needs to be calculated outside of the Transactor class
  because, it can change during transaction processing, and the function
  is static, so we need to be sure to call the right version
* Rename Transactor::calculateFee to minimumFee
2018-06-19 11:25:20 -07:00
Joe Loser
0b2f33d23a Prefer std::array over C-style array in base_uint 2018-06-19 11:25:16 -07:00
Mike Ellery
08382d866b Support ipv6 for peer and RPC comms:
Fixes: RIPD-1574

Alias beast address classes to the asio equivalents. Adjust users of
address classes accordingly. Fix resolver class so that it can support
ipv6 addresses. Make unit tests use ipv6 localhost network. Extend
endpoint peer message to support string endpoint
representations while also supporting the existing fields (both are
optional/repeated types). Expand test for Livecache and Endpoint.
Workaround some false positive ipaddr tests on windows (asio bug?)
Replaced usage of address::from_string(deprecated) with free function
make_address. Identified a remaining use of v4 address type and
replaced with the more appropriate IPEndpoint type (rpc_ip cmdline
option). Add CLI flag for using ipv4 with unit tests.

Release Notes
-------------

The optional rpc_port command line flag is deprecated. The rpc_ip
parameter now works as documented and accepts ip and port combined.
2018-06-19 09:32:54 -07:00
Nik Bougalis
8429dd67e6 Set version to 1.0.1 2018-06-04 16:37:47 -07:00
Nik Bougalis
0439dcfa7a Fix a corner case when decoding base64:
Under some corner cases, the base64 decoder would not allocate
enough memory, which could result in spurious errors.

Acknowledgements:
Ripple thanks Guido Vranken for originally noticing this issue
and responsibly disclosing it to Ripple.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
2018-06-04 16:37:45 -07:00
seelabs
00df097e5f Improve json exception handling 2018-06-04 12:09:48 -04:00
seelabs
fd4636b056 Set version to 1.1.0-b2 2018-06-01 13:29:57 -04:00
Scott Schurr
34d3f93868 Don't read Amount field if it is not present (RIPD-1623) 2018-06-01 13:29:52 -04:00
Joe Loser
57ab0a00b5 Rename member function in NetworkOPs.h 2018-06-01 13:29:52 -04:00
Joe Loser
f0cec3b2f1 Rename LoadEvent member function reName to setName 2018-06-01 13:29:52 -04:00
Mike Ellery
1c2b8417b9 Correct copy in io_latency_probe:
Fixes: https://github.com/ripple/rippled/issues/2521

Copy ctor missed one member. Also added move since we have some rvalues
passed around here.
2018-06-01 13:29:52 -04:00
Mike Ellery
ca29c2b906 Improve unity/nounity description in linux doc 2018-06-01 13:29:52 -04:00
Joe Loser
7c785d0d7c Add missing override keyword:
* Enable the `suggest-override` warning for gcc
* Fix all functions that were flagged by that warning
2018-06-01 13:29:52 -04:00
seelabs
0ae157a5c3 Enable c++-11 for soci 2018-06-01 13:02:52 -04:00
Joe Loser
a6f59081cc Remove deprecated protocol/types.h header 2018-06-01 13:01:45 -04:00
Mike Ellery
201f1aaa39 Prompt for manual approval on non-collaborator PRs 2018-06-01 13:01:10 -04:00
Mike Ellery
ae73878c59 Build an unsigned rpm in dev pipeline 2018-06-01 13:01:10 -04:00
Mike Ellery
cfdc64d7cf Enable manual tests in CI:
Fixes: RIPD-1575. Fix argument passing to runner. Allow multiple unit
test selectors to be passed via --unittest argument. Add optional
integer priority value to test suite list. Fix several failing manual
tests. Update CLI usage message to make it clearer.
2018-06-01 12:57:12 -04:00
seelabs
95eb5e1862 Fix manual offer test 2018-06-01 12:57:12 -04:00
Joe Loser
dc0d5996e2 Convert macros in STTX.h into an enum 2018-06-01 12:56:09 -04:00
seelabs
817d2339b8 Set version to 1.1.0-b1 2018-05-15 16:58:33 -04:00
Scott Schurr
008ff67ac2 Add DepositPreauth ledger type and transaction (RIPD-1624):
The lsfDepositAuth flag limits the AccountIDs that can deposit into
the account that has the flag set.  The original design only
allowed deposits to complete if the account with the flag set also
signed the transaction that caused the deposit.

The DepositPreauth ledger type allows an account with the
lsfDepositAuth flag set to preauthorize additional accounts.
This preauthorization allows them to sign deposits as well.  An
account can add DepositPreauth objects to the ledger (and remove
them as well) using the DepositPreauth transaction.
2018-05-15 16:58:31 -04:00
seelabs
b444196bf9 Remove pre-boost beast 2018-05-15 16:58:30 -04:00
seelabs
27703859e7 Convert code to use boost::beast 2018-05-15 16:58:30 -04:00
Nikolaos D. Bougalis
2ac1c2b433 Improve invariant checking:
Add a new invariant checker that verifies that we never charge a
fee higher than specified in the transaction; we will charge less
in some corner cases where the transacting account cannot afford
the fee.

Detect more anomalous conditions, and improve the logged error
messages.

Clarify the code flow associated with invoking the invariant checker
from `Transactor`, add extra comments and improve naming to make the
code self-documenting.
2018-05-15 11:28:50 -04:00
Scott Schurr
118c25c0f0 Compile time check preflight returns no tec (RIPD-1624):
The six different ranges of TER codes are broken up into six
different enumerations.  A template class allows subsets of
these enumerations to be aggregated.  This technique allows
verification at compile time that no TEC codes are returned
before the signature is checked.

Conversion between TER instance and integer is provided by
named functions.  This makes accidental conversion almost
impossible and makes type abuse easier to spot in the code
base.
2018-05-15 11:28:50 -04:00
Howard Hinnant
7d163a45dc Replace UptimeTimer with UptimeClock
* UptimeClock is a chrono-compatible seconds-precision clock.

* Like UptimeTimer, its purpose is to make it possible for clients
  to query the uptime thousands of times per second without a
  significant performance hit.

* UptimeClock decouples itself from LoadManager by managing its
  own once-per-second update loop.

* Clients now traffic in chrono time_points and durations instead
  of int.
2018-05-15 09:56:47 -04:00
Joe Loser
717f874767 Add missing virtual destructors:
Some classes had virtual methods, but were missing a virtual
destructor.

Technically, every unit test that inherits from the Beast test suite
would get flagged by `-Wnon-virtual-dtor` but I did not think it would
be a great idea to go sprinkle a virtual destructor for every Ripple
test suite.
2018-05-15 09:55:28 -04:00
Brad Chase
681df58b61 Refactor ledger replay logic (RIPD-1547):
Also switch to use ReadView for TxQ updates.
2018-05-15 09:54:00 -04:00
Nikolaos D. Bougalis
f31ca2860f Set version to 1.0.0 2018-05-11 10:29:41 -07:00
Nikolaos D. Bougalis
d702e736ca Set version to 1.0.0-rc1 2018-05-07 11:37:16 -07:00
Brad Chase
6156ff3eb7 Remove validation cookie support code 2018-05-07 11:36:27 -07:00
Joe Loser
04f1388860 Remove extra semicolons:
Several functions had an extra semicolon. This removes them.
2018-05-07 11:36:27 -07:00
Joe Loser
c1c332f0b0 Remove redundant type qualifier:
The extra `const` type qualifier on the return type has no effect.
Clang emits `-Wignored-qualifiers` warning with `-Wextra`.
2018-05-07 11:36:27 -07:00
seelabs
93780c25f7 Resolve gcc8 warnings 2018-05-07 11:31:23 -07:00
Nikolaos D. Bougalis
a442d3fdb3 Set version to 1.0.0-b5 2018-04-29 02:04:37 -07:00
Scott Schurr
7bc163ee4c Add delivered_amount to tx result for CheckCash (RIPD-1623) 2018-04-28 13:46:04 -07:00
Scott Schurr
6bd0b850a0 Fixes for PerfLog unit test in a Docker container 2018-04-28 13:45:25 -07:00
Nikolaos D. Bougalis
1eece9b1fd Set version to 1.0.0-b4 2018-04-09 09:52:30 -07:00
Miguel Portilla
859d18adb0 Add command import node store to shards 2018-04-09 09:52:13 -07:00
Scott Schurr
c4a9b73a66 Add check, escrow, and pay_chan to ledger_entry (RIPD-1600) 2018-04-08 02:34:37 -07:00
Mark Travis
8eb8c77886 Performance logging and counters:
* Tally and duration counters for Job Queue tasks and RPC calls
    optionally rendered by server_info and server_state, and
    optionally printed to a distinct log file.
    - Tally each Job Queue task as it is queued, starts, and
      finishes running. Track total duration queued and running.
    - Tally each RPC call as it starts and either finishes
      successfully or throws an exception. Track total running
      duration for each.
  * Track currently executing Job Queue tasks and RPC methods
    along with durations.
  * Json-formatted performance log file written by a dedicated
    thread, for above-described data.
  * New optional parameter, "counters", for server_info and
    server_state. If set, render Job Queue and RPC call counters
    as well as currently executing tasks.
  * New configuration section, "[perf]", to optionally control
    performance logging to a file.
  * Support optional sub-second periods when rendering human-readable
    time points.
2018-04-08 02:24:38 -07:00
Markus Teufelberger
ef3bc92b82 Rename xxhash.c to xxhash.cpp 2018-04-08 01:52:12 -07:00
Brad Chase
f7a4a94c3b Add cookie to validation (RIPD-1586):
Each validator will generate a random cookie on startup that it will
include in each of its validations. This will allow validators to detect
when more than one validator is accidentally operating with the same
validation keys.
2018-04-08 01:52:12 -07:00
Brad Chase
3dc0714273 Add testnet to example configs (RIPD-1622) 2018-04-08 01:52:12 -07:00
Mike Ellery
deb9e4ce3c Remove BeastConfig.h (RIPD-1167) 2018-04-08 01:52:12 -07:00
Mike Ellery
4bc300e251 Quiet protobuf warning in XCode build 2018-04-08 01:52:11 -07:00
Mike Ellery
d65e208a99 Eliminate objective-c sources 2018-04-08 01:52:11 -07:00
Howard Hinnant
f8fb1f6c7d Remove unneeded macOS-specific code 2018-04-08 01:52:11 -07:00
Howard Hinnant
db3b4dd396 Prevent accidental aggregates
*  The compiler can provide many non-explicit constructors for
   aggregate types.  This is sometimes desired, but it can
   happen accidentally, resulting in run-time errors.

*  This commit assures that no types are aggregates unless existing
   code is using aggregate initialization.
2018-04-08 01:52:11 -07:00
Nikolaos D. Bougalis
b7335fdff5 Remove unused headers for LevelDB and HyperlevelDB 2018-04-08 01:52:11 -07:00
Nikolaos D. Bougalis
d45556ec82 Improve checking of transaction flags (RIPD-1543) 2018-04-08 01:52:10 -07:00
Nikolaos D. Bougalis
cebb9c6604 Remove unused capture in lambdas 2018-04-08 01:52:08 -07:00
Nikolaos D. Bougalis
b7692b7bc1 Remove nodestore dependency on Snappy 2018-04-08 01:52:07 -07:00
Nikolaos D. Bougalis
327377cb2d Use xxhash and remove unused hash functions:
We had several hash functions implemented, including SipHash,
SpookyHash and FNV1a.

Default to using xxhash and remove the code for the remaining
hash functions.
2018-04-08 01:52:06 -07:00
Nikolaos D. Bougalis
75c4dbb0a1 Set version to 1.0.0-b3 2018-03-24 12:54:09 -07:00
Brad Chase
f0b9506617 Remove scons support 2018-03-24 12:53:53 -07:00
Howard Hinnant
b4e1b3c1b1 Remove undefined behavior from <ctype.h> calls:
For the functions defined in <ctype.h> the C standard requires
that the value of the int argument be in the range of an
unsigned char, or be EOF.  Violation of this requirement
results in undefined behavior.
2018-03-24 12:53:44 -07:00
seelabs
02c487348a Remove warning about plantuml:
Plantuml is only relevant when building docs. On most systems it is OK if it is
not installed.
2018-03-24 12:53:37 -07:00
David Schwartz
5db5e31140 Allow relayed ledger requests to check the shard store 2018-03-24 12:53:29 -07:00
Miguel Portilla
4869a0d00e Verify SQLite ledgers exist in node store 2018-03-24 12:53:21 -07:00
Miguel Portilla
d9be0de269 Add shard configuration example 2018-03-24 12:53:11 -07:00
Miguel Portilla
0b18b36186 Make earliest ledger sequence configurable 2018-03-24 12:53:01 -07:00
Nikolaos D. Bougalis
8d9dffcf84 Clarify Escrow semantics (RIPD-1571):
When creating an escrow, if the `CancelAfter` time is specified but
the `FinishAfter` is not, the resulting escrow can be immediately
completed using `EscrowFinish`. While this behavior is documented,
it is unintuitive and can be confusing for users.

This commit introduces a new fix amendment (fix1571) which prevents
the creation of new Escrow entries that can be finished immediately
and without any requirements.

Once the amendment is activated, creating a new Escrow will require
specifying the `FinishAfter` time explicitly or requires that a
cryptocondition be specified.
2018-03-24 12:52:40 -07:00
Nikolaos D. Bougalis
2b8893dfca Merge master (0.90.1) into develop (1.0.0-b2):
The merge also updates the RELEASENOTES.md with the release
notes for the 0.90.1 which were accidentally not included
in that release.
2018-03-24 12:51:23 -07:00
Howard Hinnant
881cd4cfad Limit STVar recursion during deserialization (RIPD-1603):
Constructing deeply nested objects could allow an attacker to
cause a server to overflow its available stack.

We now enforce a 10-deep nesting limit, and signal an error
if we encounter objects that are nested deeper.

Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing this
issues.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled codebase and urge reviewers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit
https://ripple.com/bug-bounty
2018-03-23 14:18:36 -07:00
Nikolaos D. Bougalis
067dbf299c Set version to 0.90.1 2018-03-21 20:39:20 -07:00
Miguel Portilla
8e9495f487 Use lock when creating peer shard rangeset 2018-03-21 20:39:19 -07:00
Howard Hinnant
40dc6b1458 Limit STVar recursion during deserialization (RIPD-1603):
Constructing deeply nested objects could allow an attacker to
cause a server to overflow its available stack.

We now enforce a 10-deep nesting limit, and signal an error
if we encounter objects that are nested deeper.

Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing this
issues.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled codebase and urge reviewers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit
https://ripple.com/bug-bounty
2018-03-21 20:39:18 -07:00
Nikolaos D. Bougalis
d5f981f5fc Address issues identified by external review:
* RIPD-1617, RIPD-1619, RIPD-1621:
  Verify serialized public keys more strictly before
  using them.

* RIPD-1618:
    * Simplify the base58 decoder logic.
    * Reduce the complexity of the base58 encoder and
      eliminate a potential out-of-bounds memory access.
    * Improve type safety by using an `enum class` to
      enforce strict type checking for token types.

* RIPD-1616:
  Avoid calling `memcpy` with a null pointer even if the
  size is specified as zero, since it results in undefined
  behavior.

Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these
issues.

Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
2018-03-21 20:39:18 -07:00
Nikolaos D. Bougalis
25de6b0a5f Remove STValidation::getFlags:
The function is non-virtual and hides the virtual function specified in
the base class.

Falling back to the virtual function in the base class is the correct
solution.
2018-03-15 21:17:49 -07:00
seelabs
9af994ceb4 Set version to 1.0.0-b2 2018-03-15 14:38:09 -04:00
seelabs
5549ff4c8a Fix boost::error_code conversions to int:
* Boost 1.67 removes boost::error_code automatic conversions to int
2018-03-15 14:38:06 -04:00
Mike Ellery
3730d46dad Fix intermittent failure in Subscribe_test:
Fixes: RIPD-1601

Fix intermittent failure in server stream sub/unsub test.
Root cause is LoadManager thread *sometimes* running and causing a
fee change event which got published before our test could
unsubscribe. Fixed by explicitly stopping the LoadManager for this test.
2018-03-15 14:38:06 -04:00
Brad Chase
1507ed66a8 Check consensus hash consistency (RIPD-1456):
These changes use the hash of the consensus transaction set when
characterizing the mismatch between a locally built ledger and fully
validated network ledger. This allows detection of non-determinism in
transaction process, in which consensus succeeded, but a node somehow
generated a different subsequent ledger.
2018-03-15 14:38:06 -04:00
seelabs
3a5a6c3637 Remove unused variables 2018-03-15 14:21:18 -04:00
seelabs
4b2afc8f42 Detect when a unit test child process crashes (RIPD-1592):
When a test suite starts and ends, it informs the parent process. If the parent
has received a start message without a matching end message it reports that a
child may have crashed in that suite.
2018-03-15 14:20:25 -04:00
Mike Ellery
deef322b07 Remove outputDebugString, replace getComputerName 2018-03-15 14:19:29 -04:00
Miguel Portilla
9456b83576 Change permessage-deflate and compress defaults (RIPD-506) 2018-03-15 14:18:32 -04:00
Mike Ellery
864844086e Set version to 1.0.0-b1 2018-03-02 07:37:16 -08:00
Mike Ellery
755849115e Add dev docs generation to Jenkins:
Fixes: RIPD-1521

Switch to pure doxygen HTML for developer docs. Remove docca/boostbook
system. Convert consensus document to markdown. Add existing markdown
files to doxygen input set. Fix some image paths and scale images for
use with MD links. Rename/cleanup some files for consistency.
Add pipeline logic for windows slaves. Add ninja and parallel test run
option. Add make doc target build in build-and-test.sh. Cleanup README
files. Add nounity windows build. Add link to jenkins summary table.
Add rippled_classic build (win). Improve formatting of summary table.
2018-03-02 07:37:15 -08:00
Mike Ellery
605ace7645 Unroll some unity files in the nounity build:
FIXES: RIPD-1597

Add includes, remove unused getStackBacktrace() implementation.
2018-03-02 07:37:10 -08:00
Howard Hinnant
1a245234f1 Cleanup some Json::Value methods:
* Rename isArray to isArrayOrNull
* Rename isObject to isObjectOrNull
* Introduce isArray and isObject
* Change as many uses of isArrayorNull to isArray as possible
* Change as many uses of isObjectorNull to isObject as possible
* Reject null JSON arrays for subscribe and unsubscribe
2018-03-01 15:59:40 -08:00
Brad Chase
20defb4844 Update validations on UNL change (RIPD-1566):
Change the trust status of existing validations based when nodes are
added or removed from the UNL.
2018-03-01 13:27:28 -08:00
Markus Teufelberger
8b909d5c17 Include 2 missing headers:
<cerrno> for ETIMEDOUT
<sys/time.h> for gettimeofday()
2018-03-01 13:25:32 -08:00
Edward Hennis
4a3a40174e Strip down Travis CI support toward future deprecation:
* Remove all builds except cmake gcc & clang debug.
* Time some dependency and build operations, using a custom format to avoid
  interfering with other timers.
* Use Travis's "trusty" infrastructure.
* Install more dependencies via apt.
* Update boost version to 1.65.1.
* Do not run unit tests under gdb - several security features prevent
  it from running correctly.
* Run test job with two processes.
2018-03-01 13:23:49 -08:00
Miguel Portilla
2fee75bfc1 Use lock when creating peer shard rangeset 2018-02-26 12:24:56 -05:00
Nikolaos D. Bougalis
6230204e42 Set version to 0.90.0 2018-02-20 14:18:31 -08:00
Brad Chase
5807ce2127 Set version to 0.90.0-rc3 2018-02-16 12:13:21 -05:00
Scott Schurr
da43775d1b Unit test that NoRipple and DepositAuth are orthogonal 2018-02-16 12:12:44 -05:00
Edward Hennis
099a050c0b Support --unittest-jobs in Test.py scons builds:
* Replace Ubuntu scons dependency with cmake
* Remove dependency installation support for Ubuntu 12.04 Precise Pangolin.
  It is EOL'd as of 2017-04-28. http://releases.ubuntu.com/12.04/
2018-02-16 12:12:44 -05:00
Elliot Lee
6fb2482886 Document AccountLines "out" fields 2018-02-16 12:12:37 -05:00
Scott Schurr
b07da94c31 Remove two inaccurate asserts:
The assert in LedgerHistory.cpp was incorrect since
getLedgerByHash() can return a null object.  The assert has
been seen to fire inappropriately during online_delete.

The assert in LedgerMaster.cpp was simply misguided.  If a
null hash were returned by getLedgerHashForHistory(), then we
should not forward that hash to peers.  The assert only fires
in a debug build, so a release build would forward the request
for a null hash to peers.
2018-02-16 12:06:03 -05:00
MarkusTeufelberger
531e153144 Add missing includes 2018-02-16 12:03:06 -05:00
Brad Chase
079f346efd Improve handling of malformed JSON-RPC requests 2018-02-16 12:00:32 -05:00
seelabs
6d3b2b404d Set version to 0.90.0-rc2 2018-02-13 09:08:17 -05:00
seelabs
9685e756e6 Fix boost compilation issues 2018-02-13 09:08:14 -05:00
seelabs
8e365ea44a Revert "Remove pre-boost beast"
This reverts commit ba43bfc646.
2018-02-12 11:55:59 -05:00
seelabs
9a210cfda5 Revert "Convert code to use boost::beast"
This reverts commit cc9c976b76.
2018-02-12 11:55:59 -05:00
seelabs
060692aad4 Set version to 0.90.0-rc1 2018-02-07 10:55:55 -05:00
Brad Chase
3e9e0c4b53 Set version to 0.90.0-b6 2018-02-03 07:07:11 -05:00
Scott Schurr
88570df135 Allow account_objects RPC to filter by "check" (RIPD-1589):
Fixes #2350 issue on GitHub.
2018-02-03 07:06:56 -05:00
Scott Schurr
c6a307fcd7 Unit test pre-expired offer that also cancels an offer 2018-02-02 20:52:33 -05:00
Brad Chase
fd00e6e035 Use boost 1.66 in macos build instructions 2018-02-02 20:46:37 -05:00
Brad Chase
94c6a2a850 Use LedgerTrie for preferred ledger (RIPD-1551):
These changes augment the Validations class with a LedgerTrie to better
track the history of support for validated ledgers. This improves the
selection of the preferred working ledger for consensus. The Validations
class now tracks both full and partial validations. Partial validations
are only used to determine the working ledger; full validations are
required for any quorum related function. Validators are also now
explicitly restricted to sending validations with increasing ledger
sequence number.
2018-02-02 20:38:38 -05:00
seelabs
1c44c4a43e Set version to 0.90.0-b5 2018-01-29 20:28:33 -05:00
Nikolaos D. Bougalis
7e936187ac Remove deprecated wallet_seed RPC endpoint 2018-01-29 20:28:28 -05:00
Nikolaos D. Bougalis
6328fabd5c Improve log scrubbing:
Per issue #2354, when the log level of a server was configured at
"trace", sensitive keying meterial generated by the `wallet_propose`
command could be written to the server's log file, if one was
configured.

This commit improves the log scrubbing code to account for the
sensitive information generated by a `wallet_propose`.

** Important security consideration **

We still caution everyone *against* executing this command on a
server that they do not control: a malicious server operator could
intercept the generated keypair, or operate a modified server that
returns keypairs that are not securely generated.
2018-01-29 20:28:28 -05:00
Brad Chase
125316f2a6 Update Visual Studio build instructions 2018-01-29 17:59:03 -05:00
Scott Schurr
ed750a84dd Reduce noise in log 2018-01-29 11:56:01 -05:00
Scott Schurr
e0ee58c92d Remove unused beast::currentTimeMillis() 2018-01-29 11:56:01 -05:00
Scott Schurr
6286a9708e Avoid stack overflow on AppVeyor:
o Reduce json_reader max recursion, and
o Use a GCE VM for AppVeyor
2018-01-29 11:56:01 -05:00
Mike Ellery
5e4cb78208 Update macos build instructions:
Added workaround for include path order anomaly with Xcode (discovered
while testing the xcode build instructions)
2018-01-29 11:56:01 -05:00
Howard Hinnant
a8481e369d Better error messages for out of range integral values
* This change passes detailed error messages from the JSON parser
  on the server side, back to the client for inclusion into the
  reply's error message.

* Errors originating from the server's inability to parse are
  reclassified from rpcINTERNAL to rpcINVALID_PARAMS.
2018-01-29 11:56:01 -05:00
Scott Schurr
35cc341544 Unit test that sign_for returns a correct hash (RIPD-1583) 2018-01-29 11:56:01 -05:00
Mike Ellery
f6013c1c37 Sconstruct fix for clang sanitizer build 2018-01-29 11:56:01 -05:00
Mike Ellery
992a390d75 Fix cmake clang build for sanitizers 2018-01-29 11:56:01 -05:00
seelabs
ba43bfc646 Remove pre-boost beast 2018-01-29 11:56:00 -05:00
seelabs
cc9c976b76 Convert code to use boost::beast 2018-01-29 11:56:00 -05:00
Scott Schurr
eaff9a0e6a Set version to 0.90.0-b4 2018-01-17 17:19:23 -08:00
Mike Ellery
2a4eac6eb8 Force boost static linking for macos builds 2018-01-17 16:02:27 -08:00
Howard Hinnant
0ec66b3dbc Limit nesting of json commands 2018-01-17 15:33:15 -08:00
Miguel Portilla
718d217158 Implement Shards 2018-01-17 13:43:54 -08:00
Miguel Portilla
aeda2430cd Normalize SHAMap visit functions 2018-01-17 11:41:43 -08:00
Brad Chase
819ea46bf0 Add RangeSet serialization 2018-01-17 11:21:53 -08:00
Scott Schurr
2d5ddbf1bf Checks (RIPD-1487):
Introduce a new ledger type: ltCHECK
Introduce three new transactions that operate on checks:

- "CheckCreate" which adds the check entry to the ledger.  The
  check is a promise from the source of the check that the
  destination of the check may cash the check and receive up to
  the SendMax specified on the check.  The check may have an
  expiration, after which the check may no longer be cashed.

- "CheckCash" is a request by the destination of the check to
  transfer a requested amount of funds, up to the check's SendMax,
  from the source to the destination.  The destination may receive
  less than the SendMax due to transfer fees.

  When cashing a check, the destination specifies the smallest
  amount of funds that will be acceptable.  If the transfer
  completes and delivers the requested amount, then the check is
  considered cashed and removed from the ledger.  If enough funds
  cannot be delivered, then the transaction fails and the check
  remains in the ledger.

  Attempting to cash the check after its expiration will fail.

- "CheckCancel" removes the check from the ledger without
  transferring funds.  Either the check's source or destination
  can cancel the check at any time.  After a check has expired,
  any account can cancel the check.

Facilities related to checks are on the "Checks" amendment.
2018-01-17 10:00:20 -08:00
Mark Travis
76ad06ef47 Control transaction dispatch rate:
Do not process a transaction received from a peer if it has
been processed within the past ten seconds.

Increase the number of transaction handlers that can be in
flight in the job queue and decrease the relative cost for
peers to share transaction and ledger data.

Additionally, make better use of resources by adjusting the
number of threads we initialize, by reverting commit
68b8ffdb63.

Performance counter modifications:
  * Create and display counters to track:
    1) Pending transaction limit overruns.
    2) Total peer disconnections.
    3) Peers disconnections due to resource consumption.

Avoid a potential double-free in Json library.
2018-01-17 09:23:29 -08:00
Nikolaos D. Bougalis
49b5c42e85 Set version to 0.90.0-b3 2018-01-12 13:42:28 -08:00
Mike Ellery
d9337ad43a Make gold linker less aggressive at discarding boost libs 2018-01-12 13:42:23 -08:00
MarkusTeufelberger
0c22e0262d Link Boost statically only when requested
Fixes #2261
2018-01-10 00:12:23 -08:00
MarkusTeufelberger
593a53253c Poll if process is still alive in Test.py
In some cases this script did not detect properly that the subprocess is already terminated. The "append" loop afterwards caused memory to fill up and the script to fail eventually.
2018-01-10 00:12:23 -08:00
Mike Ellery
0c4f0fdd01 Fix jenkins pipeline for branches 2018-01-10 00:12:23 -08:00
Miguel Portilla
ecd1528197 Update Appveyor dependencies package:
* Use the Visual Studio 2017 image
* Update to rippled_deps17.01
2018-01-10 00:12:23 -08:00
Scott Schurr
cc0ce7163a PayChan and Escrow should ignore DisallowXRP (RIPD-1462) 2018-01-10 00:12:23 -08:00
Scott Schurr
259394029a Support for lsfDepositAuth (RIPD-1487):
The DepositAuth feature allows an account to require that
it signs for any funds that are deposited to the account.
For the time being this limits the account to accepting
only XRP, although there are plans to allow IOU payments
in the future.

The lsfDepositAuth protections are not extended to offers.
If an account creates an offer it is in effect saying, “I
will accept funds from anyone who takes this offer.”
Therefore, the typical user of the lsfDepositAuth flag
will choose never to create any offers.  But they can if
they so choose.

The DepositAuth feature leaves a small gap in its
protections.  An XRP payment is allowed to a destination
account with the lsfDepositAuth flag set if:

- The Destination XRP balance is less than or equal to
  the base reserve and

- The value of the XRP Payment is less than or equal to
  the base reserve.

This exception is intended to make it impossible for an
account to wedge itself by spending all of its XRP on fees
and leave itself unable to pay the fee to get more XRP.

This commit

- adds featureDepositAuth,

- adds the lsfDepositAuth flag,

- adds support for lsfDepositAuth in SetAccount.cpp

- adds support in Payment.cpp for rejecting payments that
  don't meet the lsfDepositAuth requirements,

- adds unit tests for Payment transactions to an an account
  with lsfDepositAuth set.

- adds Escrow and PayChan support for lsfDepositAuth along
  with as unit tests.
2018-01-10 00:12:23 -08:00
seelabs
a307d2d03f Allow channel_verify to specify public key in hex (RIPD-1467) 2018-01-10 00:12:23 -08:00
seelabs
ad4ba44394 Add validated field to account_channels command (RIPD-1466) 2018-01-10 00:12:23 -08:00
seelabs
6b56426719 Unit test logging fixes (RIPD-1568):
* Check message queue before shutting down unit tests
* Logging should be on if not quiet
2018-01-10 00:12:23 -08:00
Brad Chase
44e5e8bccf Make Journal assignable 2018-01-10 00:12:23 -08:00
Brad Chase
edf58820cf Cleanly report invalid [server] settings (RIPD-1562) 2018-01-10 00:12:23 -08:00
Elliot Lee
20cdb4dca0 Update rippled-example.cfg:
The quorum is now calculated automatically and does not need to be
manually specified.
2018-01-10 00:12:22 -08:00
Howard Hinnant
d8bbcf21be Add test for size_varint 2018-01-10 00:12:22 -08:00
Howard Hinnant
7ff6d34a49 Force json commands to be objects
* Null json values can be objects or arrays.
* json arrays are now interpreted as batch commands.
* json objects are single commands.
* null jsons are ambiguous as to whether they are single or batch
  commands and should be avoided.
2018-01-09 16:29:39 -08:00
Nikolaos D. Bougalis
e3499b5df8 Merge master (0.81.0) into develop (0.90.0-b2) 2018-01-09 15:13:08 -08:00
Nikolaos D. Bougalis
4e8c8deeaa Set version to 0.81.0 2018-01-04 05:19:57 -08:00
Nikolaos D. Bougalis
92c987a6b4 Transition to dynamic validator lists:
In connection with Ripple's decentralization strategy, the example
validators.txt file is being updated, and the [validators] field,
which contained a static list of trusted validators, is being
replaced with two new fields:

- The [validator_list_keys] field which contains the keys that can be
  used to sign its recommended validator list.
- The [validator_list_sites] field which specifies the URLs where
  dynamic validator list can be found.

The initial configuration links to Ripple's default validator list,
which is located at https://vl.ripple.com.

To read more about Ripple's decentralization strategy, please visit
the Ripple Dev Blog:

https://ripple.com/dev-blog/decentralization-strategy-update/
2018-01-04 05:19:56 -08:00
Mark Travis
5e4dac41a7 Optimize SQL queries used in handling account_tx:
Profiling and research indicates that the SQLite query planner executed
our existing SQL queries sub-optimally by not using the index efficiently.

Restructuring the SQL query works around this issue and allows queries
to be executed efficiently and without unnecessary delay.
2018-01-04 04:41:00 -08:00
Brad Chase
dc9e9f498a Set version to 0.90.0-b2 2017-12-18 14:38:12 -05:00
Brad Chase
0d3ed84864 Update LZ4 support 2017-12-18 14:38:06 -05:00
Brad Chase
468b1e22f5 Update lz4 to 1.8.0 2017-12-18 14:38:00 -05:00
Brad Chase
c89bb5f4b1 Squashed 'src/lz4/' changes from baf78e7e4..c10863b98
c10863b98 fix : asan error in dctx, due to increased maximum frame header size, reported by Craig Young
1e92bb0af Merge pull request #382 from lz4/installVars
edb434365 updated lz4frame manual
a1f3a27e9 Merge branch 'dev' into installVars and fixed conflicts
af9d72b7f Merge pull request #383 from lz4/blockChecksum
8593ba883 Merge pull request #380 from lz4/dictID
01cdbfb5f lz4c legacy commands are now enabled at runtime based on link/binary name "lz4c"
731cff120 fix minor markdown display issues
aea7d521a better respect GNU standard Makefile conventions
930a69211 cli : restored command -BX to enable block checksum (#322)
77f99d292 restored block checksum capability at lz4frame API level
228e26ee6 Merge pull request #378 from deymo/dev
757497ae3 implemented lz4frame decompression API
4531637ec support dictionary compression with independent blocks
8d597d62d fixed gcc prototype warning
d8aafe2c5 dictionary compression correctly uses compression level
ca2fb166a fixed C++ conversion warnings
bf8daa2fd fixed uninitialization error in lz4frame
31f2cdf4d implemented dictionary compression in lz4frame
1d1737aaf fixed frameCompress example
a82dadfba added dictID inside LZ4F_frameInfo_t
e98a52857 updated Frame specification
76ef6d0ab Allow to predefine FORCE_INLINE macro.
9a967030d Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
c5a015bc3 Merge pull request #376 from ido/patch-1
4891f909e Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
dba4d70a1 updated man page
8aeaf6bb3 Fix typos preventing installation of static lib.
ac5ed8e4b Merge pull request #374 from ferdnyc/patch-2
63d500568 Generate updated man page from Markdown source
ef0655510 Merge pull request #373 from ferdnyc/patch-2
872f767fe Fix formatting of concatenation example
6ad3a983d fix #369
2ef4afeeb Merge pull request #371 from jku/tests-LIBDIR
e14b4c5a3 tests/Makefile: don't use LIBDIR as variable
1525fd1f5 Merge pull request #367 from Chocobo1/fallthrough
7982a905e Merge pull request #368 from Chocobo1/readme
9d453a188 Update README.md
a4ec7fa7b Fix gcc7 Wimplicit-fallthrough warnings
89b9f026c made level 10 a bit faster
a6fd0f9d0 -g compilation flag not by default for lz4 cli
bfc180660 clarified lz4frame api comment (#350)
d18084c65 report where decompression ends (#313)
139a387d1 updated NEWS
0beaa356f fixed minor scan-build warning
775e63ee0 refactored simple_buffer.c example (#363)
7e15e240a added a paragraph on overlap matches
03d8586fc cli accept block sizes with KB / MB prefixes
8c355d17f Merge pull request #360 from Chocobo1/md
1c9d4091d [Doc] Fix markdown
6614be7cc Merge pull request #359 from PierreNav/dev
4e359f5c3 Add DLL files to the INSTALL target
5c97cdfa0 Merge pull request #352 from lz4/resetDCtx
2600a154b fix (minor) g++ compatibility for frametest
e60cbb5ca added test for LZ4F_resetDecompressionContext()
2012e4de9 fixed c_standards tests
a90212761 bumped version number to 1.8.0
b8575f2d2 updated Makefile
fe932c452 expose LZ4F_resetDecompressionContext()
1efa48831 minor readability changes
a8dd86d93 changed macro HEAPMODE into LZ4_HEAPMODE
11bfedb6c Merge pull request #349 from svpv/master
6cf1f7d75 lz4hc.c: clamp compression levels > 12
a41df535f lz4cli.c: fix a comment: LZ4HC_DEFAULT_CLEVEL -> LZ4HC_CLEVEL_MAX
f3460fc14 liz4hc.h: fix a comment: LZ4HC_MAX_CLEVEL -> LZ4HC_CLEVEL_MAX
37ef330cd Merge pull request #348 from terrelln/deprecate
0b3e80704 [LZ4F] Allow users to disable LZ4F_DEPRECATE
31e9ed612 Merge pull request #347 from lz4/negativeCLevels
e2c9b1912 lz4frame : Added negative compression levels
7eecd32c0 ensure lz4f_cctx internal buffer size remain valid in case of malloc error
e169edac0 lz4frame : control lz4 context creation success
e2827775e make __packed memory access default for gcc
3d4ee35da Merge pull request #344 from lz4/LZ4F_getFrameInfo
6226d52ea fixed minor Visual warning
b88df6b1b Improved comments on LZ4F_getFrameInfo()
fc31257ab added LZ4F_resetDecompressionContext()
f0a7651fc Safer LZ4_getFrameInfo()
ab547a0ef Merge pull request #342 from iburinoc/isatty
9b1f00056 Fix inline compile errors
afde27acf Fix IS_CONSOLE returning 1 for NUL on windows
84246b974 Merge pull request #341 from iburinoc/exematch
4567d0def Merge pull request #340 from lz4/optlz4opt
e9c3b14f2 Ignore extensions in exe name matching
38362c1d6 Merge pull request #338 from ldv-alt/dev
60b6d2907 improved level 10 speed for degenerated cases
8f8fc52fc Merge branch 'dev' into optlz4opt
0863931a1 fixed API comment for LZ4F_createCompressionContext()
dab3590fc Export deprecated symbols declared in lz4hc.h
13f5cb2d5 Merge pull request #337 from ldv-alt/dev
b1daffc4e cli: add GNU separator -- specifying that all following arguments are files
68044acfe Merge branch 'optlz4opt' of github.com:Cyan4973/lz4 into optlz4opt
b8bc70022 minor refactor
cea8f60df slight btopt speed improvement
f513020a6 slight btopt speed improvement
8bd32a17b made SET_PRICE macro more usable
118b47f3d improved lz4opt speed (~4%)
eff6166eb minor price function optimization
0d073d4d2 added `extern C` for lz4.h static section
aae447fff LZ4_compress_HC_continue_destSize() works as intended up to level 10
cd35f0d98 LZ4_compress_HC_destSize() uses LZ4HC_compress_generic() code path
6b2a1b360 updated lz4 version number
430b7d32b created LZ4_HC_STATIC_LINKING_ONLY section
9144520dc fixed Visual compilation error
a33bf89c6 fix #332 : do not modify /dev/null permissions
e945a27f9 fix #333 : expose obsolete decoding functions
e1b2b370e added Oleg @remittor contribution
b9132bedc Merge pull request #331 from ldv-alt/dev
50fe87f13 Merge pull request #329 from remittor/dev-hc2
883ebdcee Export only those symbols that are part of public API
25b243588 Export deprecated symbols
7aeecbff7 Explicitly set visibility of public API functions when gcc is used
93b3b08ef Merge pull request #330 from inikep/dev
cb1671955 bench.c: respect LZ4_MAX_INPUT_SIZE limit
517d446d7 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
36842ebb1 lz4hc: Fix LZ4HC_compress_hashChain for backward compatibility
baa155088 lz4hc: Fix LZ4HC_compress_hashChain for full support destSize variant
66b26a389 tests: fuzzer: Add test for LZ4_compressHC_destSize
f007153e3 lz4hc: Add LZ4_compressHC_destSize and LZ4_compress_HC_continue_destSize
45b592b7e lz4hc: Cleanup function LZ4HC_encodeSequence
534f8fa5d lz4hc: Cleanup function LZ4HC_compress_hashChain
9141ada8d Merge pull request #328 from ldv-alt/dev
0dfb0b9da Fix test-lz4-basic
97df1c978 updated LICENSE
5bfdff96b Merge pull request #320 from inikep/dev
d3eaf7883 remove MOREFLAGS from test-lz4c32
5bb6ba794 Merge pull request #319 from inikep/dev
ca2c5fd3c CFLAGS=-I/usr/include/x86_64-linux-gnu
ace22424d circle.yml: fixed lz4c32 target (2)
0d77c85d9 circle.yml: fixed lz4c32 target
24007bb4e Merge pull request #318 from esiegerman/eks/mklib
4ee9bd5c3 improved Circle CI tests
c0a8d0ad8 added circle.yml
3ec1591db Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
68a358769 Avoid fseek()'s 2GiB barrier with MinGW
40ad1e85d upgraded util.h
eb51b2b8d Explicitly create $(DESTDIR)$(LIBDIR)/ at install time
69dbafc1c Merge pull request #317 from esiegerman/eks/no-andand-oror
b89cac7b2 Don't use "foo && false || true"
7e57c398c Merge pull request #316 from inikep/dev
f4b9c1d03 Merge pull request #315 from bebehei/patch-1
c139eb40b added OpenBSD NetBSD SunOS to list of supported `make install` OSes
5e1a925be update repolink in makefile
272699a4e Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
c7b14967a updated platform.h
4f261b828 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
04e5eaf9b added DragonFly to list of supported `make install` OS
04ec09269 updated comments on block sizes
7ed257191 Merge pull request #312 from mgorny/cmake-pc-version
96e150e7d [cmake] Fix substituting version in liblz4.pc
67abd75e6 Merge pull request #307 from inikep/dev
8e1fd97d1 lz4frame.h: added Introduction
d56ee32b3 updated gen-lz4-manual.sh
f54c7e0e7 added lz4frame_manual.html
9071df0fa lz4frame.h: prepared to generate manual
6a5633e99 lz4 manual updated to v1.7.5
f9f48f8ed lz4.h: improved manual generation
128244371 improved gen_manual
489d1618c added "This Makefile is validated for"
339e79320 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
ca083dd4d Merge pull request #304 from Chocobo1/cppcheck
db6f733ec Use logical or instead of bitwise or
44f95e92e Fix printf specifier
9683a1ae8 LZ4_MEMORY_USAGE can be modified from compilation command line
16d69c8d0 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
05e27ade6 updated NEWS
c18a81c02 Merge pull request #296 from mgorny/cmake-fixes
d7969e49a cmake: Install manpages
1380c33b7 cmake: Install lz4cat and unlz4 symlinks
28db4acc9 cmake: Support building shared & static libs simultaneously
9e867db90 cmake: Fix SOVERSION to match Makefiles
cb9599449 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
8c4de60d0 Merge pull request #295 from inikep/opt-parser
d2b51c22d fuzzer: tests more compression levels
312d88249 removed nextToUpdateBT
3d5bb3897 clean logging
1c80b9af4 LZ4HC_getSearchNum
581da2bf8 Merge pull request #293 from inikep/dev
98f9d6c72 improved logging
12aae846b fixed table update in LZ4_loadDictHC
c8b31263c added FUZ_CLEVEL_DEFAULT
7a73c5c1f changed default PREFIX and MANDIR
d57ff6456 changed default PREFIX and MANDIR
a3d61cf3d improved test-lz4-opt-parser
20381a2fc fixed -BD compression
f4575f4f1 added test-lz4-opt-parser
e6536faf3 lib\Makefile: fixed INSTALL_DATA
efaa9b7bc Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
b22e2bed5 BSD: improved "make install"
ea51ad0ce Solaris: working "make install"
19c0f21b0 updated Makefile : lz4 no longer recompiled when already up-to-date
9b4b081c1 "make test" is now compatible with Solaris
b41f9bb13 Merge pull request #292 from inikep/dev
7eb16d97d updated NEWS
52cac9a97 updated a few macros names
fea95c156 use FindFirstFileA and FindNextFileA on Windows
e6af952f8 improved comments
581245063 use fseeko for 32-bit MacOS
f084b6538 test Large File support for Mac OS-X in 32-bits mode
973bc7974 util.h and platform.h based on zstd
70526a11e fixed lib/clean
bc0839c5d minor fix for travis-install test
7cf0bb97b LZ4F_compressBound(0) provides upper bound for LZ4F_flush() and LZ4F_compressEnd() [#290, suggested by @vtermanis]
385cb4f53 minor update Makefile
4f61505cc datagen.vcxproj: CharacterSet is MultiByte
253ac12cb VS projects: CharacterSet changed to MultiByte
287f7d3d6 added IncludePath
2fd7eb554 fixed Visual Studio compilation
3c6ce9c66 fixed gcc 4.4 support
64cbc4e1d improved MinGW support
9546ba62d executables use basic types from util.h
452cc970d reordered definitons in platform.h
090cb887a improved formatting (2)
e1332241a improved formatting
5f0cc8e2d fixed gcc warnings (2)
2efedaf7d fixed gcc warnings
6adf05d1e improved platform.h
2cfef1395 executables use platform.h
bb69cdf25 updated util.h
45b14ab8e added platfom.h
913b98fd6 fix `make` concurrency build (#277)
fec57e49e Merge pull request #287 from inikep/dev
4f050b68d tree update dependent from compression level
838ed341f fullUpdate as a parameter
b3ecc3e3e last match starts at least 12 bytes before end of block
d694bf911 faster opt compression
e7c42d757 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
a22e71d4a full binary tree update
626552671 updated NEWS
de93e9e5d fix #285 : lz4cat doesn't work with stdin (reported by @beiDei8z)
fb6c98c85 slightly improved lz4opt.h
20183ad7a Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
0280cf40d fixed path-1, as suggested by @t-mat
7e2d4c4c0 fixed test-lz4c32
750a50cf6 fix #284, reported by @beiDei8z : added test cases for path/to/lz4cat
01ffed25a fix : invocation of path/to/unlz4 (#284), reported by @beiDei8z
8c2f23c13 fixed LZ4_STREAMHCSIZE
76228f064 LZ4HC_DICTIONARY_LOGSIZE 17
8bfa42e61 improved ratio
128903824 15-bit LZ4HC_HASH_LOG
410ba53db improved LZ4HC_BinTree_GetAllMatches
1833be1cf fixed gcc warnings
f2ebf37bf slightly improved ratio
77b051ed7 improve code formatting
e3fee9474 LZ4HC_MAX_CLEVEL = 12
757ef1d5f removed LZ4HC_GetAllMatches
cfe6a24b0 updated LZ4HC_get_price
2113ead17 17-bit LZ4HC_MAXD
c1ef7a177 introduced LZ4HC_compress_hashChain
3f430daf7 added lz4opt.h
5ae0e789f Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
5eb547a60 fix #283 : implement LZ4_versionString().
9a6f7dfc5 updated NEWS
a9a8dbf7c fixed minor display error when using -r
6b90cfbea Updated man page
009f06e01 minor man page update
b3bc054d0 Merge pull request #279 from t-mat/md-to-man
544290d58 Add "man" target to Makefile
018ddf799 numbering
605512a7b updated NEWS
f4c74e6fa library release build compatible with environment variable
edc132b69 lz4 release build compatible with PIE
7819d5878 new test case with fPIE
de0cf5de6 fixed LZ4IO_compressMultipleFilenames
65a542579 updated README.md
2cbd0233b Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
cc21115cb minor warning fix
8875e7dbb fix minor analyzer warning
8b233b228 added a few dates
ba7a85b46 fixed minor analyzer warning
3bde02a05 added gcc 4.4 test
2fe3aa985 added examples to make all
742f2b683 added doc in code comments
2dbcebfda changes travis test order
b4db9d7e3 Add C++14 standard [[deprecated]] attribute
a38cbf9f6 fixed : shadow global variable on gcc 4.4 (minor)
19df3029e fixed #272 (compilation fails on gcc 4.4), reported by @totaam
519932afa attempt -mx32 tests on Travis CI
a91983daa highly improved speed on -mx32 mode Now -mx32 is fastest mode on x64 CPU
6cabcd151 removed usan32 tests. Let's summarize : gcc-5 and gcc-6 on Travis do not support UB sanitizer. And clang on Travis is not able to generate -m32 code. Great. It makes the combination impossible to test on Travis ... :(
7a22904a7 moved usan32 tests to gcc-6
ffc8c18c1 usan32 relies on default compiler (by default)
a4f4bf21e fixed gcc-5 conf
71814a26f switch usan32 tests to gcc-5
8e1ea59ad attempt to fix sanitize32 package dependency
8f536a43b added sanitize32 tests to Travis
3f6f57768 fixed strict aliasing warnings in fasttest
c0e55ec05 fixed versionsTest
7d5354847 added v1.x.y to versions tests
279344a17 fixed Travis test cases
1abecbc33 fix 32-bits mode. Large File support for Mac OS-X in 32-bits mode Fixed potential undefined behavior Changed makefile for 32-bits mode
7fde7438d Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
a22ca9378 updated README for v1.7.3
e9bec0fdc Merge pull request #271 from inikep/dev
b1519f861 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
2dee7bdaf include NEWS in lz4_x64.zip and lz4_x86.zip
632ab3b8d updated programs\README.md
8b48d5e44 updated example\README.md
11f6217cc created dll\example directory
873206e70 appveyor.yml: create only lz4_x64.zip and lz4_x86.zip
7b6a9d863 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
2e10aab20 MinGW doesn't require the import library at all
7cecc7b08 fixed a limit case scenario
00fa1b9c0 fixed double ;; for Visual
929badd61 fixed sign conversion warning
d613e2494 restored .travis.yml
d2be69b14 fixed minor coverity warnings
8c32a12f1 .travis.yml for coverity
2eddc8ea2 removed .travis.yml, for better rebase
2906c373c fixed appveyor badges
0d6dc079f update appveyor tokens
b44790cf9 Merge pull request #270 from inikep/dev
4235e5e25 updated visual\README.md
257b65a1d ToolsVersion="4.0"
200746427 removed _x64 and _x86
e583dd056 ImageHasSafeExceptionHandlers=false
eb160e1a2 appveyor.yml: added liblz4_x86.zip
45b8623af updated lib\dll\README.md
5625cc21e appveyor.yml: generate DLL package
c093e256d added files used by DLL package
be471c69d added fullbench-dll.sln that works with gcc/MinGW
f71340ce1 DLL dependencies moved to lib/dll/
f3f59bef4 updated appveyor.yml
b0e994cc5 upload executables to GitHub
d84a80e66 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
923cf4447 bump version number
4c5c711b7 fixed minor conversion warning
f094f5314 silence a minor msan warning
0fe5c187a fixed minor msan warning
201149f8e fixed conversion warnings
fa158b411 fixed nameless union warnings
84af32d49 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
5a5ebeec1 clang and g++ tests restricted to native only on Travis CI Ubuntu v12 environment
569897ddd Merge pull request #269 from inikep/dev
1b24cc115 fixed conversion warnings
9ad7508db Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
ecc55d19b fixed uintptr_t on pre-c99 compilers
3d3ccc154 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
d5c9b39dd minor code refactoring
4bd9db1dd Merge pull request #268 from nemequ/dev
f42130ee4 cli : add --help command
84b97f90b fuzzer code refactoring fix g++ conversion warning
151021a3a cmake: add public domain dedication, and a note about the maintainer
b6bcb84ea cmake: make PIC optional when generating a static library
94542d845 made lz4hc explicitly dependent on lz4
bb97e8a6f fixed __GNUC__ macro
cb2230826 fixed fuzzer error
3580d9698 enabled deprecation warnings on remaining obsolete functions
874f3e095 update code comments
e528a82f0 Merge pull request #267 from terrelln/strict-aliasing
3d456bdf2 fixed conversion warnings
bf6dda69e updated frametest
85aeb0e4b Expose internal types to remove strict aliasing
dbfdd5131 Merge pull request #266 from terrelln/frame-ex
7aad356ff Add frame decompression example
ec6fb477b added visual\README.md
9d075580d Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
4f9db1383 updated xxhash lib
c72d2f5b1 minor typo
7d2e4b3cf Merge pull request #265 from terrelln/rm-zstd
5b37837e6 Fix license and remove references to zstd
1f246a989 Fixed #178 fullbench on small input
5e13a6ec4 improved man on compression level
f9c9bb364 Merge pull request #263 from terrelln/dict-ex
cd4462629 Merge pull request #264 from inikep/dev
52251d970 Using MinGW+MSYS to create DLL
72a4a84df udated README.md
2ecce6f15 lz4.h: added Introduction section
93e2477ca added lz4_manual.html
a340b734b added contrib/gen_manual
2278d1f02 custom block size
d57d3e4b4 better MSYS/MinGW support
74f127787 NB_LOOPS ?= -i1
cf8ca581d reordered Travis CI tests
767a23160 changed FUZZER_TIME
6a160b72e Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
924f00c86 fix empty body
94917c9a0 Add dictionary random access example
f34808e4a fixed some static analyzer warning
46f74d79b fixed sparse on stdout
fe48cef50 fixed multiples files on lz4cat (#184, by @libor-m)
1b1680594 fix unused variable warning
1b95d60c6 fixed synthetic benchmark mode slightly optimized test mode
acc48a3f8 fixed lz4cat changed lz4-test order
0c5e2e4d1 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
fafba3973 updated NEWS
2987eb417 Merge pull request #262 from inikep/dev
063b4d92e fixed make install
773ae50b4 DLL exports only functions defined in liblz4.def
ca80a7c8f create DLL with MinGW/MSYS
67fd5dffd updated appveyor.yml
c794829a2 update .gitignore
9e7fe1062 removed <PlatformToolset>
dd14dca15 LZ4_DLL_IMPORT
325babddf added lz4.rc
ef2fdfe3d rename liblz4-dll to lz4-dll
7df658ec1 removed LZ4_DLL_EXPORT=1 (2)
4a6e7327c removed LZ4_DLL_EXPORT=1
bde0d340b added fullbench-dll.vcxproj
da48dbb57 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
fe5148417 introduced LZ4_COMPILED_AS_DLL
d5903c8e5 Merge pull request #261 from terrelln/clean-fasttest
d594d9d87 Remove fasttest when cleaning
bd88e4007 Merge pull request #260 from nemequ/dev
661c48040 cmake: significant changes to accomodate a new "bundled" mode.
ce2202690 Merge pull request #259 from inikep/dev
96ea2e363 updated IntDir
39157c9f3 added lz4-dll project
ec63747de VS project: added IntDir
c6fd78f4f VS project: added setargv.obj
1d22fae1e added test-lz4-speed.py
85f861fc4 bench.c: support notification level
2c788b372 joined arm and aarch64 tests
0a22bf7d6 err0r(LZ4F_errorCodes code) uses ptrdiff_t
d06ab4f41 test powerpc64
230264913 tests\Makefile: added test-platform
c201327f1 added platformTest
0f73ed572 new Travis tests
90981a35c Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
c7ab95faa slightly improved HC compression ratio
e9089bcbe small compression ratio and speed improvement on small files
00c6370ad Fix #243 : detect and report fread() errors, by @iyokan
f6a7c6b57 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
301c4e94e Merge pull request #258 from inikep/dev2
839085f05 Merge pull request #257 from inikep/dev
9b3dcb325 MOREFLAGS="-Werror"
7fbb163f3 added "make c_standards"
b6c407886 condensed travis.yml
2fbd7a512 updated travis.yml
ef54f9f9d added --rm option
d61bb74f2 updated declaration of LZ4F_freeDecompressionContext
7a98db266 added datagen.c to lz4.vcxproj
4b29af350 TreatWarningAsError=true only for debug
90d4401fe fixed appveyor.yml (3)
434d1230e fixed appveyor.yml (2)
58ead8dd2 fixed appveyor.yml
98be00847 Copy owner, file permissions and modification time
158156729 util.h requires CharacterSet=MultiByte
ef9d1b2d4 fixed --test
112499a73 Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
58d7a619a Merge pull request #254 from terrelln/dev
920bf2171 Fix LZ4_decompress_fast_continue() bug
207f478ba minor refactor
8195ba8f7 Updated man page
96565816b restricted deprecation warning to fairly recent gcc versions (>=6.0)
eaad740ac lz4frame obsolete enum should trigger deprecation warnings (on supported compilers)
079d5dd54 removed support for old error code enums from lz4frame_static.h, to free up namespace. note : lz4frame_static.h does not guaranteed API stability. note 2 : the macro to enable old error code enums is still present. Just needs to comment one line to re-enable them.
473acf5aa Merge pull request #252 from inikep/dev
10c5d5eee Merge pull request #253 from terrelln/exit-code
86a24c80f Fix typo in lz4 manpage
f30c56c08 Quiet gcc-4.6.3 narrowing warning
374090c7a Return error if input file does not exist.
fbede33fd fixed Travis tests
8ddaddc2d updated #include in util.h
f878c08b7 better correctness on big-endian 64-bits platforms
26dec498c small compression speed improvement on 64-bits systems
e8a6067e8 updated lz4cli.c
8b8c726a5 bench.c based on zstd
e06fcd903 redefined VOID for Windows/MSYS
e7648f4fc added -r option
6ebf8859e Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
4c496faba Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
6d6a3e0fb Merge pull request #250 from terrelln/test-mode
4c6610709 minor code refactoring
136caa552 Rename passThrough to testMode
eda8c4eb1 fix #198 : no longer requires to restart streaming decompression from where it stopped. It permits relocating remaining data into another memory space. Still, the same content must be presented.
04d728b87 Fix tests for test mode
4791cda04 fixed strict warnings
258a5e7fa updated comments
3dcafd35d Fix test mode and write to null
6f68a1692 fixed travis badges
99d11120b Merge pull request #249 from inikep/dev
46b49af37 fixed clang conversion warning
c66108ceb check fread() operation results (#243)
8847faffb updated programs/README.md
f38adde9e versionsTests moved to tests/
74ba377d7 updated .travis.yml
9f15a4c6e updated Visual Studio projects
84cedb463 updated links to LZ4 repository
e624978ea test programs moved to tests/
88cef262e documentation moved to doc/
b768da8ad Merge remote-tracking branch 'refs/remotes/lz4/dev' into dev
59adfd82d fix for ARM platform
67500142c Clarified license (#115, #244)
f17302769 moved cmake and debian directories to contrib (#245)
d05ca2ff6 fixed #247, reported by Felix Bolte
7e112cfa5 fixes static compilation under Windows (#246)
e64a92692 Merge pull request #242 from vapier/dev
92408d338 README: fix minor typo
52cdb4a5f Merge pull request #241 from vapier/dev
c5cb4aa52 Merge pull request #238 from kou/add-lz4lib-api
544abaced Merge pull request #240 from kou/add-missin-ext
43b631223 add a flag to disable static libs
829941e5a Add missing $(EXT) to lz4 program name
59771f41c Use different prefix for each file to avoid conflict
1ca408105 Merge pull request #239 from kou/use-lower-case-for-header-name
012c66536 Add LZ4LIB_API
46a7b94e8 Use lower case for header file name
3b9439ccd updated uninstall
8eae1320c Merge pull request #236 from inikep/dev
33993542a removed test artefacts
3eb143fff fix bug #232 at https://github.com/Cyan4973/lz4/issues/232
c8a78dba1 datagen depends on lz4.h for version string
53515b05e Merge pull request #235 from inikep/dev
92ee3e0ea fixed LZ4 repository links
39be6f16a versionTests require 32-bit gcc libraries
122fdd13a versionTests require python3
6adf7392a updated .travis.yml
8ac619afe Merge pull request #233 from inikep/dev
58a1616d7 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into dev
c77fdb44f disabled <RunCodeAnalysis> for Win32
28f98647e lz4 version source from lz4.h
dd51ca630 frametest depends on standard C time.h only
f327a8bdd fullbench depends on standard C time.h only
32a31663c fuzzer depends on standard C time.h only
64f556e61 lz4cli version number from lz4.h
5871585b6 coding style change
e5b0f1ea1 CFLAGS  += $(MOREFLAGS)
829ae7587 remove EnablePREfast=true for Win32
547e1943c set OutDir
ab707e8b0 update IncludePath
3a7b4457a restored lz4.vcxproj
81d90609b added appveyor.yml
c200f7e09 renamed VS project
f4111cf04 Merge pull request #231 from inikep/dev
b3ae49c7b added <string.h> for strlen()
2cf32ef0f -b# and -e# options from zstd
c1864f69e fix mingw64 compilation warnings
7f08131f9 Merge pull request #226 from gsauthof/port-make
3fa60044e don't use variable eval syntax in if()
e1e3305db don't link the tools against the shared library
b0d42cb7e remove additional -fPIC switch because
681d5010d use gnu c99 to get both c99 and POSIX
584e1fe92 request POSIX.2001 definitions for fileno()
5bd3eaa48 minor comments
9578ff539 minor refactor
a7672dc4f minor decompression speed gains
182645ee0 lz4 as phony target
0214638eb Makefile : lz4 at root
63d22e7a1 minor comments clarifications
74e6afc5d minor comments clarification
2ac9ecec6 Fixed #157 : LZ4F_getFrameInfo() fails on valid null-content frame
45ea27949 Minor fuzzer code refactoring
de69eca47 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
828332d23 fixed : fuzzer test, reported by @setharnold (#203)
b130fd592 minor refactoring changes
3c295ca4a Merge pull request #222 from kou/fix-build-error-with-mingw
0c63b7cbd Fix build error with MinGW on Linux
7316584ca Merge pull request #219 from BenWiederhake/dev
d3e2d8026 Use https wherever possible
cec38d1b7 Correctly reference self (github, not googlecode)
654003003 Use correct 'repository' hostname
91cce757f Updated xxhash library to v0.6.1
5540f4f93 stronger compilation flags
19e585cc4 modified : some more deprecated functions from lz4hc.h generate warnings
6b1600f41 changed : default benchmark LZ4_decompress_safe()
6923f3e23 Minor refactoring (code style)
410ec8154 Fixed : alignment warning
bc342ab94 minor refactoring
c8a573d32 very minor decompression speed gain
a5508b1ad fixed display result issue when benchmarking multiple files
e27e7316d minor code refactoring
2d7df8b30 Reduced bench dependency to standard C <time.h> Faster tests More accurate results on systems with high resolution clocks
8938e1074 minor compression speed improvement
c77c0c108 minor refactoring
3c0332600 minor refactoring (coding style)
1f47f3f74 Merge pull request #208 from Nimloth/dev
7bfbccb97 Add FreeBSD to install targets
5bb5d6ee1 Merge pull request #204 from irwand/dev
bc70a4b72 fixed non-C99 compilers
10e3eecc5 fixed non-C99 compiler
973baa2c1 Merge branch 'heapmode_alloc_failure_fix' into dev
3e5bb6753 alloc failure fix
8a42ee10a Merge pull request #201 from cernekee/cmake-pc
4bda09fc8 cmake: Install liblz4.pc
8b562a4f3 Merge pull request #197 from Cyan4973/fix-readme-travis
5cdf9a657 Fix wrong link for Travis
220e0b606 Merge pull request #194 from jzhuge/dev
cf6652f9b lz4cli: print library version
60ba8638c Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
219da951d Added zstd
2995a45e5 Merge pull request #185 from jwerner-chromium/dev
dc868cd5b Allow for safe in-place decoding
4fcb2e17f Remove whitespace from ends of lines
ad564c872 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
db53bbb94 updated hyperlink
b84eb3349 Merge pull request #182 from kou/add-missing-ext
b8abffc39 Merge pull request #181 from kou/fix-build-failure-on-mingw
ab3e1d253 Add missing $(EXT)
14b40fb6d Fix build error on MinGW
62bbd102f Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
2785eeedf clarified man page as suggested in #170
631dc8f2b Merge pull request #173 from flandr/liblz4
d67ae8fe4 Merge pull request #172 from psteinb/add_static_cmake
4c9d851eb added static build target for OSX and Linux to cmake
495ea43d9 Merge pull request #162 from KyleJHarper/20151021_examples
7aa004377 Forgot to make clean before commiting.  Removing binary.
7f96babc4 Several changes to address a few concerns from Yann.  See Google Group LZ4c topic 'Reusing compression/decompression resources' for details.
d1be164e4 Added a note about why char* is used.  Also removed a binary that wasn't supposed to be in there.
0d2518567 Finished the two example files.  Decided to avoid adding anything to lz4.c/h to expose LZ4_compress_generic().
02be6631b Took out the basics and placed them into basics.c.  Added decompression and a wrapper for the generic call.  I will likely break this file up into 2 examples before submission.
67b3a2470 Final tests and reporting are done.  As expected there isn't much to be gained by jumping the chain.  In most of my tests I did see a moderate performance gain when invoking LZ4_compress_generic() directly with normal text.  This could very easily be an edge case.  Either way it's interesting and worth sharing.
bdd9143e0 Didn't add my file for some reason.  Odd.
73dd936b9 Wrote the skeleton of the performance test.  Need to finish tests for the rest of the call stack and then my own copy of generic().
2c8013812 heapmode macro option for lz4hc
ff3324b8d Fix : compiler-independent macro to remove deprecation warnings
64f8b037e clarified lz4 license
2df602568 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
7d4575e54 fixed link typo
012ab2f52 Merge pull request #152 from dcolascione/dev
94a0954b1 Remove .dir-locals.el
4f1ff58f8 Tweak error message
975c06b1a Improve error messages for attempted console IO
6f1fe1b70 Add .dir-locals.el for setting tab-width
0011033e3 fixed #150 : blank lines when converting lz4.1 to .ps or .pdf
248553c5e minor : show in table that frame checksum is optional
cd18ed308 create link to dynamic library without using `cp -a` (#147)
fb6fece77 Updated Makefile and .travis
8a750aa35 Add static library target for msbuild
13f12aa02 Wrapped possibly empty string (Emscripten) with quotes (@Fordi)
4a5a003c4 updated cli help
9aa5504f0 New unalign access method
e5aee601e updated xxhash
e64345506 Merge pull request #142 from bioothod/master
d15ac1f1a Slightly improved compression speed on ARMv6
0f2bf0c54 Improved performance on ARMv6
fb4d3ef2c Added .gitignore
4b4384772 Updated xxhash to r41
2a6ad2a7f debian: changed name lz4->liblz4
2f3e4bf32 debian: changed name lz4->liblz4
31fed3167 debian: changed name lz4->liblz4
309d1e087 debian: remove sbin dir
2131eedf7 debian: renamed lz4->liblz4, updated installed library files
b47786b9e debian: updated debianization to cdbs
a8f33076a cmake: when building library make it shared
33fa0bfb7 Added debianization
b4b739c4b cmake: bump version to the latest release
5dd12b4a7 faster sanitize tests
60119e258 Makefile : generates *.o for faster processing
9517bd923 minor simplification lz4_readarch()
a60d11e46 update on decompression speed
00eedb37d minor refactor
3e579afa6 Slightly improved decompression speed Slightly decreased decompression stack usage in 64-bits mode
795a2cdb1 sligthly improved decompression speed
a3c6daba6 liblz4 : Moved XXH_NAMESPACE to CPPFLAGS
d86dc9167 Merge pull request #131 from Cyan4973/dev
dfabec2d1 Fixed a few minor visual analyzer warnings
4abc86e0b Increased stack memory considered normal by Visual static analyzer
db59c7d33 Merge pull request #130 from Cyan4973/dev
8901dcdc9 removed one malloc
1098d897a lz4io : removed one malloc
81f19264a lz4cli : add one malloc test
aa1350e13 lz4hc : added one malloc test
6fe48b918 changed datagen to remove one malloc
15603d956 Added static analyzer test in Visual
48feb909d Merge pull request #129 from Cyan4973/dev
57e14d4c1 renamed directory to versionsTest
14064beaf Updated NEWS
7273da38b minor example clarification
0cc38f9eb Updated lib readme
454d109c5 Fixed : fuzzer display messages
caba2cc0f staticAnalyze use same compilation flags as normal build
8e8dc17ca Ensured make test is not confused with directory test
192ee724a Added namespace ability to xxhash
c04df7e1b Changed : static library is no longer compiled with -fPIC by default (this option can still be added on the command line) See #53
cdef03389 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
992faef5a Merge pull request #124 from toddlipcon/master
3d9adf7af Merge pull request #122 from keszybz/master
02ff90ac9 Merge pull request #121 from keszybz/add-gitignore
91138150c Fix cmake build to use gnu flags on clang
2ce2e6008 Minor doc clarification on sparse mode support
b17d3220d Minor lib readme update
3bcf68ff3 Use lz4 binary to check output
386d216cd Add example which uses the frame api through the library
0e0c32dfb Make git ignore generated files
7a532eedd Minor update on source ports
8a7bcead9 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
1cec6cb03 Updated badges
7f3d82def Merge pull request #114 from lpsantil/djgpp
f0a0e4430 READM fixup
a3a479fc2 djgpp port now using the proposed contrib structure
00b6535b4 Reverted all `Makefile` changes
936f75b4d Renamed the djgpp README so that it gets converted to HTML by github.  Fixed breakage of `make install`.
81d6b5634 Makefile cleanups to make building with Andrew Wu's build-djgpp cross compilers (and everyone else) simpler.
faa0bce4e Fixup lib/Makefile so that is builds a library for djgpp using Andrew Wu's build-djgpp cross compilers
b2bbb2790 Fixup programs/{Makefile,lz4cli.c,lz4io.c} to build with Andrew Wu's build-djgpp cross compilers.  Add some documentation in README.DJ

git-subtree-dir: src/lz4
git-subtree-split: c10863b98e1503af90616ae99725ecd120265dfb
2017-12-18 14:38:00 -05:00
Brad Chase
7d6c2229ab Update snappy support file 2017-12-18 14:37:42 -05:00
Brad Chase
516a1b85d1 Update snappy to 1.1.7 2017-12-18 13:41:54 -05:00
Brad Chase
2836c20649 Squashed 'src/snappy/snappy/' changes from 1ff9be9b8..b02bfa754
b02bfa754 Tag open source release 1.1.7.
824e6718b Add a loop alignment directive to work around a performance regression.
55924d110 Add GNUInstallDirs to CMake configuration.
632cd0f12 Use 64-bit optimized code path for ARM64.
77c12adc1 Add unistd.h checks back to the CMake build.
c8049c582 Replace getpagesize() with sysconf(_SC_PAGESIZE).
18e2f220d Add guidelines for opensource contributions.
f0d3237c3 Use _BitScanForward and _BitScanReverse on MSVC.
71b8f8688 Add SNAPPY_ prefix to PREDICT_{TRUE,FALSE} macros.
be6dc3db8 Redo CMake configuration.
e4de6ce08 Small improvements to open source CI configuration.
c756f7f5d Support both static and shared library CMake builds.
038a3329b Inline DISALLOW_COPY_AND_ASSIGN.
a8b239c3d snappy: Remove autoconf build configuration.
27671c6ae Clean up CMake header and type checks.
548501c98 zippy: Re-release snappy 1.1.5 as 1.1.6.
513df5fb5 Tag open source release 1.1.5.
5bc9c82ae Set minimum CMake version to 3.1.
e9720a001 Update Travis CI config, add AppVeyor for Windows CI coverage.
f24f9d2d9 Explicitly copy internal::wordmask to the stack array to work around a compiler optimization with LLVM that converts const stack arrays to global arrays.  This is a temporary change and should be reverted when https://reviews.llvm.org/D30759 is fixed.
82deffcde Remove benchmarking support for fastlz.
18488d621 Use 64 bit little endian on ppc64le.
7b9532b87 Improve the SSE2 macro check on Windows.
7dadceea5 Check for the existence of sys/uio.h in autoconf build.
83179dd8b Remove quicklz and lzf support in benchmarks.
c8131680d Provide a CMakeLists.txt.
ed3b7b242 Clean up unused function warnings in snappy.
8b60aac4f Remove "using namespace std;" from zippy-stubs-internal.h.
7d7a8ec80 Add Travis CI configuration to snappy and fix the make build.
1cd3ab02e Rename README to README.md. It already in markdown, we might as well let github know so that it renders nicely.
597fa795d Delete UnalignedCopy64 from snappy-stubs since the version in snappy.cc is more robust and possibly faster (assuming the compiler knows how to best copy 8 bytes between locations in memory the fastest way possible - a rather safe bet).
039b3a7ac Add std:: prefix to STL non-type names.
3c706d223 Make UnalignedCopy64 not exhibit undefined behavior when src and dst overlap.
d3c6d20d0 Add compression size reporting hooks.
626e1b9fa Use #ifdef __SSE2__ for the emmintrin.h include, otherwise snappy.cc does not compile with -march=prescott.
2d99bd14d 1.1.4 release.
8bfb028b6 Improve zippy decompression speed.
818b58338 adds std:: to stl types (#061)
27c5d8652 Re-work fast path for handling copies in zippy decompression.
4a7409408 Speed up Zippy decompression in PIE mode by removing the penalty for global array access.
38a5ec5fc Re-work fast path that emits copies in zippy compression.
094c67de8 Speed up the EmitLiteral fast path, +1.62% for ZFlat benchmarks.
fce661fa8 Speed up zippy decompression by removing some zero-extensions.
e788e527d Avoid calling memset when resizing the buffer.
32d6d7d8a Merge pull request #6 from deviance/provide-pkg-config-data
971613510 Add #ifdef to guard against macro redefinition if this is included in another Google project that also defines this.
0000f997d Merge pull request #13 from huachaohuang/patch-1
d53de1879 Make heuristic match skipping more aggressive.
2b9152d9c Default to glibtoolize instead of libtoolize if it exists, and also make it customizable through the environment variable $LIBTOOLIZE.
0800b1e4c Work around an issue where some compilers interpret <:: as a trigraph. Also correct the namespace name.
e7d2818d1 Unbreak the open-source build for ARM due to missing ATTRIBUTE_PACKED declaration.
7525a1600 Fix an issue where the ByteSource path (used for parsing std::string) would incorrectly accept some invalid varints that the other path would not, causing potential CHECK-failures if the unit test were run with --write_uncompressed and a corrupted input file.
ef5598aa0 Make UNALIGNED_LOAD16/32 on ARMv7 go through an explicitly unaligned struct, to avoid the compiler coalescing multiple loads into a single load instruction (which only work for aligned accesses).
b8cd908a8 Allow to compile in nested packages.
96a2e340f Update URLs in the Snappy README to reflect the move to GitHub.
0852af760 Move the logic from ComputeTable into the unit test, which means it's run automatically together with the other tests, and also removes the stray function ComputeTable() (which was never referenced by anything else in the open-source version, causing compiler warnings for some) out of the core library.
d80342922 Fix signed-vs.-unsigned comparison warnings.
d2cb73b6a Provide pkg-config data
efb39e81b Release Snappy 1.1.3; getting the new Uncompress variant in a release is nice, and it's also good to finally get an official release out after the migration to GitHub.
eb66d8176 Initialized members of SnappyArrayWriter and SnappyDecompressionValidator. These members were almost surely initialized before use by other member functions, but Coverity was warning about this. Eliminating these warnings minimizes clutter in that report and the likelihood of overlooking a real bug.
b2312c4c2 Add support for Uncompress(source, sink). Various changes to allow Uncompress(source, sink) to get the same performance as the different variants of Uncompress to Cord/DataBuffer/String/FlatBuffer.
b2ad96006 Changes to eliminate compiler warnings on MSVC
e7a897e18 Fixed unit tests to compile under MSVC.
86eb8b152 Change a few branch annotations that profiling found to be wrong. Overall performance is neutral or slightly positive.
11ccdfb86 Sync with various Google-internal changes.
22acaf438 Change some internal path names.

git-subtree-dir: src/snappy/snappy
git-subtree-split: b02bfa754ebf27921d8da3bd2517eab445b84ff9
2017-12-18 13:41:54 -05:00
JoelKatz
80050c110f RocksDB support updates:
* Update unity build for RocksDB changes
* Log RocksDB options on startup
* Support RocksDB option strings
* Support full file bloom filters

You can now configure most RocksDB options with RocksDB's option
string scheme.

Set "filter_full" to 1 to make bloom filters for an
entire file rather than each block. More memory will be
needed during compaction but less memory will be needed
during fetching for large databases. Does nothing unless
bloom filters are enabled with "filter_bits".

Example:
options = max_compaction_bytes=64;max_bytes_for_level_multiplier=64
clock_cache_mb = 96
filter_bits = 10
filter_full = 1
2017-12-18 13:35:08 -05:00
Nikolaos D. Bougalis
d2fc4e3569 Set version to 0.80.2 2017-12-15 13:06:10 -05:00
Mark Travis
fbfb4bd74e Control transaction dispatch rate:
Do not dispatch a transaction received from a peer for
processing, if it has already been dispatched within the
past ten seconds.

Increase the number of transaction handlers that can be in
flight in the job queue and decrease the relative cost for
peers to share transaction and ledger data.

Additionally, make better use of resources by adjusting the
number of threads we initialize, by reverting commit
68b8ffdb63.
2017-12-14 16:04:04 -08:00
Brad Chase
8fff0fa6a9 Update RocksDB to 5.8.7 2017-12-02 13:44:49 -05:00
Brad Chase
d0dac38ea8 Squashed 'src/rocksdb2/' changes from 1fdd726a8..aead40417
aead40417 fix HISTORY.md typo
a0cdc3cec Bump version to 5.8.7
7513f6350 Fix IOError on WAL write doesn't propagate to write group follower
9e47084ce Bump version to 5.8.6
36074ba5d Enable cacheline_aligned_alloc() to allocate from jemalloc if enabled.
aa00523e0 Add -DPORTABLE=1 to MSVC CI build
cf2b98237 Bump version to 5.8.5
e8c9350f2 Blob DB: not using PinnableSlice move assignment
4907d2463 Bump version to 5.8.4
5d928c795 Blob DB: Fix race condition between flush and write
725bb9d66 Blob DB: Fix release build
b7367fe84 Bump version to 5.8.3
13b2a9b6f Blob DB: use compression in file header instead of global options
5dc70a15c Fix PinnableSlice move assignment
9019e9125 dynamically change current memtable size
7f1815c37 Bump version to 5.8.2
2584a18ef Blob DB: Fix BlobDBTest::SnapshotAndGarbageCollection asan failure
17f67b546 PinnableSlice move assignment
6fb56c582 Blob DB: Add compaction filter to remove expired blob index entries
f90ced92f Blob DB: fix snapshot handling
632f36dcd Blob DB: option to enable garbage collection
11bacd578 Blob DB: Fix flaky BlobDBTest::GCExpiredKeyWhileOverwriting test
f98efcb1e Blob DB: Evict oldest blob file when close to blob db size limit
c1e99eddc Blob DB: cleanup unused options
ffc3c62ca Blob DB: Initialize all fields in Blob Header, Footer and Record structs
9e8254090 Blob DB: update blob file format
d66bb21e1 Blob DB: Inline small values in base DB
05d5c575a Return write error on reaching blob dir size limit
2b8893b9e Blob DB: Store blob index as kTypeBlobIndex in base db
419b93c56 Blob DB: not writing sequence number as blob record footer
8afb0036c fix lite build
dded348dd Blob DB: Move BlobFile definition to a separate file
374736123 add GetLiveFiles and GetLiveFilesMetaData for BlobDB
8cff6e945 Enable WAL for blob index
c29347290 Add ValueType::kTypeBlobIndex
eae53de3b Make it explicit blob db doesn't support CF
65aec19df Fix memory leak on blob db open
30b38c98c TableProperty::oldest_key_time defaults to 0
2879f4beb Bump version to 5.8.1
88595c882 Add DB::Properties::kEstimateOldestKeyTime
266ac245a Bumping version to 5.8
64185c23a update HISTORY.md for DeleteRange bug fix
e83d6a02e Not using aligned_alloc with gcc4 + asan
0980dc6c9 Fix wrong smallest key of delete range tombstones
b76797231 avoid use-after-move error
c41744270 CMake: Fix formatting
c21ea8f7a CMake: Add support for CMake packages
544434558 add Erlang to the list of language bindings
2972a702a Minor updates to FlushWAL blog
fbfa3e7a4 WriteAtPrepare: Efficient read from snapshot list
b01f426f5 Blog post for FlushWAL
503db684f make blob file close synchronous
3c840d1a6 Allow DB reopen with reduced options.num_levels
92bfd6c50 Fix DropColumnFamily data race
7fdf735d5 Pinnableslice examples and blog post
7fbb9ecca support disabling checksum in block-based table
19cc66dc4 fix clang bug in block-based table reader
7eba54eb9 test compaction input-level split range tombstone assumption
cd26af347 Add unit test for WritePrepared skeleton
a12479819 Improved transactions support in C API
c10b39131 LANGUAGE-BINDINGS.md: add another rust binding
90177432e Remove leftover references to phutil_module_cache
234f33a3f allow nullptr Slice only as sentinel
ccf7f833e Use PinnableSlice in Transactions
1dfcdb15f Extend pin_l0 to filter partitions
39ef90055 stop calling memcmp with nullptrs
78cb6b611 Provide byte[] version of SstFileWriter.merge to reduce GC Stall
867fe92e5 Scale histogram bucket size by constant factor
f004307e9 CMake improvements
09ac6206a Circumvent ASAN false positive
5b68b114f Blob db create a snapshot before every read
4624ae52c GC the oldest file when out of space
8ace1f79b add counter for deletion dropping optimization
0d8e992b4 Revert the mistake in version update
5358a8056 add VerifyChecksum to HISTORY.md
ed0a4c93e perf_context measure user bytes read
1efc600dd Preload l0 index partitions
bddd5d363 Added mechanism to track deadlock chain
c1384a707 fix db_stress uint64_t to int32 cast
29877ec7b Fix blob db crash during calculating write amp
8f2598ac9 Enable Cassandra merge operator to be called with a single merge operand
9a44b4c32 Allow merge operator to be called even with a single operand
ac8fb77af fix some misspellings
23593171c minor improvements to db_stress
af012c0f8 fix deleterange with memtable prefix bloom
1c8dbe2aa update scores after picking universal compaction
eb6425303 Update WritePrepared with the pseudo code
132306fbf Remove PartialMerge implementation from Cassandra merge operator
71598cdc7 Fix false removal of tombstone issue in FIFO and kCompactionStyleNone
3204a4f64 Fix missing stdlib include required for abort()
7aa96db7a db_stress rolling active window
dfa6c23c4 Update RocksDBCommonHelper to use escapeshellarg
e367774d1 Overload new[] to properly align LRUCacheShard
ad42d2fcb Remove residual arcanist_util directory
279296f4d properly set C[XX]FLAGS during CMake configure-time checks
c5f0c6cc6 compile with correct flags to determine SSE4.2 support
185ade4c0 cmake: support more compression type
5449c0990 rocksdb: make buildable on aarch64
a144a9782 Fix for CMakeLists.txt on Windows for RocksJava
acf935e40 fix deletion dropping in intra-L0
8254e9b57 make sst_dump compression size command consistent
74f18c130 db_bench support for non-uniform column family ops
5de98f2d5 approximate histogram stats to save cpu
3f5888430 Fix c_test ASAN failure
e5a1b727c Fix blob DB transaction usage while GC
6f051e0c7 fix corruption_test valgrind
ac098a462 expose set_skip_stats_update_on_db_open to C bindings
666a005f9 Support prefetch last 512KB with direct I/O in block based file reader
ad77ee0ea Revert "Makefile: correct faligned-new test"
b87ee6f77 Use more keys per lock in daily TSAN crash test
25df24254 Add column families related functions (C API)
64f848435 block_cache_tier: fix gcc-7 warnings
0cecf8155 Write batch for `TransactionDB` in C API
6a9de4347 Windows.h macro call fix
23c7d1354 fix comment
1fbad84b6 Makefile: correct faligned-new test
7848f0b24 add VerifyChecksum() to db.h
47ed3bfc3 fix WinEnv assertions
d97a72d63 Try to repair db with wal_dir option, avoid leak some WAL files
36375de76 gcc-7/i386: markup intentional fallthroughs
bdc056f8a Refactor PessimisticTransaction
a9a4e89c3 Fix valgrind complaint about initialization
4ca11b4b0 Update USERS.md
c9804e007 Refactor TransactionDBImpl
20dc5e74f Optimize range-delete aggregator call in merge helper.
0d4a2b733 Avoid blob db call Sync() while writing
627c9f1ab Don't add -ljemalloc when DISABLE_JEMALLOC is set
dce6d5a83 db_bench background work thread pool size arguments
4f81ab38b Makefile: fix for GCC 7+ and clang 4+
92afe830f Update all blob db TTL and timestamps to uint64_t
5883a1ae2 Fix /bin/bash shebangs
cc01985db Introduce bottom-pri thread pool for large universal compactions
0b814ba92 Allow concurrent writes to blob db
2c45ada4c Blob DB garbage collection should keep keys with newer version
58410aee4 Fix the overflow bug in AwaitState
c3d5c4d38 Refactor TransactionImpl
060ccd4f8 support multiple CFs with OPTIONS file
345387067 Fix statistics in RocksJava sample
1900771bd Dump Blob DB options to info log
3218edc57 Fix universal compaction bug
6a36b3a7b fix db get/write stats
a84cee812 Add a missing "once" in .h
21696ba50 Replace dynamic_cast<>
e85f2c64c Prevent empty memtables from using a lot of memory
ac748c57e Fix FIFO Compaction with TTL tests
aaf42fe77 Move blob_db/ttl_extractor.h into blob_db/blob_db.h
aace46516 Fix license headers in Cassandra related files
50a969131 CacheActivityLogger, component to log cache activity into a file
6083bc79f Blob DB TTL extractor
710411aea fix asan/valgrind for TableCache cleanup
3a3fb00b7 TARGETS file not setting sse explicitly
fca4d6da1 Build fewer tests in Travis platform_dependent tests
8f553d3c5 remove unnecessary internal_comparator param in newIterator
7f6d012d7 "ccache -C" in Travis
d12691b86 move TableCache::EraseHandle outside of db mutex
f33f11368 fix db_bench argument type
e7697b8ce Fix LITE unit tests
3ce20e985 Fix use of RocksDBCommonHelper in cont_integration.sh
c281b4482 Revert "CRC32 Power Optimization Changes"
9980de262 Fix FIFO compaction picker test
2289d3811 CRC32 Power Optimization Changes
30b58cf71 Remove the orphan assert on !need_log_sync
fe1a5559f Fix flaky write_callback_test
addbd279c 5.6.1 release blog post
30edff308 buckification: remove explicit `-msse*` compiler flags
2b259c9d4 Lower num of iterations in DeadlockCycle test
277f6f23d Release note for partitioned index/filters
5e731a138 Remove unused rocksdb arcanist lib
9b11d4345 Fix broken links
06f191744 add vcpkg as an windows option
ea8ad4f67 Fix compaction div by zero logging
34112aeff Added db paths to c
1d8aa2961 Gcc 7 ParsedInternalKey replace memset with clear function.
a4c42e800 Fix UBSAN issue of passing nullptr to memcmp
16e038820 LRUCacheShard cache line size alignment
216644c61 enable UBSAN macro in TARGETS
e67b35c07 Add Iterator::Refresh()
a34b2e388 Fix caching of compaction picker's next index
72502cf22 Revert "comment out unused parameters"
1d7048c59 comment out unused parameters
534c255c7 Cassandra compaction filter for purge expired columns and rows
63163a8c6 Remove make_new_version.sh
0302da47a Reduce blob db noisy logging
3e5ea29a8 Fix Flaky DeleteSchedulerTest::ImmediateDeleteOn25PercDBSize
a22b9cc6f overlapping endpoint fixes in level compaction picker
ffd2a2eef delete ExpandInputsToCleanCut failure log
3e6e863b1 Remove arcanist_util directory
36651d14e Moving static AdaptationContext to outside function
6e3ee015f Update java/rocksjni.pom
ecff9d5e3 Include write_buffer_manager in ImmutableDBOptions::Dump
ae28634e9 Remove some left-over BSD headers
33b1de82a Remove format compatibility hack
2f375154e checkout local branch in check_format_compatible.sh
ddb22ac59 avoid collision with master branch in check format
0c03a7f17 set the remote for git checkout
7ac184c6d Revert cmake -DNDEBUG for non-MSVC
0655b5858 enable PinnableSlice for RowCache
00464a314 Fix column_family_test with LITE build
b2dd192fe tools/write_stress.cc: Correct "1204" typos.
cbaab3044 table/block.h: change memset
f1a056e00 CodeMod: Prefer ADD_FAILURE() over EXPECT_TRUE(false), et cetera
4a2e4891f Add back the LevelDB license file
a7321fc97 Remove the licensing description in CONTRIBUTING.md
3c327ac2d Change RocksDB License
132013366 Make TARGETS file portable
ccf5f08f8 Set CACHE_LINE_SIZE for s390, PPC, ARM64
67510eeff db_crashtest.py: remove need for shell
4267eb00d Remove punit tests
5bfb67d90 Enable write rate limit for updaterandom benchmark
20a691d98 Update HISTORY to release 5.7
98d1a5510 db_bench to by default verify checksum
26ce69b19 Update blob db to use ROCKS_LOG_* macro
43e4eef77 remove unnecessary fadvise
21b17d768 Fix BlobDB::Get which only get out the value offset
70440f7a6 Add virtual func IsDeleteRangeSupported
7550255c5 Add JAVAC_ARGS as Makefile variable
7a0b5de77 Gcc 7 ignored quantifiers
000bf0af3 Improve the design and native object management of Stats in RocksJava
269d383d5 Bump version to 5.7
c32f27223 Fixes db_bench with blob db
fcd99d27c db_bench_tool: fix buffer size
87128bd5c fix regression test
8f927e5f7 Fix undefined behavior in Hash
643b787c7 Added a note about LZ4 compression dependency
56656e12d Temporarily disable FIFOCompactionWithTTLTest
b5fb85ec5 fix valgrind init complaint
657df29ea Add max_background_jobs to db_bench
a43c053ad remove duplicated utilities/merge_operators/cassandra/test_utils.cc in src.mk
7c4a9e6c9 Initialize a variable in ldb to make code analysis tool happy
98669b535 init filters_in_partition_
0013bf14e fix asan and valgrind leak report in test
521b4c28b rocksdb 5.5.1 release post
33042573d Fix GetCurrentTime() initialization for valgrind
f6b9d9355 Fix clang error in PartitionedFilterBlockBuilder
45b9bb033 Cut filter partition based on metadata_block_size
f4ae1bab0 update history for OnBackgroundError and DeleteRange fix
1cb8c6de6 Add -enable_pipelined_write to db_bench and add two defaults
7604b463b Update the AddDBStats in LITE
1e34d07e1 Simplify and document sync rules for logs_ etc
d310e0f33 Regression test for empty dedicated range deletion file
e9f91a517 Add a fetch_add variation to AddDBStats
c1b375e96 skip generating empty sst
67b417d62 fix format compatible test
afbef6518 Bug fix: Fast CRC Support printing is not honest
397ab1115 Improve Status message for block checksum mismatches
18c63af6e Make "make analyze" happy
01534db24 Fix the reported asan issues
1cd45cd1b FIFO Compaction with TTL
982cec22a Fix TARGETS file tests list
b49b37109 allow numa >= 2.0.8
e517bfa2c CLANG Tidy
dc3d2e4d2 update compatible test
89468c01d Fix Windows build broken by 5c97a7c0664d4071768113814e9ba71fe87e18cf
51778612c Encryption at rest support
7061912c2 Trivial typo in HISTORY.md
2a9cd8799 Fix jni WriteBatchThreadedTest
0025a3640 revert perf_context and io_stats to __thread
5c97a7c06 Unit Tests for sync, range sync and file close failures
4cee11f4e Intra-L0 blog post
857e9960b Improve the error message for I/O related errors.
d757355cb Fix bug that flush doesn't respond to fsync result
8e6345d2d Update rename of ParanoidCheck
499ebb3ab Optimize for serial commits in 2PC
0ac4afb97 Sanitize partitioning options
521724ba8 fixed wrong type for "allow_compaction" parameter
71f5bcb73 Introduce OnBackgroundError callback
88cd2d96e Downgrade option sanitiy check level for prefix_extractor
6837a1762 Fix Data Race Between CreateColumnFamily() and GetAggregatedIntProperty()
af1746751 WriteBufferManager will not trigger flush if much data is already being flushed
9467eb614 Fix flush assertion with tsan
048446fc7 Fix cassandra ASAN use-after-free
a21db161c Implement ReopenWritibaleFile on Windows and other fixes
c430d69ee fix coredump for release nullptr
0d278456c default implementation for InRange
cbd825dee Create a MergeOperator for Cassandra Row Value
2c98b06bf Remove pin_slice option by making it the default
c80c6115d add db_bench options for partitioning
6a3377f45 Synchronize statistic enumeration values between statistics.h and java API
53dda8797 Do not run RateLimiterTest.Rate test on Travis+Mac OSX.
ae8571f5c Fix blob db compression bug
7a380deff Update blob_db_test
89ad9f3ad Allow ignoring unknown options when loading options from a file
6b5a5dc5d fixed typo
0f228be3b fixed typo in util/dynamic_bloom.h
c217e0b9c Call RateLimiter for compaction reads
91e2aa3ce write exact sequence number for each put in write batch
6f4154d69 record index partition properties
5d5a28a98 Fix Clang release build broken by 5582123dee8426a5191dfd5e846cea8c676c793c
0175d58c3 Make direct I/O write use incremental buffer
7a270069b GNU C library for struct tm has 2 additional fields.
d713471da Limit trash directory to be 25% of total DB
9bb91e932 Dedup release
27b450165 Update HistogramTypes in the Java API
e97304c68 update history for 5.6
5582123de Sample number of reads per SST file
db818d2d1 Fix RocksDB Lite build with CLANG
a472c4ae4 update 5.5 change log
bc09c8a0d Fix crash in PosixWritableFile::Close() when fstat() fails
6d0f22e42 Fix mock_env.cc uninitialized variable
c2012d488 Java APIs for put, merge and delete in file ingestion
85dace2af Disable DBRangeDelTest::TailingIteratorRangeTombstoneUnsupported for ubsan
d4f7731b6 fix travis error with init time in mockenv
550a1df72 Fix clang errors by asserting the precondition
cc5f9339e Fix concurrency issue with filter_block_set_
2e64f450d bump version to 5.6
afbc2d0d2 Force travis to build with clang on MacOS
b172a3f1a Fix warnings while generating RocksJava documentation
52a7f38b1 WriteOptions.low_pri which can throttle low pri writes if needed
26a8a8071 Switch from CentOS 5 to CentOS 6 for crossbuilding RocksJava
dba9f3722 Fix db_write_test clang/windows build failure
c7662a44a fixed typo
7e8d95cc0 Fix the Java build which was broken by a4d9c02
7e5fac2c3 remove test dir before exit when current regression is running
7f6c02dda using ThreadLocalPtr to hide ROCKSDB_SUPPORT_THREAD_LOCAL from public…
138b87eae Fix interaction between CompactionFilter::Decision::kRemoveAndSkipUnt…
95b0e89b5 Improve write buffer manager (and allow the size to be tracked in block cache)
a4d9c0251 Pass CF ID to MemTableRepFactory
f68d88be5 Fix DBWriteTest::ReturnSequenceNumberMultiThreaded data race
215076ef0 Fix TSAN: avoid arena mode with range deletions
3a8a848a5 account for L0 size in estimated compaction bytes
0fae3f5dd codemod: format TARGETS with buildifier [5/5] (D5092623)
872199606 add checkpoint support for single db in regression test
5a9b4d743 Retire memenv https://github.com/facebook/rocksdb/pull/2082
d6019651b sync internal/external TARGETS
bbaba51bb Add missing index type to C-API
292edfd51 travis: test with xcode8.3 (OS X 10.12)
0dc3040d5 db: avoid `#include`ing malloc and jemalloc simultaneously
9b3ed8350 fix regression test
9c9909bf7 Support ingest file when range deletions exist
ad19eb868 Fixing blob db sequence number handling
51ac91f58 Histogram of number of merge operands
345878a7f update blob_db_test
cbc821c25 change regression rebuild to one level
103d0692e Avoid unsupported attributes when not building with UBSAN
5fd04566c travis: reduce the number of travis builders
2d05002b9 RocksDB 5.4.5 release blog post
7eca90f18 Update RocksDB blog authors
d03c34497 update comment of GetNextFile
f7bb1a006 support merge and delete in file ingestion
c2c62ad4e Reorder variables of ReadOptions
7bb1f5d48 Increase of compaction threads should be logged at info level instead of a warning
6c456ecae Clean zstd files
347e16f83 codemod: replace `headers = AutoHeaders.*` with `auto_headers`
0be636bf7 Fix db_bench build break with blob db
135ee6a3f fix tsan crash data race
a99fb9928 fix column_family_test asan
f41bffb3d travis: clang-3.6 -> clang-4.0
e7612798b update buckifer/TARGETS
bb01c1880 Introduce max_background_jobs mutable option
5a2530462 Fix the CMakeLists for RocksJava
41cbb7274 options.delayed_write_rate use the rate of rate_limiter by default.
506803466 range sync should be enabled
02594b5f1 Fix build errors in blob_dump_tool with GCC 4.8
52d9e5f7b Fix column family seconds_up accounting
7d8207f1f Fix errors in clang-analyzer builds
85b8569ae Fix release build on Linux
69ec8356b Allow SstFileWriter to use the rate limiter
6cc9aef16 New API for background work in single thread pool
9d0a07ed5 Fix rocksdb.estimate-num-keys DB property underflow
578fb0b1d Simple blob file dumper
ac39d6bec Core-local statistics
3e86c0f07 disable direct reads for log and manifest and add direct io to tests
15ba4d6c4 Address MS Visual Studio 2017 issue with autovector
88c818e43 Replace deprecated RocksDB#addFile with RocksDB#ingestExternalFile
228f49d20 Fix data races caught by tsan
4c9d2b104 remove #include port/port.h in public header file
07bdcb91f New WriteImpl to pipeline WAL/memtable write
d746aead1 Suppress clang-analyzer false positive
217b866f4 column_family_test: EnvCounter::num_new_writable_file_ to be atomic
9f839a7f6 keep util/build_version.cc when make clean
7eecd40a4 add emacs tags file - etags
9bbba4fec Remoe unused BlockBasedTable::compaction_optimized_
f5ba131bf Fixed some spelling mistakes
146b7718f Fix mingw compilation with -DNDEBUG
a36220ccf fix unity test
0ebdd7057 fixed typo
8032f4cb3 Remove -pie in TSAN
362ba9b02 Release RocksDB 5.5.0
ba685a472 Support ingest_behind for IngestExternalFile
01ab7b528 Add ROCKSDB_LIBRARY_API macro to a few C APIs, to fix windows build
cb9392a09 add Transactions and Checkpoint to C API
445f1235b s/std::snprintf/snprintf
cd593c283 Fix travis java_test
f720796e2 fixed typo
a48a62d5b define NDEBUG in CMake non-debug builds
1ca723dbd C API: support pinnable get
2ef15b85d Core-local stats blog post
4f9e69ccf fix log err
11c5d4741 cross-platform compatibility improvements
d00433302 Put lib files into suitable path in RPM package
86d549253 Fix build error with blob DB.
254c46800 Fix the RocksJava Release on Windows
7a47b431f Fix .gitignore pattern
fa5a15ceb Make sure that zstd is statically linked correctly in the Java static build
3fa9a39c6 Add GetAllKeyVersions API
1a60982a5 Simplified instructions for CentOS
a5cc7ecec Facility for cross-building RocksJava using Docker
ccd3dddf5 Blog post for partitioned index/filters
b145c34d7 Update blog authors
bbe9ee7dd core-local array type conversions
c2be43430 Build and link with ZStd when creating the static RocksJava build
c61e72c42 Add missing files of blob_db to CMake file
3907c94ff Fix ColumnFamilyTest:BulkAddDrop
cda5fde2d CoreLocalArray class
93949667c update TARGETS
4e83b8001 title: Bulkoading -> title: Bulkloading
d85ff4953 Blob storage pr
492fc49a8 fix readampbitmap tests
be421b0b1 portable sched_getcpu calls
0f559abdb Add NO_UPDATE_BUILD_VERSION option to makefile
3a04a254a Flink state
35df23fe8 Fix suite exclisions
e7cea86f7 Fixes the CentOS 5 cross-building of RocksJava
259a00eac unbiase readamp bitmap
a62096696 port: updated PhysicalCoreID()
df035b682 Print compaction_options_universal.stop_style in LOG file
4897eb250 dont skip IO for filter blocks
3f73d54bb Add C API to set max_file_opening_threads option
0b69e5079 Define CACHE_LINE_SIZE only when it's not defined
2cd00773c Add bulk create/drop column family API
40af2381e Object lifetime in cache
fdaefa030 travis: add Windows cross-compilation
a30a69603 do not read next datablock if upperbound is reached
2d42cf5ea Roundup read bytes in ReadaheadRandomAccessFile
264d3f540 Allow IntraL0 compaction in FIFO Compaction
8c3a180e8 Set lower-bound on dynamic level sizes
7c1c8ce5a Avoid calling fallocate with UINT64_MAX
a45e98a5b max_open_files dynamic set, follow up
6b99dbe04 fix memory alignment with logical sector size
e7ae4a3a0 Max open files mutable
60847a3b0 port: updated PhysicalCoreID()
b551104e0 support PopSavePoint for WriteBatch
498693cf3 Remove orphaned Java classes
5e2ebf2bd travis: add CMake compilation
af6fe69e4 Fix an issue of manual / auto compaction data race
6798d1f3b Revert "Delete filter before closing the table"
89833577a Delete filter before closing the table
47a09b0a8 Avoid pinning when row cache is accessed
aeaba07b2 Remove an assert that causes TSAN failure.
0b90aa951 Set VALGRIND_VER
a2b05210e Make PosixLogger::flush_pending_ atomic
da4b2070b Fix WriteBatchWithIndex address use after scope error
d616ebea2 Add GPLv2 as an alternative license.
4439b4596 Add documentation license
0ca3ead0c add GetRootDB() in DeleteFilesInRange
cdad04b05 Remove double buffering on RandomRead on Windows.
e15382c09 Disable two flaky tests
2150cc1f3 fix WritableFile buffer size in direct IO
efc361ef7 Add user stats Reset API
6616e4d62 add prefetch to PosixRandomAccessFile in buffered io
f6a27d0bc Extract statistics tests into separate file
7eddecce1 support bulk loading with universal compaction
3b4d1b7a4 add <sys/sysmacros.h> to avoid warning with glibc 2.25
e5e545a02 Reunite checkpoint and backup core logic
72c21fb3f call GetRootDB() before cast to DBImpl* in CancelAllBackgroundWork
4c9447d88 Add erase option to release cache
04d58970c AIX and Solaris Sparc Support
afff9951e Respect deprecated flag in table options
066cfbacc Adding -noprofile to CMakeLists for Windows
cb885bccf set compaction_iterator earliest_snapshot to max if no snapshot
7534ba7bd StackableDB should pass ResetStats()
c1fbf91b2 Fixing Solaris Sparc crash due to cached TLS
963eeba48 Revert how check_format_compatible.sh checkout release branches.
97005dbd5 tools/check_format_compatible.sh to cover option file loading too
8f6196788 Add cpu usage to regression benchmarks (4th attempt)
df74b775e Limit backups opened
1dd776051 Change L0 compaction score using level size
927bbab25 Revert "Add cpu usage to regression benchmarks (3rd attempt)"
8e84a388e Re-add index_per_partition but as deprecated
1553659d6 Add more recent versions to tools/check_format_compatible.sh
966ebb02f Hide event listeners from lite build
476e80be8 Add cpu usage to regression benchmarks (3rd attempt)
c49d70465 Add DB:ResetStats()
0fcdccc33 Blob storage helper methods
a6439d797 CMake: compile with -O2
e67f0adf3 enable O2 optimization for lz4
bc3973259 CMake: add support for SSE4.2
7d5f5aa97 Separate compile and link for shared library
071652734 remove warning
6e8d6f429 readahead backwards from sst end
ca96654d8 Change Build Env to gcc-5
e9e6e5324 Simplify write thread logic
6799c7e00 Pass in remote as a param to branch creation script
44fa8ece9 change use_direct_writes to use_direct_io_for_flush_and_compaction
13b50358f add space for buggy kernel warning
b6b9359ec Fix BYTES_WRITTEN accounting
13369fbd4 Update ShipIt to honor TARGETS updates
f2449ce92 Remove .deprecated_arcconfig
415be221c RocksDB Release 5.4 : Update HISTORY.md and build version.
3eab41d7c java dependencies test -s -> use test -d
a22ed4eab internal_repo_rocksdb to build Java and RocksDB LITE
9f2cc59ec sync TARGETS file
10d754696 set readahead buffer size from roundup(user_size) + 4k to roundup(use…
ba7da434a fix db_stress crash caused by buggy kernel warning
6257837d8 Add ROCKSDB_JAVA_NO_COMPRESSION flag
6a6723ee1 Move MergeOperatorPinning tests to be with other merge operator tests
6a8d5c015 Revert "Report cpu usage using time command"
8f47a9751 File level histogram should be printed per CF, not per DB
9300ef545 Fix shared lock upgrades
1f8b119ed Limit maximum memory used in the WriteBatch representation
97ec8a134 Report cpu usage using time command
20778f2f9 Adding comments to the write path
7124268a0 Reduce the number of params needed to construct DBIter
04abb2b2d FreeBSD only requires WITH_JEMALLOC, not the rest
61730186d dummy diff
360e9960f Summary: Remove .arcconfig
69a5e6461 Deprecate .arcconfig
9690653db Add a verify phase to benchmarks
dd8f9e38e Fix compilation for GCC-5
c2954f9b6 Add Travis job to build examples
72fc1e9d0 avoid non-existent O_DIRECT on OpenBSD
ff9728701 Refactor compaction picker code
9e7293902 only FALLOC_FL_PUNCH_HOLE when ftruncate is buggy
343b59d6e Move various string utility functions into string_util
1d068f606 Fix CompactRange incorrect buffer release
df6f5a377 Move memtable related files into memtable directory
107c5f6a6 CMake: more MinGW fixes
d2dce5611 Move some files under util/ to separate dirs
c50e3750d Use a human readable size for level report
ce64b8b71 Divide db/db_impl.cc
02799ad77 Revert "delete fallocate with punch_hole"
e2a7b202c Release note for partition filters
af256eb2b build db every monday
e5a1372b2 Rework test running script.
d659faad5 Level-based L0->L0 compaction
a12306fab Add a notice on gflags installation in INSTALL.md
43010a929 Revert "[rocksdb][PR] CMake: more MinGW fixes"
a30b75cdc Add buckifier script to github repo
3450ac8c1 CMake: more MinGW fixes
90cfd4645 update IterKey that can get user key and internal key explicitly
e2c6c0636 add TimedEnv
9e4453180 Refactor WriteImpl (pipeline write part 1)
6ef8c620d Move auto_roll_logger and filename out of db/
a1c469d71 Add release notes for PinnableSlice
0537f515c fix run_remote with strong quoting
72e600094 fixed misses on Centos library installation instructions
88cc81df5 auto_roll_logger_test to move away from real sleep
d25e28d58 replace sometimes-undefined uint type with unsigned int
a1d7e487b Add L0 write-amp to compaction level stats
b6d609063 CMake: support AVX2 in MinGW
bd7d13835 test remote instead run remote in regression test
c81a805fe test db existence on the remote host
5fc1e6765 add -rf when remove db in regression test
4ab4049f2 gflags has moved to GitHub
4e0065015 make all DB::Get overloads virtual
6401a8b76 Fix build with MinGW
80fe5b385 disable test: DeleteSchedulerTest.DynamicRateLimiting1
a9c86f51b backup garbage collect shared_checksum tmp files
da175f7ec exit with code 2 when there is already a db_bench running in regression test
0ee7f0403 Added missing options to RocksJava
c6d04f2ec Option to fail a request as incomplete when skipping too many internal keys
58179ec4a Cleanup of ThreadStatusUtil structures should use the DB's reference
f3607640a add ldb build to regression test
8a8c96746 Enable Fast CRC32 for Win64
f9813b853 Added SstFileWriter construtor without explicit comparator to JNI api
8d3cb4f20 Added naming of backup engine threads
67d762379 Expose the stalling information through DB::GetProperty()
0fd574926 delete fallocate with punch_hole
41fe9ad75 Hide usage of compaction_options_fifo from lite build
e7731d119 Configure index partition size
69c8d524a Fix jni library name for PowerPC Architecture
34a70859b Fix segmentation fault caused by #1961
8dee8cad9 Enable fifo compaction benchmark to db_bench
a5c8b5434 fix a header include
91b5feb37 Fix Windows Build broken by a recent commit
41ccae6d2 Add C API functions (and tests) for WriteBatchWithIndex
88bb6f6bf non_shm CI should run tests on /tmp
8888de2b1 Update .gitignore file in examples
203136e79 Fix Compilation errors when using IBM Java
f4fce4751 Fix clang compile error - [-Werror,-Wunused-lambda-capture]
a084b26a5 Blog post for releasing 5.2.1
15950fe3a Remove ASSERT_EQ(boolean, ...)
3e56c7e0c make total_log_size_ atomic
909028e21 HISTORY.md for log_size_for_flush in CreateCheckpoint()
be723a8d8 Optionally construct Post Processing Info map in MemTableInserter
e474df947 db_bench: not need to check mmap for PlainTable
8b0097b49 Readers for partition filter
9ef3627fd Allow checkpointing without flushing
17866ecc3 Allow Users to change customized ldb tools' header in help printing
a2a883318 remove deleted option from benchmark.sh
78cb19559 add checkpoint to ldb
4b04addfc updated solution if "make format" command fails
8f5bf0446 Flush triggered by DB write buffer size picks the oldest unflushed CF
6908e24b5 dynamic setting of stats_dump_period_sec through SetDBOption()
93c68b642 change regression bash file with debug mode
21d8c3179 remove LIKELY from public headers
36ad75778 INSTALL: document USE_SSE
9272e12f1 avoid ftruncate twice in buffered io
d52f334cb Break stalls when no bg work is happening
e66221add fix db_bench rate limiter callsites
dbae438a0 Replace "DEPRECATED" comment to "not supported"
995618a82 Support SstFileManager::SetDeleteRateBytesPerSecond()
e19163688 Add macros to include file name and line number during Logging
d525718a9 cleanup direct io flag in WritableFileWriter
5fa927aa9 Add Xpress and ZSTD CompressionType values to C header
11526252c Pinnableslice (2nd attempt)
e5bd8def1 update history.md for fixing the bug that skips keys
1ffbdfd9a Add a new SstFileWriter constructor without explicit comparator
ebd5639b6 Add ability to search for key prefix in sst_dump tool
e6725e8c8 Fix some bugs in MockEnv
900c62be6 fix compile for VS2015
fe1835617 release 5.3
5dae01947 Revert "Report cpu usage using time command"
f2817fb7f avoid ASSERT_EQ(false, ...);
5b11124e3 add max to histogram stats
d43adf21b Report cpu usage using time command
18fc1bc0e minor changes for rate limiter test flakiness
12ba00ea6 Reset DBIter::saved_key_ with proper user key anywhere before pass to DBIter::FindNextUserEntry
c9df05d1e Fix random access alignment
f64991537 Add Bulkoading IngestExternalFile blog post
54b434110 Builders for partition filter
97edc72d3 Add a memtable-only iterator
72202962f fix db_sst_test flakiness
5f65dc877 Expose DB::DeleteRange and WriteBath::DeleteRange in Java
58b12dfe3 Set logs as getting flushed before releasing lock, race condition fix
f8a4ea020 Move db_test and external_sst_file_test out of Travis's MAC OS run
534581a35 Fix a bug in tests in options operator=
a2f7a514d Refactoring
2a5daa06f Add stderr log level for ldb backup commands
4561275c2 fix rate limiter test flakiness
7c80a6d7d Statistic for how often rate limiter is drained
0ad5af42d Clarify VerifyBackup behavior
6fb901344 sanitize readahead when direct read enabled
f89b3893c Remove skip_table_builder_flush and default it to true
cc253982d Use more default options in db_bench
8432bcf55 Make compaction_pri settable through option string
d5b607a43 Make db_wal_test slightly faster
ba4c77bd6 Divide external_sst_file_test
e877afa08 Remove bulk loading and auto_roll_logger in rocksdb_lite
90d835507 Fix the wrong address for PREFETCH in DynamicBloom::Prefetch
08864df21 Move advanced column family options to advanced_options.h
2ca2059f6 Get unique_ptr to use delete[] for char[] in DumpMallocStats
253799c06 Add missing include for `abort()`
c6d464a9d Fixed various memory leaks and Java 8 JNI Compatibility
be3e5568b Fix unaligned reads in read cache
8ad0fcdf9 Separate small subset tests in DBTest
6c951c43c Run fewer tests in OSX
f7997f134 add direct I/O to version notes 5.2.0
3b8ba703c Fix flaky DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest2
96c7e1504 Fix Java build
e0b87afc7 Black list some slow valgrind tests
e67232cff Handle failed Finish() in SST file writer
8efb5ffa2 [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp…
1ba2804b7 Remove XFunc tests
e7d902e69 add direct_io and compaction_readahead_size in db_stress
1ef5f50e8 detect logical sector size
ed50308d2 check backup directory exists before listing children
4d7c06ced Make WriteBatchWithIndex moveble
5040414e6 Gracefully handle previous backup interrupted
f206af56f add use_direct_io() to ReadaheadRandomAccessFile
082493442 truncate patch
286a36db7 posix writablefile truncate
f0879e4c3 Page size isn't always 4k on linux
18eeb7b90 Fix interference between max_total_wal_size and db_write_buffer_size checks
1560b2f5f Temporarly return deprecated functions to fix MongoRocks build
2a0f3d0de level compaction expansion
ebc8a7998 alignment is on in ReadaheadRandomAccessFile::Read()
381fd3224 Remove timeout_hint_us from WriteOptions
fce7a6e19 Fail IngestExternalFile when bg_error_ exists
a618a16f4 New subcode for IOError to detect the ESTALE errno
7ab005183 Remove deprecated DB::AddFile and DB::CompactRange
401667366 Adding Dlang to the list
756c5924e Allow adding external v1 sst file with no global seqno support
aa0298fa9 Remove trailing whitespace from examples Makefile
db2b4eb50 avoid direct io in rocksdb_lite
43e9f01c2 Fix repair_test on ROCKSDB_LITE
7106a994f Use monotonic time points in write_controller.cc and rate_limiter.cc
c2247dc1c Make DBImpl::has_unpersisted_data_ atomic
eb912a927 Remove disableDataSync option
0227c16d6 Update static library versions and add checksums
b3aae4d07 Add repair_test to make check
421ce7c2b Add support for JNI Library on Linux on PowerPC.
9afa20cf2 Increase build version and HISTORY.md for releasing 5.2
a5adda064 Fix repair issues
b48e4778b Consolidate file cutting logic in compaction loop
ac2a77a74 Announce the experimetnal two-level index feature in HISTORY.md
c4a37dcb4 Print the missed last layer in cfstats
a12818afc Blog post for 5.1.2 release
3b4ac8076 Clarify ldb column family argument
d70ce7ee0 Move db_bench flags out of unnamed namespace
186c7eedb Remove incorrect statistics warning
53bb01516 [rocksdb][PR] compaction_style and compaction_pri should output their value as a st…
69d5262c8 Two-level Indexes
0a4cdde50 Windows thread
1aaa898cf Adding GetApproximateMemTableStats method
9fc23c55f Use gcc-4.9-glibc-2.20-fb python in precommit_checker
b797e4215 Dump compression dictionary meta-block
036d668b1 Fix wrong result in data race case related to Get()
574b543f8 Rename merger.h -> merging_iterator.h
add8b50cc Move ThreadLocal implementation into .cc
71d2496af Fix arc setting for Facebook internal tools
f289d9f4a Fix OSX build break after the fallocate change
4a3e7d320 Change the default of delayed slowdown value to 16MB/s
0513e21f9 RangeSync() should work with ROCKSDB_FALLOCATE_PRESENT not set
8b369ae5b Cleaner default options using C++11 in-class init
ec79a7b53 Dedup code in option.cc and db_options.cc
b96372dea improving the C wrapper
04c4ec41d Change corruption_test to use 4 bits.
2d75cd40d NewLRUCache() to pick number of shard bits based on capacity if not given
f25f1ec60 Add test DBTest2.GetRaceFlush which can expose a data race bug
37d4a79e9 Deserialize custom Statistics object in db_bench
3b35134e4 Avoid cache lookups for range deletion meta-block
94a0c32e7 Fix LRU Ref() for handles with external references only
17c118060 Generalize Env registration framework
07dddd5f7 EnvPosixTestWithParam should wait for all threads to finish
5dad9d6d2 Avoid logs_ operation out of DB mutex
a7b13919b Fix CompactFiles() bug when used with CompactionFilter using SuperVersion
616a1464e Fix DeleteRange including sentinels in output files
c918c4b76 Update USERS.md add user Pika
03ca2ac8a Remove function from DBImpl that are not used anywhere
b0029bc7f Test merge op covered by range deletion in memtable
d438e1ec1 Test range deletion block outlives table reader
fba726e55 Version librocksdb.so
9da4d542f Range deletions unsupported in tailing iterator
f2b4939da fixed typo
973f1b78f memtable: delete merge value for range deleteion
aebfd1703 fix non-portable behavior in encoder
753ff84a3 Fix get approx size
d7ea44f2f Fixup a couple of builds errors on Linux.
537da370d c: allow set savepoint to writebatch
af6ec4d78 fix batchresult handle leak
e29bb934f Zlib 1.2.8 is no longer available, switched to 1.2.10
5ac97314e Fix std::out_of_range when DBOptions::keep_log_file_num is zero
4e35ffdfa cmake: check -momit-leaf-frame-pointer before using it
3c0852d1d Make ingest external file backward compatible
0e8dfd606 Fix OptimizeForPointLookup()
e840213d6 Change DB::GetApproximateSizes for more flexibility needed for MyRocks
9239103cd Flush job should release reference current version if sync log failed
da54d36a9 Disable IngestExternalFile in ReadOnly mode
5cf176ca1 Fix for 2PC causing WAL to grow too large
4a73bb0b4 Split travis jobs
c70d3c7ad Enable DBTest.GroupCommit as it runs in a reasonlable time now.
602c13a96 Remove fadvise with direct IO read
f9d18e22d Fix DeleteRange file boundary correctness issue with max_compaction_bytes
3ce091fd7 Add KEEP_DB env var option
77b480662 Fix 2PC with concurrent memtable insert
e8a096000 util/thread_local.h: silence a clang-build warning
324a0f988 Follow up for DirectIO refactor
bc5d7b702 travis: For linux, do all tests under gcc
3e6899d11 change UseDirectIO() to use_direct_io()
d4e07a845 fix warning of unused direct io helper functions
dc2584eea direct reads refactor
d18dd2c41 Abort compactions more reliably when closing DB
62384ebe9 Guarding extra fallocate call with TRAVIS because its not working pro…
9f246298e Performance: Iterate vector by reference
fe395fb63 Allow incrementing refcount on cache handles
2172b660e Fix build on FreeBSD
3c233ca4e Fix Windows environment issues
763173456 Fix the error in ColumnFamiliesTest
7a02ad070 Update travis to ubuntu trusty
60c509ff1 Fix valgrind failure in test CurrentFileModifiedWhileCheckpointing2PC
d0ba8ec8f Revert "PinnableSlice"
54d94e9c2 PinnableSlice
e04480fae Fix MS warnings. Use ROCKSDB_Prsz for size_t.
c081f7215 5.0.1 release blog post
ac73d7558 Add GetSupportedCompressions() convenience function
b104b8781 Maintain position in range deletions map
640d72480 Update db_bench and sst_dump to test with block cache mid-point inser?
653ac1f9c C API: support total_order_mode
85ac1a320 Fix rocksdb::Status::getState
76711b6e7 Make ExternalSSTFileTest::CompactionDeadlock more deterministic
c963460db Fix tests under GCC_481
33c86d677 Fix backupable db test
e425ec116 utilities/backupable: backup should limit the copy size of wal.
0712d541d Delegate Cleanables
d58ef52ba Allow SstFileWriter to Fadvise the file away from page cache
17a4b75cc Always fsync the file after file copying
2fb70dc79 examples: Add options_file_example to target all
a738af8f8 db/pinned_iterators_manager.h: bugfix
906523d98 Add description to the 2PC checkpooint bug in HISTORY.md
438f22bc5 Fix bug of Checkpoint loses recent transactions with 2PC
335981d47 Fix the directory path for RocksDB repo
548b62805 Enable conditionally using adaptive mutexes
4e07b08ef include/rocksdb/utilities/env_librados: fix typo
ab48c165a Print cache options to info log
972f96b3f direct io write support
989e644ed Remove sst_file_manager option from LITE
1beef6569 Fix c_test
3d692822f persistent_cache: fix two timer
046099c9b The array is malloced by backtrace_symbols(), and must be freed
6ff2c8d7f Remove gflags as travis build dependency
3cd9ed1c3 Show sandcastle URL in phabricator
50e305de9 Collapse range deletions
5d1457dbb Dump persistent cache options
7bd725e96 db_bench: introduce --benchmark_read_rate_limit
296691847 Update Netflix section of USERS.md
342370f1d Simplify MemTable::Update
1a136c1f1 Expose file size
fbff4628a Reduce compaction iterator status checks
bd6cf7b51 WritableFileWriter: default buffer size equal min(64k,options.writabl?
fc0c6fd98 "make format" format diff since last commit from master
816c1e30c gcc-7 requires include <functional> for std::function
c27073586 Iterator should be in corrupted status if merge operator return false
a8bf4d63f Make format shows wrong curl command for retrieving clang-format-diff.py
8f5d24ae6 C API: support get usage and pinned_usage for cache
0ab6fc167 Gcc-7 buffer size insufficient
b7239bf7e Gcc 7 fallthrough
477b6ea57 std::remove_if requires <algorithm>
83f9a6fd2 Fail BackupEngine::Open upon meta-file read error
a79eae4b0 Add pcache documentation images
d71e728c7 Print user collected properties in sst_dump
7004a6f7b Add missing copyright header
3cdfaeca3 Fixes for MSVC compilation
e097222e6 util/logging.cc: buffer of insufficient size (gcc-7 -Werror=format-length)
cfc34d7c4 Missing break in case in DBTestBase::CurrentOptions
bfbcec233 Gcc 7 error expansion to defined
6653e32ac build: make it easier to pass PORTABLE
67adc937b intentional fallthough (prevents gcc-7/clang-4 error)
1a146f89c break Flush wait for dropped CF
c3e5ee715 util/histogram.cc: HistogramStat::toString buffer insufficient
5334d8b44 table/block_based_table_builder.cc: intentional fallthrough - comment to match gcc pattern
36d42e65d Disable test to unblock travis build
b57dd9262 C API: support writebatch delete range
2ba59b5a1 Disallow ingesting files into dropped CFs
1f6f7e3e8 cast to signed char in ldb_cmd_test for ppc64le
243975d5d More accurate error status for BackupEngine::Open
f0c509e2c Return finer-granularity status from Env::GetChildren*
dc64f46b1 Add db_bench option for stderr logging
2cabdb8f4 Increase buffer size
4a17b47bb Remove unnecessary header include
8c2b921fd Fixed a crash in debug build in flush_job.cc
20ce081fa Fix issue where IngestExternalFile insert blocks in block cache with g_seqno=0
5241e0dbf fix db_bench argument type
c04f6a0b4 Specify shell in makefile
45c7ce137 CompactRangeOptions C API
2c2ba6824 db_stress support for range deletions
b821984d3 DeleteRange read path end-to-end tests
2f4fc539c Compaction::IsTrivialMove relaxing
1dce75b2e Update USERS.md
304b3c706 Update USERS.md
fa50fffaf Option to expand range tombstones in db_bench
c26a4d8e8 Fix compile error in trasaction_lock_mgr.cc
ed8fbdb56 Add EventListener::OnExternalFileIngested() event
2005c88a7 Implement non-exclusive locks
0b0f23572 Mention IngestExternalFile changes in HISTORY.md
23db48e8d Update HISTORY.md for 5.0 branch
beb36d9c1 Fixed CompactionFilter::Decision::kRemoveAndSkipUntil
67f37cf19 Allow user to specify a CF for SST files generated by SstFileWriter
9053fe2a5 Made delete_obsolete_files_period_micros option dynamic
edde954e7 fix clang build
56281f3a9 Add memtable_insert_with_hint_prefix_size option to db_bench
4a21b1402 Cache heap::downheap() root comparison (optimize heap cmp call)
e39d08087 Fix travis (compile for clang < 3.9)
3f407b065 Kill flashcache code in RocksDB
b77007df8 Bug: paralle_group status updated in WriteThread::CompleteParallelWorker
247d0979a Support for range skips in compaction filter
96fcefbf1 c api: expose option for dynamic level size target
00197cff3 Add C API to set base_backgroud_compactions
5b219eccb deleterange end-to-end test improvements for lite/robustness
aad119176 pass rocksdb oncall to mysql_mtr_filter otherwise tasks get created w…
e33352899 DeleteRange write path end-to-end tests
7784980fc Fix mis-reporting of compaction read bytes to the base level
3c6b49ed6 Fix implicit conversion between int64_t to int
b3b875657 Remove unused assignment in db/db_iter.cc
4f6e89b1d Fix range deletion covering key in same SST file
a2bf265a3 Avoid intentional overflow in GetL0ThresholdSpeedupCompaction
52fd1ff2c disable UBSAN for functions with intentional -ve shift / overflow
1886c435b Fix CompactionJob::Install division by zero
63c30de80 fix options_test ubsan
13e66a8f5 Fix compaction_job.cc division by zero
01eabf737 Fix double-counted deletion stat
7ffb10fc1 DeleteRange compaction statistics
236d4c67e Less linear search in DBIter::Seek() when keys are overwritten a lot
cd7c4143d Improve Write Stalling System
dfb6fe675 Unified InlineSkipList::Insert algorithm with hinting
3068870cc Making persistent cache more resilient to filesystem failures
734e4acaf Eliminate redundant cache lookup with range deletion
182b940e7 Add WriteOptions.no_slowdown
4118e1333 Persistent Cache: Expose stats to user via public API
f2a8f92a1 rocks_lua_compaction_filter: add unused attribute to a variable
4444256ab Remove use of deprecated LZ4 function
548d7fb26 Fix fd leak when using direct IOs
fd43ee09d Range deletion microoptimizations
23a18ca5a Reword support a little bit to more clear and concise
481856ac4 Update support to separate code issues with general questions
a0deec960 Fix deadlock when calling getMergedHistogram
fe349db57 Remove Arena in RangeDelAggregator
e63350e72 Use more efficient hash map for deadlock detection
a13bde39e Skip ldb test in Travis
73843aa63 Direct I/O Reads Handle the last sector correctly.
9d60151b0 Implement PositionedAppend for PosixWritableFile
3f6221521 Lazily initialize RangeDelAggregator's map and pinning manager
41e77b839 cmake: s/STEQUAL/STREQUAL/
c1038d283 Release RocksDB 5.0
635a7bd1a refactor TableCache Get/NewIterator for single exit points
f39452e81 Fix heap use after free ASAN/Valgrind
a4eb7387b Allow plain table to store index on file with bloom filter disabled
36e4762ce Remove Ticker::SEQUENCE_NUMBER
86eb2b9ad Fix src.mk
0765babe1 Remove LATEST_BACKUP file
647eafdc2 Introduce Lua Extension: RocksLuaCompactionFilter
760ef68a6 fix deleterange asan issue
327085b7b fix valgrind
715591bba Ask travis to use JDK 7
972e3ff29 Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default
420bdb42e option_change_migration_test: force full compaction when needed
1543d5d92 Report memory usage by memtable insert hints map.
018bb2ebf DeleteRange support for db_bench
dc51bd716 CMakeLists.txt: FreeBSD has jemalloc as default malloc
48e8baebc Decouple data iterator and range deletion iterator in TableCache
4b0aa3c4c Fix failed compaction_filter_example and add it into make all
53b693f5f ldb support for range delete
661e4c926 DeleteRange unsupported in non-block-based tables
489d14280 DeleteRange interface
eba99c28e Fix min_write_buffer_number_to_merge = 0 bug
2ef92fea5 Remove all instances of relative_url until GitHub pages problem is fixed.
91300d01f Dynamic max_total_wal_size option
ec2f64794 Consider subcompaction boundaries when updating file boundaries for range deletion
800e51553 Fix CSS issues again
b952c898b Parallize persistent_cache_test and transaction_test
3b192f618 Handle full final subcompaction output file with range deletions
6c5795200 Make range deletion inclusive-exclusive
425210cc4 CSS issues are arising on the Github Pages side. Temp fix.
1ea79a78c Optimize sequential insert into memtable - Part 1: Interface
df5eeb85c Optimize sequential insert into memtable - Part 2: Implementation
5ed650857 Fix SstFileWriter destructor
adb665e0b Allowed delayed_write_rate option to be dynamically set.
307a4e80c sst_dump support for range deletion
361010d44 Exporting compaction stats in the form of a map
672300f47 Use relative Urls for stylesheets
b39b2ee12 do not call get() in recovery mode
1ca5f6d13 Fix 2PC Recovery SeqId Miscount
e095d0cbc Rocksdb contruns to new Sandcastle API
14c0380e7 Convenience option to parse an internal key on command line
c90fef88b fix open failure with empty wal
4e20c5da2 Store internal keys in TombstoneMap
a9fb346e4 Fix RocksDB Lite build failure in c_test.cc
d133b08f6 Use correct sequence number when creating memtable
144cdb8f1 16384 as e.g .value for compression_max_dict_bytes
9bd191d2f Fix deadlock between (WriterThread/Compaction/IngestExternalFile)
a9fae0a9d CSS problems again :(
193221e0a Fix Forward Iterator Seek()/SeekToFirst()
e48f3f8b9 remove tabs and duplicate #include in c api
85bd8f518 Minor fix to GFLAGS usage in persistent cache
a7875272d c: support seek_for_prev
0f17f9279 Make the header links a bit more flexible
cf19f559d single quotes in feed
2dc019e09 Fix header links
f1aedda06 More Jekyll 3.3 fixes
c54cdc378 More Jekyll 3.3 updates
2bcaf8246 Update product and feature template for Jekyll 3.3
24bceb096 Java API - Implement GetFromBatch and GetFromBatchAndDB in WBWI
815f54afa Insert range deletion meta-block into block cache
9e7cf3469 DeleteRange user iterator support
5c5d01ae7 Fix wrong comment (Maximum supported block size)
f998c9790 DeleteRange Get support
879f36636 Add C api for RateLimiter
557034f36 Remove all instances of baseurl
437942e48 Add avoid_flush_during_shutdown DB option
2b16d664c Change max_bytes_for_level_multiplier to double
16fb04434 expose IngestExternalFile to c abi
ce22ea99a Fix casts for MSVC
196af035c Introduce FAIL_ON_WARNINGS CMake variable (default ON)
40a2e406f DeleteRange flush support
d5555d95a Fix MSVC compile error in 32 bit compilation
da61f348d Print compression and Fast CRC support info as Header level
f9eb56791 db_bench: --dump_malloc_stats takes no effect
6a4faee5c fix freebsd build include path err and so & jar file name
c90c48d3c Show More DB Stats in info logs
1b295ac8a DBTest.GetThreadStatus: Wait for test results for longer
25f5742f0 Update documentation to point at gcc 4.8
b50a81a2b Add a test for tailing_iterator
04751d534 L0 compression should follow options.compression_per_level if not empty
2946cadc4 Improve RangeDelAggregator documentation
0a9fd05c2 Update Vagrant file (test internal phabricator workflow)
fcd1e0bf6 Make rocksdb work with internal repo
0aab5e55f FreeBSD: malloc_usable_size is in <malloc_np.h> (#1428)
9c0bb7f17 cmake: drop "-march=native" from CXX_FLAGS (#1429)
eeb27e1bb Add handy option to turn on direct I/O in db_bench (#1424)
c6168d13a removed some declarations from c.h which resulted in undefined symbols (#1407)
bc429de49 revert fractional cascading in farward iterator
b9bc7a2aa Use skiplist rep for range tombstone memtable
60a2bbba9 Makefile: generate util/build_version.cc from .in file (#1384)
9ee84067f Disable DBTest.RepeatedWritesToSameKey (#1420)
f41df3045 OptionChangeMigration() to support FIFO compaction
2e8004e60 Changing the legocastle run to use valgrind_test instead of _check
9de2f7521 revert Prev() in MergingIterator to use previous code in non-prefix-seek mode
24495186d DBSSTTest.RateLimitedDelete: not to use real clock
1168cb810 Fix a bug that may cause a deleted row to appear again
99c052a34 Fix integer overflow in GetL0ThresholdSpeedupCompaction (#1378)
f83cd64c0 Fix a bug that mistakenly disable regression_test.sh to update commit (#1415)
0e926b84f Passing DISABLE_JEMALLOC=1 to valgrind_check if run locally
4dfaa6610 Make IsDeadlockDetect() virtual member of Transaction
59a7c0337 Change ioptions to store user_comparator, fix bug
869ae5d78 Support IngestExternalFile (remove AddFile restrictions)
1d9dbef64 Restrict running condition of UniversalCompactionTrivialMoveTest2
4edd39fda Implement deadlock detection
48fd619a4 Minor fixes to RocksJava Native Library initialization (#1287)
48e4e842b Disable auto compactions in memory_test and re-enable the test (#1408)
fb2e41294 column_family_test: disable some tests in LITE
5af651db2 fix data race in compact_files_test
a0ba0aa87 Fix uninitialized variable gcc error for MyRocks
b88f8e87c Support SST files with Global sequence numbers [reland]
08616b493 [db_bench] add filldeterministic (Universal+level compaction)
52c9808c3 not split file in compaciton on level 0
5e0d6b4cc fix db_stress assertion failure
ab5399837 Bump RocksDB version to 4.13 (#1405)
b4d07123c SamePrefixTest.InDomainTest to clear the test directory before testing
aa09d0338 Avoid calling GetDBOptions() inside GetFromBatchAndDB()
6fbe96baf Compaction Support for Range Deletion
257de78d9 Remove "-Xcheck:jni" from Java tests (#1402)
d88dff4ef add seeforprev in history
5027dd17a Fix a minor bug in the ldb tool that was not selecting the specified (#1399)
fea6fdd67 Fix @see in two Java functions (#1396)
b1031d6c1 Remove function local statics that interfere with memory pooling (#1392)
f47054015 Handle WAL deletion when using avoid_flush_during_recovery
e29d3b67c Make max_background_compactions and base_background_compactions dynamic changeable
21e8daced fix assertion failure in Prev()
b9311aa65 Implement WinRandomRW file and improve code reuse (#1388)
a249a0b75 check_format_compatible.sh to use some branch which allows to run with GCC 4.8 (#1393)
040328a30 Remove an assertion for single-delete in MergeHelper::MergeUntil
8cbe3e10c Relax the acceptable bias RateLimiterTest::Rate test be 25%
f26a139d8 Log successful AddFile
5691a1d8a Fix compaction conflict with running compaction
017de666c fixup commit
1b7af5fb1 Redo handling of recycled logs in full purge
27bfe327b Editorial change to README.md
89cc404de A bit of doc restructuring
9e7fda829 Fix arcanist
2e4b5cab0 Add missing RateLimiter class to the Windows build (#1382)
ce4963fdf [doc] Document that Visual Studio 2015+ is now required for Windows builds (#1389)
e48927098 Fix scoped arena iterator (#1387)
f8d8cf53f Fix log_write_bench -bytes_per_sync option. (#1375)
02b3e3985 Make txn->GetState() const
447f17127 new Prev() prefix support using SeekForPrev()
991b585ee More block cache tickers
d6ae6dec6 Add Statistics::getAndResetTickerCount().
aea3ce4c8 Avoid string CONCAT which is not supported in cmake 2.6 (#1383)
2ad68b971 Support running consistency checks in release mode
67501cfc9 Fix -ve std::string::resize
04b02dd12 Testing asset links after config change
8c55bb87c Make Lock Info test multiple column families
d06232897 Revert "Support SST files with Global sequence numbers"
5cd28833a [RocksJava] Adjusted RateLimiter to 3.10.0 (#1368)
37737c3a6 Expose Transaction State Publicly
2c1f95291 Add facility to write only a portion of WriteBatch to WAL
043cb62d6 Fix record_size in log_write_bench, swap args to std::string::assign. (#1373)
4985f60fc env_mirror: fix a few leaks (#1363)
5aded67dd update of c.h (#1371)
912aec13c "Recent Posts" -> "All Posts"
7cbb298db Make sure that when contribtuing we call out creating appropriate directories
a06ad4711 Add top level doc information to CONTRIBUTING.md
3fdd5b971 A little more generic CONTRIBUTING.md
ed4fc31db Add link to CONTRIBUTING.md to main docs README.md
e4922e181 Forgot to truncate one blog post
6d8cd7ede Add CONTRIBUTING.md for rocksdb.org contribution guidance
bd55e5a1e Fix some formatting of compaction blog post
0f60358b0 CRLF -> LF mod (including removing trailing whitespace for those files)
b90e29c90 Truncate posts on the main /blog/ page
0d7acadaf Add author fields to blog posts
01be44181 Add GitHub link to the landing page header
9d6c96138 Fix Mac build
ab01da543 Support SST files with Global sequence numbers
d346ba246 Minor fixes around Windows 64 Java Artifacts (#1366)
e91b4d0cf Add factory method for creating persistent cache that is accessible from public
be1f1092c Expose transaction id, lock state information and transaction wait information
6009c473c Store range tombstones in memtable
3c21c64c7 Use size hint for HashMap in multiGet. Similar to https://github.com/facebook/rocksdb/pull/1344 (#1367)
13f7a01f6 Fixing JNI release build for gcc (#975)
7260662b3 Add Java API for SstFileWriter
26388247a delete unused variable for PrevInterval()
87dfc1d23 Fix conflict between AddFile() and CompactRange()
eb44ed655 Update 2016-09-28-rocksdb-4-11-2-released.markdown
e4437610d Update 2016-09-28-rocksdb-4-11-2-released.markdown
501f05108 Update 2016-09-28-rocksdb-4-11-2-released.markdown
dec9009f8 Update 2016-09-28-rocksdb-4-11-2-released.markdown
4ed69dd0b Create 2016-09-28-rocksdb-4-11-2-released.markdown
21f4bb5a8 cmake support for linux and osx (#1358)
4defe306f fix typo in comments (#1360)
f517d9dd0 Add SeekForPrev() to Iterator
eb3894cf4 Recompute compaction score on SetOptions (#1346)
5c64fb67d Fix AddFile() conflict with compaction output [WaitForAddFile()]
9e9f5a0b9 Fix CompactFilesTest.ObsoleteFiles timeout (#1353)
c2a62a4cb not cut compaction output when compact to level 0
9ed928e7a Split DBOptions into ImmutableDBOptions and MutableDBOptions
4bc8c88e6 Recover same sequence id from WAL (#1350)
0a1bd9c50 add cfh deletion started listener
da5a9a65c Fix mac build
d45eb6c6d Fix typo (#1349)
abc0ae462 Add AddFile() InternalStats for Total files/L0 files/total keys ingested
715256338 forbid merge during recovery
5735b3dc2 Fix compiling under -Werror=missing-field-initializers
654ed9a28 loose the assertion condition of rate_limiter_test
e4d3f5d9b Fix DBImpl::GetWalPreallocateBlockSize Mac build error
7afbb7420 solve the problem of table_factory_to_write_=nullptr (#1342)
d78a4401b DBImpl::GetWalPreallocateBlockSize() should return size_t
42ac9c5f1 Retry getting arcanist token on failure
b666f8544 Consider more factors when determining preallocation size of WAL files
4c3f4496b Add TableBuilderOptions::level and relevant changes (#1335)
3edb9461b Avoid hard-coded sleep in EnvPosixTestWithParam.TwoPools
0a88f38b7 Remove ColumnFamilyData::options()
41a9070f8 Fix java makefile dependencies
8d9bf5c49 Fix DBOptionsTest.GetLatestOptions
40cfa3e02 Fix DBWALTest.RecoveryWithLogDataForSomeCFs with mac
06b4785fe Fix recovery for WALs without data for all CFs
d7242ff4d Fix GetSortedWalFiles when log recycling enabled
17f76fc56 DB::GetOptions() reflect dynamic changed options
215d12826 Fix typo (#903)
a958c2643 Rename jvalue to jval in rocksjni
0a165bd7d Have Facebook link point to RocksDB on FB
3639f3288 Fix bug in UnScSigned-off-by: xh931076284 <931076284@qq.com> (#1336)
8e061f974 Refactor GetMutableOptionsFromStrings
81747f1be Refactor MutableCFOptions
ba65c816b Support POSIX RandomRWFile
1d980a8e3 Create CNAME
2adab1dde Add API links to the header bar
a182b2981 Preserve blog comments in markdown
f54de9230 Adding Dgraph to list of Users (#1291)
9e4aa798c Summary: (#1313)
a10e8a056 Fix C api memtable rep bugs. (#1328)
eb1d4d53c Release RocksDB 4.12
22d88e24d Allow an offset as well as a length to be specified for byte[] operations in RocksJava JNI (#1264)
b06b19136 add C api for set wal_recovery_mode (#1327)
1cca09129 Temporarily revert Prev() prefix support
de28a2553 Update HISTORY.md for thread-local stats
0fcb6dbed Remove extraneous function prototypes from c.h (#1326)
52ee07b02 Move AddFile() tests to external_sst_file_test.cc
66a91e260 Add NoSpace subcode to IOError (#1320)
67036c040 Fix Flaky ColumnFamilyTest.FlushCloseWALFiles
0e2da497c fix typo in option.h's comment (#1321)
6d61358a0 Add real Google Analytics ID
2d9d36ea4 Have "Edit on GitHub" point to master instead of gh-pages
937751898 Update landing page content
1ec75ee76 Add redirects from old blog posts link to new format
607628d34 Support ZSTD with finalized format
ce1be2ce3 Fix build error on Windows (AppVeyor) (#1315)
f7669b40b Fix Windows Build
22696b088 Fix uninitlized CompactionJob::SubcompactionState::current_output_file_size
c1865e0f7 Trigger more tests per diff
a88677d2c Remove ImmutableCFOptions from public API
80c75593e Fix data race in AddFile() with multiple files + custom comparator bug
5051755e3 Fix db_bench memory use after free (detected by clang_analyze)
4fd08f4b8 Ensure Correct Behavior of StatsLevel kExceptDetailedTimers and kExceptTimeForMutex  (#1308)
e14fbaae2 Add FAQ based on the front page of the current rocksdb.org
3c2262400 Migrate the RocksDB Worpdress blog over to Jekyll
ee0e2201e Transfer the current Getting Started contents to GitHub Pages
5a0e9a4cf Initial Landing Page
9447a8540 Remove the `doc` directory
32149059f Merge options source_compaction_factor, max_grandparent_overlap_bytes and expanded_compaction_factor into max_compaction_bytes
4590b53a4 add stats to Cache::LookUp()
85bb30825 Expose Utility function StringToMap() (#1306)
8ce1b8440 Fix Travis on Mac
380e651af Fix Mac build failure (#1309)
1613fa949 Thread-specific histogram statistics
6a14d55bd add prefix_seek_mode to db_iter_test
de47e2bd4 Fix ClockCache memory leak
f099af4c7 Fix travis
db74b1a21 fix bug in merge_iterator when data race happens
b18f9c9ea add nullptr check to internal_prefix_transform
4e395e875 Update docs README.md
2482d5fb4 support Prev() in prefix seek mode
7541c7a79 Fix cache_test valgrind_check failure
c8513cde0 Update the download location of Snappy (#1304)
b49b92cf2 Introduce Read amplification bitmap (read amp statistics)
c7004840d store prefix_extractor_name in table
4ad928e17 add comment to SimCache to estimate actual capacity
e9b2af87f Expose ThreadPool under include/rocksdb/threadpool.h
23a057007 Document memtable flush behavior in CancelAllBackgroundWork()
dade61ac2 Mitigate regression bug of options.max_successive_merges hit during DB Recovery
cce702a6e [db_bench] Support single benchmark arguments (Repeat for X times, Warm up for X times), Support CombinedStats (AVG / MEDIAN)
3586901f8 cat tests logs sorted by exit code
b2ce59537 Persist data during user initiated shutdown
4b3438d2d Fix parallel valgrind (valgrind_check)
a081f798b Relax consistency for thread-local ticker stats
b10d65c2a Update and slightly clarify instructions in build_detect_platform (#1301)
f85f99bf6 Fix the Windows build of RocksDB Java. Similar to https://github.com/facebook/rocksdb/issues/1220 (#1284)
7b8109517 Fix a crash when compaction fails to open a file
7c9586837 Thread-specific ticker statistics
ea9e0757f Add initial GitHub pages infra for RocksDB documentation move and update. (#1294)
2a9c97108 [Flaky Test] Disable DBPropertiesTest.GetProperty
d76ddf327 Disable ClockCache db_crashtest
cec2c6436 fix data race in NewIndexIterator() in block_based_table_reader.cc
badbff65b Not insert into block cache if cache is full and not holding handle
4a16c32ec Option to cache index/filter blocks with priority
99c4af716 Make ClockCache available with TSAN build
f57bc1d03 Fix lambda expression for clang/windows
5440675c3 Fix lambda capture expression for windows
6584cec8f Fold function for thread-local data
817eeb29b Add singleDelete to RocksJava (#1275)
ffdf6eee1 Add Status to RocksDBException so that meaningful function result Status from the C++ API isn't lost (#1273)
ecf900386 Fix bug in printing values for block-based table
72f8cc703 LRU cache mid-point insertion
6a17b07ca Add TablePropertiesCollector support in SstFileWriter
78837f5d6 TableBuilder / TableReader support for range deletion
4cc37f59e Introduce ClockCache
ff17a2abf Adding TBB as dependency.
49d88be02 c abi: allow compaction filter ignore snapshot (#1268)
0b63f51fb fixes 1215: execute_process(COMMAND mkdir ${DIR}) fails to create a directory with cmake on Windows (#1219)
3981345be Small nits (#1280)
2a2ebb6f5 Move LRUCache structs to lru_cache.h header
2fc2fd92a Single Delete Mismatch and Fallthrough statistics
3771e3797 WriteBatch support for range deletion
236756f2c Make SyncPoint return immediately when disabled
64a0082c6 Fix DBSSTest::AddExternalSstFileSkipSnapshot valgrind fail
dd7a748cf Fix java build
4fe12baa6 Make db_bench less space for --stats_per_interval
6525ce4ca Compaction stats printing: "batch" => "commit group"
a117891b4 Fixed typo (#1279)
b248e98cf Fix a destruction order issue in ThreadStatusUpdater
deda159b5 Added min/max/avg data block size output to sst_dump
e408e98c8 add Name() to Cache
a297643f2 Fix valgrind memory leak
d11c09d9e Eliminate memcpy from ForwardIterator
d36755502 Added further Java API options for controlling concurrent writes
ebdfe34cc Exposed further Java API options for controlling compaction
d1be59463 Improve documentation of SliceTransform.
6056d6317 Improve comment and bug fix for GetOptionsFromMap functions in convenience.h
76a67cf74 support stackableDB as the baseDB of transactionDB
67c1ae883 Travis build break fix
b693ba68b Minor PinnedIteratorsManager Refactoring
db3dfb164 Fixes for arcanist config (#1271)
87c91bd87 Persistent Read Cache (8) Benchmark tooling
2914de64e add sim_cache stats to Statistics
8b79422b5 [Proof-Of-Concept] RocksDB Blob Storage with a blob log file.
4beffe001 Fix test data race in two FaultInjectionTest tests
821bcb0b3 util/arena.cc: FreeBSD: More portable use of mmap(MAP_ANON) (#1254)
5370f44a8 Increase RocksDB version
56dd03411 read_options.background_purge_on_iterator_cleanup to cover forward iterator and log file closing too.
ccecf3f4f UniversalCompaction should ignore sorted runs being compacted (when compacting for file num)
1b0069ce2 Remove non-gtest from parallelized tests
638c49f24 Change HISTORY.md for release 4.11
6b8e9c68b fix vs generator (#1269)
c38b075e7 Update HISTORY.md
8f399e3fe Update HISTORY.md
98d0b78ea Added check_snapshot option in the DB's AddFile function (#1261)
9fd68b7fb set travis open file descriptor limit
59ddb5059 Fix travis build break
f4d986364 Added SetOptions support to RocksJava (#1243)
7882cb977 Make DBOptionsTest::EnableAutoCompactionAndTriggerStall less falky
44f5cc57a Add time series database (resubmitted)
7c4615cf1 A utility function to help users migrate DB after options change
5bb0a7f73 Update appveyor.yml
86396cc18 Update appveyor.yml
c1db098dc Update appveyor.yml
7da2eaf0d Update appveyor.yml
34723b4c4 Cleanup unused variable pending_fsync_.
7cc0dbd66 cat all logs in sandcastle output
9253767a6 Correct geHistogramData() -> getHistogramData() (#1257)
f35b16f24 db_bench add an option of --base_background_compactions
c3a4bea5d Fix flaky test `ObsoleteFiles`
8234faabf Fix failed test
4990c0d1a Remove deprecated LEVELDB_PLATFORM_POSIX
7323e4c8a Fix clang on macOS
ee027fc19 Ignore write stall triggers when auto-compaction is disabled
e4609a749 Fix Windows build issues (#1253)
2306167d3 Fix clang build failure and refactor unit test
343304e1d Use StopWatch to do statistic job in db_impl_add_file.cc
cdc4eb689 Add a GetComparator() function to the ColumnFamilyHandle base class so that the user's comparator can be retrieved.
712dd27e6 Build break fixes
0155c73de Fix parallel tests `make check -j`
c49ea68c5 Fix to enable running CI jobs locally
726c2f7e5 Build break fix
d51dc96a7 Experiments on column-aware encodings
c116b4780 Persistent Read Cache (part 6) Block Cache Tier Implementation
64046e581 Write a benchmark to emulate time series data
9ae92f50b More granular steps in the Makefile, can help with running all or single Java tests (and with ASAN build - https://github.com/facebook/rocksdb/wiki/JNI-Debugging) (#1237)
7c01d6534 [Fix Java] Remove duplicate cases in LoggerJniCallback::Logv
8796934af Added missing Java ReadOptions settings (#1109)
5e2c79658 Make DBTest.CompressionStatsTest more deterministic
557748ff7 Fix db_stress failure (pass merge_operator even if not used)
811ee2111 Bugfix to ensure that logging can be achieved from threads that are not known to the JVM (#1106)
afad5bd1c Simplify thread-local static initialization
6920cde89 Remove an extra apostrophe
e72ea485e add InDomain regression test
9c8ac144b Avoid duplicate task creation for RocksDB contruns
d4c45428a db_stress shouldn't assert file size 0 if file creation fails
d3bfd3397 Testing out parallel sandcastle changes
7efd9c25c Increse timeout in some tests
50b8d29b9 fixes 1230: Error:string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command (#1231)
5c858ddd2 fix errata in libnuma test (#1244)
e5b5f12b8 Change options memtable_prefix_bloom_huge_page_tlb_size => memtable_huge_page_size and cover huge page to memtable too
0ce258f9b Compaction picker to expand output level files for keys cross files' boundary too.
ac0d93b08 fixes 1217: rocksdbjni javac and javah execute_processes fail on windows (#1218)
1ae46094d Appveyor badge to show master branch
8745f013f [Fix java build] Stop using non standard std::make_unique
e12270dfe fix previous typo
bbd6a5a18 ldb restore subcommand
9498069fc Run error-filtering script on diff-triggered tests
f8061a237 Fix Statistics TickersNameMap miss match with Tickers enum
16e225f70 Fix MergeContext::copied_operands_ strings moving
a4955b39a Run sandcastle tests in /dev/shm
ae0ad719d Fix flaky DBSSTTEST::DeleteObsoleteFilesPendingOutputs
b2a8016df Update db_bench_tool.cc (#1239)
c6654588b Disable two dynamic options tests under lite build
2a6d0cde7 Ignore stale logs while restarting DBs
ee8bf2e41 fixes 1228: rockdbjni loadLibraryFromJarToTemp fails when file is already present (#1232)
f85df120f Re-enable tsan crash white-box test with reduced killing odds
89e4c4882 Update README.md to include appveyor badge
b06ca5f86 ldb load, prefer ifsteam(/dev/stdin) to std::cin (#1207)
4ea0ab3cc Revert "Remove bashism from `make check` (#1225)"
12767b313 fixes 1220: rocksjni build fails on Windows due to variable-size array declaration (#1223)
a9d512a76 Update .gitignore for internal release
d5a51d4de Need to make sure log file synced before flushing memtable of one column family
89f319c2d Fix unit test which breaks lite build
b50632920 Add unit test not on /dev/shm as part of the pre-commit tests
b9a97181a Bump next release version
663afef88 Add EnvLibrados - RocksDB Env of RADOS (#1222)
32604e660 Fix flush not being commit while writing manifest
9ab38c45a Remove %z Format Specifier and Fix Windows Build of sim_cache.cc (#1224)
08ab1d83a Remove bashism from `make check` (#1225)
f9b14be49 Re-enable TSAN crash test but only with black box crash test
68f3eb746 Run release build for CLANG and GCC 4.8.1 in pre-commit tests too
e70020e4f Only cache level 0 indexes and filter when opening table reader
7bedd9440 Build break fix
68a8e6b8f Introduce FullMergeV2 (eliminate memcpy from merge operators)
e70ba4e40 MemTable::PostProcess() can skip updating num_deletes if the delta is 0
2a282e5f5 DBTablePropertiesTest.GetPropertiesOfTablesInRange: Fix Flaky
d9cfaa2b1 Persistent Read Cache (6) Persistent cache tier implentation - File layout
9430333f8 New Statistics to track Compression/Decompression (#1197)
515b11ffa fixes #1210: rocksdb/java/CMakeLists.txt lacks cmake_minimum_required (#1214)
876cb8bfb fixes #1212: rocksdbjni maven build does not escape slashes in groovy script (#1213)
21c55bdb6 DBTest.DynamicLevelCompressionPerLevel: Tune Threshold
4b9525358 Refactor cache.cc
c6a8665b3 Update LANGUAGE-BINDINGS.md
880ee363e ldb backup support
6797e6ffa Avoid updating memtable allocated bytes if write_buffer_size is not set
dda6c72ac Add DestroyColumnFamilyHandle(ColumnFamilyHandle**) to db.h
56222f57d Avoid FileMetaData copy
15b7a4ab8 Fixed output size and removed unneeded loop
6ea41f852 Fix deadlock when trying update options when write stalls
efd013d6d Miscellaneous performance improvements
e6f68faf9 Update Makefile to fix dependency
816ae098e fix test failure
e295da126 Fix Log() doc for default level
8e6b38d89 update DB::AddFile to ingest list of sst files
296545a2c Fix clang analyzer errors
61dbfbb6c Add release build to RocksDB per-diff/post-commit tests
907f24d0e Concurrent memtable inserter to update counters and flush state after all inserts
0f691c4b5 CLI option & Rename() allow overwrite
7c190070b delete unnessary pointer cast in beginInternalTransaction() function
e1b3ee8a7 Cleanup auto-roll logger flush-while-rolling test
cd4178a01 Add a new feature to enforce a sync point only active on a thread
b954847fc Fix release build for MyRocks by using debug-only code only in debug builds
a00bf1b3c Add More Logging to track total_log_size
01f77cb19 Update USER.md to include more services at Facebook.
eb53c05a3 Add comment for GetBackupInfo about returned BackupInfos order
32df9733d Add options.write_buffer_manager: control total memtable size across DB instances
5aaef91d4 group multiple batch of flush into one manifest file (one call to LogAndApply)
a45ee8318 Fix a bug that accesses invalid address in iterator cleanup function
38fae9e65 fix typos in HISTORY.md (#1192)
1a11c934d Disable some persistent cache tests on linux/clang
9b5adea97 Add More Logging to track total_log_size
95d96eeeb remove LockFile
ff45d1b54 if read only backup engine can't find meta dirs, return NotFound() instead of IOError()
cb2476a0c fix rate limiter to avoid starvation
6b7167651 Run env_basic_test on Env::Default
9eb0b5395 Move env_basic_test cleanup to TearDown
1fe3bf829 Re-enable linux on travis
43692793e Fixed Minor Bug on Windows Build and db_bench_tool.cc (#1189)
95c192475 writable file close before reset
197b832af Update USERS.md
bdb1d19a6 Fix UBSan build break caused by variable not initialized
b726bf596 FreeBSD does not have std::to_string (#1190)
faa7eb3b9 Improve regression_test.sh
c4cef07f1 Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0
892e9d304 make transaction WriteOptions modifiable
4f2b0946d fix simple typos (#1183)
3b7ed677d ColumnFamilyOptions API [CF + RepairDB part 3/3]
56ac68629 Detect column family from properties [CF + RepairDB part 2/3]
3fc713ed9 delete 2nd level children for default env
343507afb Refactor to use VersionSet [CF + RepairDB part 1/3]
aa432be4b Workarounds for continuous build implementation
8cd9f04fe Test change to verify new commit detection
8a4ee7e90 Trivial change to test cont. build
af6248d8b Fix max_bytes_for_level_base comment
0d7b26123 add tests to env_basic_test.cc
6576fa05a Fix minor typos and PHP source file name used to trigger the builds
c4e19b77e Add a read option to enable background purge when cleaning up iterators
fa813f747 Update DB::AddFile() to ingest the file to the lowest possible level
d6b79e2fd Remove filter_deletes from crash_test
a52e4d7d0 Framework for enabling continuous RocksDB build and tests
f9bd66779 Makefile warning for invalid paths in make_config.mk
88a2776db Update SstFileWriter to use bottommost_compression if avaliable
e87d5df1a TiKV use-case (#1172)
7b79238b6 Deprectate filter_deletes
4939fc389 Bulk load mode shouldn't stop ingest
3a2bccc84 Fixed a crash bug that incorrectly parse deprecated options in options_helper
cf8adc971 Allow arcanist_util to work with both new and old arc versions
30a24f2d3 Add InternalStats and logging for AddFile()
d26a84807 Temporarily disable travis on linux
249e796df Fix Flaky DBCompactionTest.SkipStatsUpdateTest
162c9170d Make sandcastle access secure
8366e10ff Fix clang build
0babce57f Move away from enum char value -1
812dbfb48 Optimize BlockIter::Prev() by caching decoded entries
550bf895e Minor bug fix with log name
886af5910 Fix examples/Makefile jemalloc error
e3b1e3dfa Expose save points in Java WriteBatch and WBWI (#1092)
f5177c761 Remove wasteful instrumentation in FullMerge (stacked on D59577)
97fd2a638 Remove dead Jenkins code and support `arc diff --preview` in RocksDB
7c919decc Reuse TimedFullMerge instead of FullMerge + instrumentation
9a33a723b Remove the comments saying allow_concurrent_memtable_write and enable_write_thread_adaptive_yield are not stable
81f6b33d9 Fix tsan error
bc8af90e8 add option to not flush memtable on open()
8100ec2cd Fix libgcc broken lib path
7360db39e Add a check mode to verify compressed block can be decompressed back
2a79af1c5 Fix Java Break Related to memtable bloom bits to size ratio change
6faddd7c5 Merge db/slice.cc into util/slice.cc
5009b5326 BlockBasedTable::FullFilterKeyMayMatch() Should skip prefix bloom if full key bloom exists
2d05eaeb2 Fix name conflict in delete_shceduler_test and db_sst_test
bde7d1055 Fix clang_analyze path in fbcode_config.sh
ca3db5478 Fetch branches from github for format compatibility test
20699df84 memtable_prefix_bloom_bits -> memtable_prefix_bloom_bits_ratio and deprecate memtable_prefix_bloom_probes
e9c1face6 Minor fix to precommit-check.py
fcc47fa5f New features to precommit check script
56887f6cb Backup Options
a683d4aba URI-based Env selection for db_bench
53a4bd8a6 duplicate line
3e8686961 release 4.9 update version and history
b2973eaae Remove options builder
5b197bff4 Enabled Windows build for volatile tier implementation
281fbdddc Temporarily remove PersistentCacheOptions from persistent_cache_tier.h to fix unity build
2ae15b2d5 Format compatible test should cover forward compatibility up to 4.8.
00a058725 netflix use-case
5091dfc1e use branch names in format compatibility test
edc764e91 Use valgrind built with gcc-4.9-glibc-2.20
8ff59b2b4 Disable PersistentCacheTierTest.VolatileCacheInsertWithEviction test under TSAN temporarily
1ba452226 Fix for GCC 5.4 (#1157)
972c895c3 Previously WARN level logging became FATAL level logging in the Java API (#1089)
a73b26f60 Adding test for contiguous WAL detection
098da8348 Fix CLANG build break caused by the recent Persistent Cache change
54db29b8f Use gvfs links in dependencies.sh
d755c62f9 Persistent Read Cache (5) Volatile cache tier implementation
fda098461 Allow regression_test.sh to specify OPTIONS_FILE.  Add header comments.
0fee89684 Fix Windows build
10d46b9c8 Update tp2 clang path to fix clang build
774a6aa29 Java API - Rename geHistogramData -> getHistogramData (#1107)
3070ed902 Persistent Read Cache (4) Interface definitions
e42dc9192 Update paths for fbcode dependencies
5647fa427 stack_trace,cc: The current Stacktrace code does not compile for FreeBSD (#1153)
0d65acec0 threadpool.cc: abort() lives in stdlib.h on FreeBSD (#1155)
19dd5a61c env_chroot.cc: FreeBSD likes stdlib.h for realpaht() and friends (#1154)
5aca977be env_basic_test library for testing new Envs [pluggable Env part 3]
1147e5b05 Adding support for sharing throttler between multiple backup and restores
6e6622abb Create env_basic_test [pluggable Env part 2]
e53287794 Add statistics field to show total size of index and filter blocks in block cache
a791a2cf2 Java API - Bugfix for native linking of Compaction Filter (#1099)
af0c9ac01 Env registry for URI-based Env selection [pluggable Env part 1]
02ec8154e allow updating block cache capacity from C (#1149)
630b732cb fix flaky sim_cache_test
62d548098 Add persistent cache to Windows build system
842958651 Fix race condition in SwitchMemtable
88acd932f Allows db_bench to take an options file
3a276b0cb Add a callback for when memtable is moved to immutable (#1137)
8cf0f86d3 Allow regression test to run db_bench at a remost host
27ad17071 Fix Windows build break
936973d14 Small tweaks to logging to track the number of immutable memtables
21c047ab4 add readahead size option (#1146)
71c7eed91 Assert boundary checks for SetPerfLevel()
5d85fdb2c add missing lock
c40c4cae1 LDBCommand::SelectCommand to use a struct as the parameter
590e2617e fix delete file bug when do checkpoint (#1138)
8dfa980cb Add statically-linked library for tools/benchmarks
f62fbd2c8 Handle overflow case of rate limiter's paramters
57461fba8 In-memory environment read beyond EOF
0e2000017 LDBCommand::InitFromCmdLineArgs() to move from template to function wrapper
472c06e90 Add low and upper bound values for rocksdb::PerfLevel enum
157e0633e MutexLock -> ThreadPoolMutexLock in util/threadpool.cc
23d4cf483 include/rocksdb/sst_file_writer.h should not depend on util/mutable_cf_options.h
345fd73fa Fix flaky DBTestDynamicLevel.DynamicLevelMaxBytesBase2
8fc75de32 Minor fix to disable DynamicLevelMaxBytesBase2
9dd50d990 Fix db_bench
5d660258e add simulator Cache as class SimCache/SimLRUCache(with test)
d379d110e Update CMakeLists.txt for added test
21f847eda Direct IO fix for Mac
99765ed85 Clean up the ComputeCompactionScore() API
def2f7bd0 Expose report_bg_io_stats option in the C API. (#1131)
f89caa127 Direct IO capability for RocksDB
8f1214531 C API: Expose DeleteFileInRange (#1132)
11f329bd4 db/db_impl: restrict WALRecoveryMode when using recycled log files
2b2a898e0 db/log_reader: combine kBadRecord{Len,Checksum} for readability
34df1c94d db/log_reader: treat bad record length or checksum as EOF
7947aba68 db/log_reader: move kBadRecord{Len,Checksum} handling into ReadRecord
847e471db db/log_test: add recycle log test
4e7e41ba7 Disable lite build/testing for persistent read cache
1d725ca51 Deprecate BlockBasedTableOptions.hash_index_allow_collision=false.
0e77246ba backupable_db.cc: lambada to explictly caputre "this" when escaping scope
2073cf377 Eliminate use of 'using namespace std'.  Also remove a number of ADL references to std functions.
26adaad43 Split WinEnv into separate classes. (#1128)
bb98ca3c8 Implement GetUniqueId for Mac
1f2dca0ea Add MaxOperator to utilities/merge_operators/
f6e404c20 Added "number of merge operands" to statistics in ssts.
7383b64b3 Fix formatting of HISTORY.md (#1126)
0e665c399 Disable long running GroupCommitTest (#1125)
3c69f77c6 Move IO failure test to separate file
533cda90c Add GetStringFromCompressionType to include/rocksdb/convenience.h
c70a9335d Fix mutex unlock issue between scheduled compaction and ReleaseCompactionFiles()
05c5c39a7 Fix build
a6254f2bd Long outstanding prepare test
2ead11511 Fix TransactionTest.TwoPhaseMultiThreadTest under TSAN
1f0142ce1 Persistent Read Cache (Part 2) Data structure for building persistent read cache index
43afd72be [rocksdb] make more options dynamic
bac3be7c4 Fix build issue. (#1123)
f6aedb62c Fix Transaction memory leak
a08c8c851 Added PersistentCache abstraction
5c06e0814 [ldb] Templatize the Selector
aab91b8d8 Use generic threadpool for Windows environment (#1120)
a40033639 TransactionLogIterator sequence gap fix
fa3536d20 Store SST file compression algorithm as a TableProperty
40123b380 signed vs unsigned comparison fix
49815e384 [ldb] Export LDBCommandRunner
c1af07ce8 Disable backupable_db_test.cc on Windows
e61ba052b Isolate db env and backup Env in unit tests
560358dc9 Fix data race in GetObsoleteFiles()
5c1c90487 ldb option for compression dictionary size
c27061dae [rocksdb] 2PC double recovery bug fix
a657ee9a9 [rocksdb] Recovery path sequence miscount fix
8a66c85e9 [rocksdb] Two Phase Transaction
1b8a2e8fd [rocksdb] Memtable Log Referencing and Prepared Batch Recovery
0460e9dcc Modification of WriteBatch to support two phase commit
f548da33e Follow symlinks in chroot directory
d86f9b9c3 Fix lite build
4b3172343 Add bottommost_compression option
bfb6b1b8a Estimate pending compaction bytes more accurately
258459ed5 Properly destroy ChrootEnv in env_test
fca5aa6fc Initial script for the new regression test
e1951b6f2 Add --index_block_restart_interval option in db_bench
730f7e2e2 Fix win build
a9b3c47c8 Fix includes for clang on OS X
3f16a836a Introduce chroot Env
269f6b2e2 Revert "Modification of WriteBatch to support two phase commit"
04dec2a35 [ldb] Export ldb_cmd*.h
72c73cdc8 Java API - Add missing HEADER_LEVEL logging (#1104)
4d02bfa3a Add support for PauseBackgroundWork and ContinueBackgroundWork to the Java API (#1087)
8f65feafc Have sandcastle run lite_test for every diff
0d590d999 Make max_dict_bytes optional in options string
7ccb8d6ef BlockBasedTable::Get() not to use prefix bloom if read_options.total_order_seek = true
e3c6ba37d OptimizeForSmallDb(): revert some options whose defaults were just changed
967476eae Fix valgrind (DBIteratorTest.ReadAhead)
9790b94c9 Add optimize_filters_for_hits option to db_bench
a4ea345b0 Fixing lite build
24a24f013 Enable configurable readahead for iterators
ff4b3fb5b Fix Iterator::Prev memory pinning bug
cba752d58 sst_dump won't print size for unsupported compression type
6e801b0bd Eliminate memcpy in Iterator::Prev() by pinning blocks for keys spanning multiple blocks
1b166928c Release RocksDB 4.8.0
b8cf9130f Fix #1110, 32-bit build failure on Mac OSX (#1112)
21441c09b Fix calling GetCurrentMutableCFOptions in CompactionJob::ProcessKeyValueCompaction()
4ea6e051e Fix multiple issues with WinMmapFile fo sequential writing (#1108)
f3bb024fd Fix clang build
6e56a114b Modification of WriteBatch to support two phase commit
1d2e4ef74 ldb support new WAL records
a92049e3e Added EventListener::OnTableFileCreationStarted() callback
e8115cea4 Revert "Use async file handle for better parallelism (#1049)" (#1105)
6a14f7a97 Change several option defaults
c6c770a1a Use prefix_same_as_start to avoid iteration in FindNextUserEntryInternal. (#1102)
992a8f83b Not enable jemalloc status printing if USE_CLANG=1
029022b0f Fix crash_test
a06faa632 Skip PresetCompressionDict test for lite
e7899c661 Fix build issue. (#1103)
0f428c561 Fix compression dictionary clang osx error
6d4832a99 Merge pull request #1101 from flyd1005/wip-fix-typo
af70f9ac6 Fix typo in build_tools/fbcode_config.sh
54de13aba Fix compression dictionary clang errors
0850bc514 Fix build on machines without jemalloc
4032145ad Configurable compression in db_bench
843d2e313 Shared dictionary compression using reference block
ad573b902 Temporarily disable CompactFiles in db_stress in its default setting
1c80dfab2 Print memory allocation counters
eb7398085 Fix BackupableDBTest
ac0e54b4c CompactedDB should not be used if there is outstanding WAL files
d719b095d Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData)
1995e34d6 Retrieve file size from proper Env
7c14abf2c Improve BytewiseComparatorImpl::FindShortestSeparator
f3eb0b5b8 Make EventListenerTest.CompactionReasonLevel more deterministic
7b78d623f Shouldn't report default column family's compaction stats as DB compaction stats
995353e46 Fix null-pointer-dereference detected by Infer (https://github.com/facebook/infer)
24110ce90 Correct Statistics FLUSH_WRITE_BYTES
b71c4e613 Alpine Linux Build (#990)
90ffed1f9 Update USERS.md with link to LinkedIn blog post (#1088)
99a3bf8f6 Merge pull request #1068 from daaku/c-purge-old-backups
b54c34742 Use async file handle for better parallelism (#1049)
c146c9be1 rocksdb_create_mem_env to allow C libraries to create mem env (#1066)
6da70c581 expose more options in the c api (#1067)
6f01687aa C rocksdb_create_iterators to expose NewIterators (#1069)
644f978c1 Fix RocksDB Lite build in db_stress
5bd4022fe Add comparator, merge operator, property collectors to SST file properties (again)
7a6045a3c fix typo in HISTORY.md
73a847ef8 Add per-level compression ratio property
ee221d2de Introduce XPRESS compresssion on Windows. (#1081)
874c96ac1 Merge pull request #1083 from flabby/master
6cbffd50d Enable testing CompactFiles in db_stress
b95510ddf Fix DBTest.RateLimitedDelete flakiness
6356b4d51 Fix nullptr dereference in adaptive_table
9385fd72c Delete deprecated backup classes in Java
a2466c885 [db_stress] Make subcompaction random in crash_test
c3c389d54 Fix column label for L0 write sum
ec84bef24 New legocastle output parsing
725184b04 Fix db_block_cache_test in lite build
290883d94 Fix lite build
23089fd28 write_callback_test: clean test directory before running tests
792762c42 Split db_test.cc
40b840f29 Delete deprecated *BackupableDB interface for backups
6affd45d8 Make more tests run in parallel
47833e0ab Merge branch 'master' of github.com:facebook/rocksdb
e5c614e1d Fixing snapshot 0 assertion
1b1adebe8 Fixing snapshot 0 assertion
6d436a3f8 DBTest.HardLimit made more deterministic
994d9bc82 Make parallel valgrind watch-log more readable
9d35ae649 Make DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels more deterministic
cea8ed970 Fix backupable_db_test test cases that can't run by itself
4b6833aec Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too.
3894603fe Allow valgrind_check to run in parallel
c9d668c58 Fix unit tests issues on Windows (#1078)
083cadc7e Minor fix to Java sandcastle job definition
80b74a6c6 Include ldb tool in the windows build (#914)
7c14d11eb Minor fix to java build job
535af525d BlockBasedTable::PrefixMayMatch() to skip index checking if we can't find a filter block.
09be5cad5 Minor fix to sandcastle java job definition
1aeca9733 Release RocksDB 4.7
dfc3de8b7 Split Travis unittests Job
c2c8fe47f Add Java job for sandcastle
19ef3de57 Fix ManualCompactionPartial test flakiness
b345b3662 Add a minimum value for the refill bytes per period value
dff4c48ed BlockBasedTable::PrefixMayMatch: no need to find data block after full bloom checking
0353b853c Propagate sandcastle run error to UI
b885f33a5 Parallelize travis jobs
71303e04e Update db_bench_tool.cc (#1073)
63cf15bb9 Fix option settable tests
e208575b2 using java7 in runtime for hdfs env (#1072)
13e6c8e97 Relax an assertion in Compaction::ShouldStopBefore
ae21d71e9 Fixed a bug in RocksDB Statistics where flush is considered as compaction
8e0e22f76 Fix Windows build by replacing strings.h include
5675d5037 Revert travis commit
91f0f1f5e fix travis
a23c6052c Don't run DBOptionsAllFieldsSettable under valgrind
30d72ee43 PrefixTest.PrefixAndWholeKeyTest should run against a different directory from prefix_test
0e3cc2cf1 Add column family info to TableProperties::ToString()
2448f8037 Make sure that if use_mmap_reads is on use_os_buffer is also on
114a1b879 Fix build errors for windows
052299035 Improve sst_dump help message
0930e5e99 Update comments on include/rocksdb/perf_context.h
3b977bcdc instructing people to use java7 for hdfs (#1063)
1518b733e Change default number of cache shard bit to be 6 and max_file_opening_threads to be 16.
ada88b63f fix wrong assignment of level0_stop_writes_trigger in spatialdb (#1061)
2391ef721 Embed column family name in SST file
ab4c62332 Don't use version in the error message
d02eb8d00 Fix unused variable warning
9278097f8 Merge pull request #1056 from facebook/igorcanadi-patch-1
cc87075d6 No need to limit to 20 files in UpdateAccumulatedStats() if options.max_open_files=-1
8a1a603fd Eliminate std::deque initialization while iterating over merge operands
f38540b12 WriteBatchWithIndex micro optimization
200654067 Merge pull request #1053 from adamretter/benchmark-java-comparator
f2c43a4a2 Stderr info logger
b55e2165b Rocksdb backup can store optional application specific metadata
9b5198752 Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes.
2feafa3db Change some RocksDB default options
a558830f8 Fixed compile warnings in posix_logger.h and coding.h
51c9464df Merge pull request #980 from adamretter/java-arm
925b5d002 Merge pull request #1054 from DCEngines/magic12
63e8f1b55 Formatted lines to adhere to 80 char limit
994b3bd69 Add support for UBsan builds to RocksDB
21700a510 to/from hex refactor
24420947d Replace kHeader by WriteBatchInternal::kHeader in few more places kHeader was moved from write_batch.cc to header file because it is being used wherever the number "12" was being used to check for record size
3bdbe8961 Merge branch 'magic12' of https://github.com/dcengines/rocksdb into magic12
78711524b In all the places where log records are read, there was a check that record.size() should not be less than 12.
e7c64fb11 Imporve sst_file_manager comment
99ffb3d53 Fix perf_context::merge_operator_time_nanos calculation
07bb12d97 Update internal jemalloc and other versions
ad2fdaa82 Correct a typo in a comment
be9816b3d Fix data race issue when sub-compaction is used in CompactionJob
e3802531f Merge pull request #1050 from yuslepukhin/support_db_test2
e7cc49cbd Add support for db_test2 for dev and CI runs
3996770d0 Add comments to perf_context skip counters
4e85b7479 Make WritableFileWrapper not screw up preallocation
ec458dcde Merge pull request #1047 from PraveenSinghRao/wal_filter_ex
583157f71 Avoid overloaded virtual function
b9d4fa890 Options settable tests to use a different special charactor
60e34baef Merge pull request #1044 from PraveenSinghRao/wal_filter_ex
136b8e0ca Merge from master
2dcbb3b4f Addressed review comments
b1fafcaca Revert "Adding pin_l0_filter_and_index_blocks_in_cache feature."
5f8741a69 Revert "Fix failing Java unit test."
43bbb5619 tools/check_format_compatible.sh to use consistent version when testing backward and forward compatibility
d7ae42b0f Fix failing Java unit test.
fbbb8a614 Add test for Snapshot 0
e182f03c1 Add unit tests for RepairDB
7d371863e travis build fixes
fbea4dc66 Merge pull request #1042 from SherlockNoMad/HistFix
780d2b04c Update format compatible checking tool
4f1c74a46 merge from master
f8c218930 Publish log numbers for column family to wal_filter, and provide log number in the record callback
44756260a Reset block cache in failing unit test.
522de4f59 Adding pin_l0_filter_and_index_blocks_in_cache feature.
be2227126 Merge pull request #1041 from yuslepukhin/adjust_for_jemalloc
4ecc03c03 Fix in HistogramWindowingImpl
2ca0994cf Latest versions of Jemalloc library do not require je_init()/je_unint()   calls. #ifdef in the source code and make this a default build option.
90aff0c44 Update --max_write_buffer_number for compaction benchmarks
72224104d Forge current file for checkpoint
33d568611 Merge pull request #1040 from bureau14/master
02e62ebbc Fixes warnings and ensure correct int behavior on 32-bit platforms.
9cad56861 Merge pull request #1039 from bureau14/master
3d29f9146 Improve documentation of the allow_os_buffer parameter.
3ff98bd20 Fix no compression test
b9cc42a72 Merge pull request #1038 from SherlockNoMad/HistFix
f76b260ef Fix FB internal CI build failure
774922c68 Merge pull request #1026 from SherlockNoMad/Hist
17b879b91 Merge pull request #1037 from SherlockNoMad/BuildFix
f11b0df12 Fix AppVeyor build error
6b03f93d4 Fix the build break on Ubuntu 15.10 when gcc 5.2.1 is used
697fab820 Updates to RocksDB subcompaction benchmarking script
58379bfb5 remove division from histogramwidowing impl
1a2cc27e0 ColumnFamilyOptions SanitizeOptions is buggy on 32-bit platforms.
e778c34e7 Merge pull request #1035 from bureau14/master
5bd3da1c5 Added quasardb to the USERS.md file
b2ae5950b Index Reader should not be reused after DB restart
0267655da Update change log for 4.6 release
08304c086 Expose RepairDB as ldb command
fd664a27b Fix Build Error
580fede34 Aggregate hot Iterator counters in LocalStatistics (DBIter::Next perf regression)
54f6b9e16 Histogram Concurrency Improvement and Time-Windowing Support
790252805 Add multithreaded transaction test
e8e6cf017 fix: handle_fatal_signal (sig=6) in std::vector<std::string, std::allocator<std::string> >::_M_range_check | c++/4.8.2/bits/stl_vector.h:794 #174
d9620239d Cleanup stale manifests outside of full purge
f71fc77b7 Cache to have an option to fail Cache::Insert() when full
ee8cc3520 Merge pull request #938 from alexander-fenster/master
765597fa7 Update compaction score right after CompactFiles forms a compaction
f0161c37b formatting fix
aa3f02d50 Improve comment in compaction.h and compaction_picker.h
2200295ee optimistic transactions support for reinitialization
badd6b784 Ignore db_test2
200080ed7 Improve snapshot handling for Transaction reinitialization
171c8e80b Update dependencies / Fix Clang
294bdf9ee Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number"
bf1c4089d Use pure if-then check instead of assert in EraseColumnFamilyInfo
a7d4eb2f3 Fix a bug where flush does not happen when a manual compaction is running
68189f7e1 Update benchmarks used to measure subcompaction performance
dfe96c72c Fix WriteLevel0TableForRecovery file delete protection
451678c8c Merge pull request #1025 from SherlockNoMad/BuildFix
58ecd9132 Fix Windows build
501927ffc [backupable db] Remove file size embedded in name workaround
ef204df7e Compaction always needs to be removed from level0_compactions_in_progress_ for universal compaction
e79ad9e18 Add Iterator Property rocksdb.iterator.version_number
19ea40f8b Subcompaction boundary keys should not terminate after an empty level
deb08b822 Add parsing of missing DB options
f8e90e875 Get file attributes in bulk for VerifyBackup and CreateNewBackup
12fd9b186 Change BlockBasedTableOptions.format_version default to 2
4572a2d8c Update current version to 4.6
74b660702 Rename iterator property "rocksdb.iterator.is.key.pinned" => "rocksdb.iterator.is-key-pinned"
6743135ea Fix DB::AddFile() issue when PurgeObsoleteFiles() is called
432f3adf2 Add DB Property "rocksdb.current_version_number"
188bb2e7a Fix formatting identified by `arc lint`
0f2d2fcff Refactored tests to use try-with-resources
f8e02c782 Deprecate org.rocksdb.AbstractNativeReference#dispose() and implement java.lang.AutoCloseable
0f2fdfe23 Fix the javadoc and the formatting of the base-classes for objects with native references
b5b1db167 Recompute compaction score after scheduling manual compaction
5ea9aa3c1 TransactionDB:ReinitializeTransaction
1f5954147 Introduce Iterator::GetProperty() and replace Iterator::IsKeyPinned()
67789419f Merge pull request #1020 from gongsu832/master
69c471bd9 Handle concurrent manifest update and backup creation
3373c81fa Modify build_tools/build_detect_platform to detect and set -march=z10 on Linux s390x.
990509045 Merge branch 'master' of https://github.com/gongsu832/rocksdb
3492889ab Merge pull request #1019 from javacruft/wip-omit-leaf-frame-pointer-archs
7ca731b12 build: Improve -momit-leaf-frame-pointer usage
21f17aaa6 Modified Makefile and build_tools/build_detect_platform to compile on Linux s390x.
8800975fb Make DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels more robust
cd3fe675a Remove stale TODO
69c98f043 Reorder instance variables in backup test for proper destruction order
82f15fb15 Add test to make sure DropColumnFamily doesn't impact existing iterators
38201b359 Fix assert failure when DBImpl::SyncWAL() conflicts with log rolling
2568985ab IOStatsContext::ToString() add option to exclude zero counters
b04691665 Redo SyncPoints for flush while rolling test
291ae4c20 Revert "Revert "Fixed the bug when both whole_key_filtering and prefix_extractor are set.""
eef63ef80 Fixed CompactFiles() spuriously failing or corrupting DB
79ca039eb Relax the check condition of prefix_extractor in CheckOptionsCompatibility
4b1b4b8ae Merge pull request #1004 from yuslepukhin/child_attr
9ea2968d2 Implement ConsistentChildrenAttribute   by using default implementation for now as it works.
c7f1a8a46 Fix LITE build thread_local_test
0914f0ca5 Merge pull request #1003 from yuslepukhin/fix_mutexlock_pthread_build
d37d348da This addresses build issues on Windows  https://github.com/facebook/rocksdb/issues/1002
d825fc70d Use condition variable in log roller test
6b2a047df Fix SstFileManager uninitialized data member
a3db93c26 Remove the SyncPoint usage in the destructor of PosixEnv
df9ba6df6 Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage
3943d1678 Fix race conditions in auto-rolling logger
d733dd572 [build] Fix env_win.cc compiler errors
cf38e56f2 Fix broken appveyor build caused by D53991
351252b49 Merge pull request #998 from fengjian0106/master
133605249 fix ios build error
d08d50295 Fix transaction locking
730a422c3 Improve the documentation of LoadLatestOptions
a7b6f0748 Improve write_with_callback_test to sync WAL
5bcf952a8 Fix WriteImpl empty batch hanging issue
871cc5f98 fix build without gflags
c90d63a23 can_unlock set but not used
44371501f Fixed a segfault when compaction fails
2f084d39b Merge pull request #992 from jofusa/jdennison/options-typo-fix
7bd284c37 Separeate main from bench functionality to allow cusomizations
bd5f842bb fixes typo in options logging
1c868d684 Fix includes for env_test
545a19395 Add J to commit_prereq so comilation/execution happens in parallel
5bb7371ca [build] Evaluate test names only when db_test exists
6a2b4fcb8 Add flag to forcibly disable fallocate
92a9ccf1a Add a new compaction priority that picks file whose overlapping ratio is smallest
3dc3d1c14 Merge pull request #984 from petermattis/pmattis/comparator-iterate-upper-bound
239aaf2fc Use user_comparator when comparing against iterate_upper_bound.
908100399 Fixed a dependency issue of ThreadLocalPtr
337671b68 Add universal compaction benchmarks to run_flash_bench.sh
3a67bffaa Fix an ASAN error in transaction_test.cc
c5af85eca Fix a memory leak of Slice objects from org.rocksdb.WBWIRocksIterator#entry1
e84137c8a Remove unnessecary java.util.List expense in JNI
76e8beeeb Pass by pointer from/to Java from JNI not by object
0e7e6f6e4 Improve Javadoc
18eb56305 Improve the speed and synchronization around the construction of Java/JNI objects
2a04268be Temporarily disable unstable tests in memory_test.cc
08a78b6b4 Merge pull request #979 from facebook/update_licenses
21e95811d Updated all copyright headers to the new format.
59b3ee658 Env function for bulk metadata retrieval
4a8cbf4e3 Allows Get and MultiGet to read directly from SST files.
fe93bf9b5 Transaction::UndoGetForUpdate
2608219cc crash_test: cover concurrent memtable insert in default crash test
a76e9093f Fix LITE db_test build broken by previous commit
b1887c5dd Explictly fail when memtable doesn't support concurrent insert
8ed343877 Add option to run fillseq with WAL enabled in addition to WAL disabled
73a9b0f4b Update version to 4.5
6f71d3b68 Improve perf of Pessimistic Transaction expirations (and optimistic transactions)
8e6172bc5 Add BlockBasedTableOptions::index_block_restart_interval
34a40bf91 Add --allow_concurrent_memtable_write in stress test and run it in crash_test
73bf330c7 Merge pull request #973 from yuslepukhin/moveout_testcode
f7c0f4e3e perf_context.cc and iostats_context.cc use different output macro (fix unity build)
9656eab00 This partially addresses issue https://github.com/facebook/rocksdb/issues/935   testutil.cc and testharness.cc could not be moved out at this time   as they are used by 4 benchmarks in release builds.
14a322033 Remove references to files deleted in commit abb405227848581d3e6d2ba40d94dbc0a5513902
8445e5380 Add a mechanism to run all tests in sandcastle
461cec4e8 Merge pull request #972 from adamretter/wb-threads
9ab269ab3 Threaded tests for WriteBatch
bf767c641 Minor fix to makefile
2c1db5ea5 always invalidate sequential-insertion cache for concurrent skiplist adds
c12ff20ab Merge pull request #965 from koldat/jni_for_windows
a09ce4fcd Skip some of the non-critical tests in ./tools/run_flash_bench.sh
284aa613a Eliminate duplicated property constants
94be872ea Merge branch 'master' of github.com:facebook/rocksdb
0c2bd5cb4 Removing data race from expirable transactions
5fcd1ba30 disable kConcurrentSkipList multithreaded test
466c2c1bf Generate tags for *.c files
70c068c97 Merge pull request #960 from koldat/masterFixes2
a62c519bb RollLogFile tries to find non conflicting file until there is no conflict.
57a95a700 Making use of GetSystemTimePreciseAsFileTime dynamic - code review fixes
502d41f15 Making use of GetSystemTimePreciseAsFileTime dynamic to not break compatibility with Windows 7. The issue with rotated logs was fixed other way.
52153930d Adding support for Windows JNI build - fix Java unit test for release build of JNI DLL
e2972803a Adding support for Windows JNI build
9c2cf9479 Fix for --allow_concurrent_memtable_write with batching
ac3fa9a6f Travis CI to disable ROCKSDB_LITE tests
7b943da1b Merge pull request #967 from SherlockNoMad/ValueSize
b5750790e Merge pull request #968 from yuslepukhin/one_shot_buffer
1ad818295 Fix WriteBatchTest.ManyUpdates, WriteBatchTest.LargeKeyValue under clang
ad7ecca72 Add unit tests to verify large key/value
fdd70d149 Skip filters for last L0 file if hit-optimized
aa5e3b7c0 PerfContext::ToString() add option to exclude zero counters
36300fbbe Enable per-request buffer allocation in RandomAccessFile  This change impacts only non-buffered I/O on Windows.  Currently, there is a buffer per RandomAccessFile  instance that is protected by a lock. The reason we  maintain the buffer is non-buffered I/O requires an aligned  buffer to work.  XPerf traces demonstrate that we accumulate a considerable  wait time while waiting for that lock.  This change enables to set random access buffer size to zero  which would indicate a per request allocation.  We are expecting that allocation expense would be much less than  I/O costs plus wait time due to the fact that the memory heap  would tend to re-use page aligned allocations especially with the  use of Jemalloc.  This change does not affect buffer use as a read_ahead_buffer for  compaction purposes.
1d854fa3d Fixed the asan error on column_family_test
37159a644 Add histogram for value size per operation
3b2a1ddd2 Add options.base_background_compactions as a number of compaction threads for low compaction debt
6ee38bb15 Slowdown of writing to the last memtable should not override stopping
d6c838f1e Add SstFileManager (component tracking all SST file in DBs and control the deletion rate)
45768ade4 transaction allocation perf improvements
77926f93e Merge pull request #964 from benoitc/fix/pi2
03a5661a1 fix build for raspberry 2
4b50f1354 Should not skip bloom filter for L0 during the query.
eadd221d3 Merge pull request #959 from koldat/master
d209076fa Merge pull request #961 from wingify/master
26c618004 Add Wingify to USERS.md
4265f81e8 Remove util/auto_roll_logger.cc (it was moved to different directory)
d7f22b6d2 Fixing generated GenerateBuildVersion.vcxproj when one builds on different locale than english. The problem is that date and time CLI utilities generates different format so that REGEX in CMake does not work.
d20915d52 Disable stats about mutex duration by default
0c433cd1e Fix issue in Iterator::Seek when using Block based filter block with prefix_extractor
035857a31 Fix RocksDB lite build
77ef87ccb Update fbcode_config4.8.1.sh to use update_dependencies.sh
955ecf8b4 Fix an ASAN error in compact_files_test
b0afcdeea Fix bug in block based tables with full filter block and prefix_extractor
167bd8856 [directory includes cleanup] Finish removing util->db dependencies
acd7d5869 [directory includes cleanup] Remove util->db dependency for ThreadStatusUtil
46f9cd46a [directory includes cleanup] Move cross-function test points
22ecb752d Add valgrind to pre-commit sandcastle testing
b7ecf3d21 Fix intermittent hang in ColumnFamilyTest.FlushAndDropRaceCondition
38e1d7fea ldb to support --column_family option
da33dfe18 Parameterize DBTest.Randomized
fb9811ee9 Add a perf context level that doesn't measure time for mutex operations
f7ef1a613 Include rest of dependencies in dependencies.sh
3e9209a07 Updated GetProperty documentation
40911e0b3 Run unit tests in parallel to find failing tests
2fbc59a34 Disallow SstFileWriter from creating empty sst files
f53c95f81 Cosmetic fixes and comments for the reader
f1ed17010 Add tests to make sure new DB or ColumnFamily options are settable through string
f57596b0c Improvements to pre-commit
538eec066 Update fbcode_config.sh to use latest versions automatically
8019aa9b5 improve test for manifest write failure
bcd4ccbc3 Revert D7809
b0a15e7fb Mechanism to run CI jobs on local branch via commit_prereq
bb2888738 Cleanup property-related variable names
29289333d Add named constants for remaining properties
2c2b72218 Disable OptionsParserTest.BlockBasedTableOptionsAllFieldsSettable under CLANG
a300d9928 Added sandcastle pre-commit
202be23e4 Add test that verifies all options in BlockBasedTableOptions is settable through GetBlockBasedTableOptionsFromString()
eceb5cb1b Split db_test.cc (part 1: properties)
94918ae84 db_bench: explicitly clear buffer in compress benchmark
fdbff4239 Crash test to make kill decision for every kill point
39c3e94ff Merge pull request #954 from davidbernard/solaris_build
df7c2f3b5 As per google coding standard replace "using" in option_builder.cc and geodb_impl.cc
12809b44b Revert "Change notification email for travis"
34704d5c7 [easy] Fixed a crash in LogAndApply() when CF creation failed
791dbafa9 Merge pull request #953 from sselva/master
594a5ef02 Merge pull request #955 from bcbrock/ppc64-build
f423f05dc Simple changes to support builds for ppc64[le] consistent with X86
3f12e16f2 Make alloca.h optional
eaa563756 Change notification email for travis
d78c6b28c Changes for build on solaris
2e9fae3f2 Add Rakuten Marketing to USERS.md
83e1de92a move internal build to use zstd 0.4.5
aec10f734 Guard falloc.h inclusion to avoid build breaks
f7ebc2f34 Update HISTORY.mc for 4.4.0
addd9545f Merge pull request #947 from yuslepukhin/align_and_yield
ac50fd3a7 Align statistics   Use Yield macro to make it a little more portable between platforms.
b54d4dd43 tools/sst_dump_tool_imp.h not to depend on "util/testutil.h"
48a8667c3 Merge pull request #929 from warrenfalk/fix32
d9bca1e14 Reduce iterator deletion overhead
45d794068 Merge pull request #940 from yuslepukhin/fix_windows_build_signed_unsigned
20d7902df Fix compile error.   Use constructor style initialization instead of a cast for   simplicity.
e16438bb8 fixing build warning
b73fbbaf6 added --no_value option to ldb scan to dump key only
1477dcb37 Merge pull request #937 from petehunt/master
c7cb1076a Add Smyte to USERS.md
df7e3b622 Include <array> in table/plain_table_key_coding.h
235b162be Not scheduling more L1->L2 compaction if L0->L1 is pending with higher priority
9a8e3f73e plain table reader: non-mmap mode to keep two recent buffers
7ece10ecb DeleteFilesInRange: Mark files to be deleted as being compacted before applying change
94d9df248 fix an unused function compiler warning in crc32c in 32-bit mode
2f01e10fa use static_cast in crc32c instead of c-style cast
601f1306a fix shorten-64-to-32 warning in crc32c
f3fb39814 Fix BlockBasedTableTest.NoopTransformSeek failure
55b37efa1 fix a compile error on 32-bit
8c71eb5af Optimize DBIter::Prev() by reducing stack overhead
73c31377b Revert "Fixed the bug when both whole_key_filtering and prefix_extractor are set."
57605d7ef Fixed the bug when both whole_key_filtering and prefix_extractor are set.
6935eb24e Add ColumnFamilyHandle::GetDescriptor()
9760c842c fix valgrind failure in backupable_db_test
b1a3b4c0d Make ldb automagically determine the file type and use the correct dumping function
ba8344736 Merge pull request #923 from petermattis/pmattis/prefix-may-match
da032495d Optimize GetLatestSequenceForKey
260c29762 Fix index seeking in BlockTableReader::PrefixMayMatch.
e541dcc8f Fix issue #921
51adc5457 fix sporadic failure in fault_injection_test
a2422f053 fix potential test SleepingTask race condition
1627c4b1b Merge pull request #918 from mkurdej/fix/assertion-on-no-disk-space
92d0850f1 Fix failing assertion in logger on Windows when the disk is full.
7699439b7 Prevent the user from setting block_restart_interval to less than 1
4041903ec Enhance db_bench write rate limit
399343205 Add Airbnb and Pinterest to USERS.md
d74c9f0a5 DeleteFilesInRange: Clean job context if no files deleted
1dec5b8f5 Merge pull request #916 from warrenfalk/capi_huge_page_option
12fa27b4f Merge pull request #915 from warrenfalk/capi_full_bloom
0fde291ab expose memtable_prefix_bloom_huge_page_tlb_size option to C API
7e81dba5c Support creation of "full" format bloom filter from C API
bae5b0a1d Fix clang build
ac16663bd use -Werror=missing-field-initializers, to closer match MyRocks build
ab5a9a66d Merge pull request #911 from shuzhang1989/fix_envhdfs_virtual_func
eb5a13904 update posix env schedule call
a41f68ac2 fix inconsistency between env_hdfs and env
7238be090 Fix clang build in db_compaction_test
c9e2490bc Fix DynamicBloomTest.concurrent_with_perf to pass TSAN
63ddb783d Delete files in given key range
d8677a8d2 Upgrade internal CLANG version for FB-internal gcc 4.8.1
edf1cd497 Not generating "__attribute__((__unused__))" for padding fields if it is not CLANG
9eb4f9596 Merge pull request #907 from siying/master
22c0ed8a5 Disable Visual Studio Warning C4351
fcafac053 Fix memory leak in ColumnFamilyTest.WriteStall*
b99d4276f Fix java test buid broken by 7d87f02799bd0a8fd36df24fab5baa4968615c86
11672df19 Fix CLANG errors introduced by 7d87f02799bd0a8fd36df24fab5baa4968615c86
7fafd52dc Merge pull request #900 from shuzhang1989/hdfs_env_fix
2b7c810db more foramt
b79ccbd57 indent
7d87f0279 support for concurrent adds to memtable
5b2587b5c DBTest.HardLimit use special memtable
b4aa82366 format
4dfdd1d92 format
298ba27ae Merge pull request #846 from yuslepukhin/enble_c4244_lossofdata
7810aa802 Merge pull request #899 from zhipeng-jia/fix_clang_warning
4c5560d70 Merge pull request #895 from zhipeng-jia/develop
d43da8ae0 DBTest.DelayedWriteRate: fix assert of sign and unsign comparison
3280ae9a2 Fix warning in release
ec2664fef Fix clang compile error under Linux
4fd23fb13 add a factory method for creating hdfs env
9c176ef90 Update liblz4 to r131
15b890226 Change default options.delayed_write_rate
73b175a77 Fix clang warnings regarding unnecessary std::move
b9f77ba12 When slowdown is triggered, reduce the write rate
445d5b8c5 Fix clang build
e089db40f Skip bottom-level filter block caching when hit-optimized
06c05495e Merge pull request #898 from zhipeng-jia/fix_move_warning
aa515823b Fix clang warning
2ba03196d Merge pull request #897 from yuslepukhin/enable_status_move
dbb8260f7 Make Status moveable   Status is a class which is frequently returned by value from functions.   Making it movable avoids 99% of the copies automatically   on return by value.
2bf9b968c Fix lite_build
d005c66fa Report compaction reason in CompactionListener
728f944f0 Fix computation of size of last sub-compaction
8ac7fb837 Merge pull request #863 from zhangyybuaa/fix_hdfs_error
e53e8219a Merge pull request #894 from zhipeng-jia/develop
e0abec158 Sorting std::vector instead of using std::set
33e09c0e1 add call to install superversion and schedule work in enableautocompactions
22c6b50ee Merge pull request #893 from zhipeng-jia/develop
24c7dae13 Fix clang warning regarding implicit conversion
eff309867 Do not use timed_mutex in TransactionDB
97ea8afaa compaction assertion triggering test fix for sequence zeroing assertion trip
521da3abb Fix BlockBasedTableTest.BlockCacheLeak valgrind failure
a48382399 Fix use-after free in db_bench
bf8ffc1d6 Merge pull request #890 from zhipeng-jia/develop
131f7ddf6 fix typo: sr to picking_sr
c37729a6a db_bench: --soft_pending_compaction_bytes_limit should set options.soft_pending_compaction_bytes_limit
7b12ae97d Add signalall after removing item from manual_compaction deque
d72b31774 Slowdown when writing to the last write buffer
6b2a3ac92 Add documentation for unschedFunction
167fb919a ZSTD to use CompressionOptions.level
32ff05e97 Bump version to 4.4
aececc209 Introduce ReadOptions::pin_data (support zero copy for keys)
e6e505a4d Fix examples
aa29cc128 Improve examples/README.md
97265f5f1 Fix minor bugs in delete operator, snprintf, and size_t usage
b68dc0f83 Merge pull request #885 from yuslepukhin/fix_size_t_formatting
b6d19adcf Use port size_t formatting
963660eb5 Merge pull request #883 from zhipeng-jia/master
99ae549d3 Fix typo
636cd3c71 Clean up listener_test (reuse db_test_util)
030215bf0 Running manual compactions in parallel with other automatic or manual compactions in restricted cases
d26a4ea62 Merge pull request #882 from SherlockNoMad/BuildFix
768a61486 Fix appVeyor Build problem
aca403d2b Fix another rebase problems.
a6fbdd64e Fix rebase issues and new code warnings.
3fa68af31 Enable MS compiler warning c4244.   Mostly due to the fact that there are differences in sizes of int,long   on 64 bit systems vs GNU.
236fe21c9 Enable MS compiler warning c4244.   Mostly due to the fact that there are differences in sizes of int,long   on 64 bit systems vs GNU.
84f98792d Transaction::SetWriteOptions()
3bfd3d39a Use SST files for Transaction conflict detection
362d819a1 Improving parser
00d6edf6a Ensure the destruction order of PosixEnv and ThreadLocalPtr
64fa43843 Merge pull request #862 from ceph/wip-env
2074ddd62 env: add EnvMirror
a3ba5915c Correct a comment in include/rocksdb/cache.h
f0a8e5a2d Fixed the valgrind error in ColumnFamilyTest::CreateAndDropRace
9e4462906 Change SingleDelete to support conflict checking
c5af8bffb Merge pull request #879 from charsyam/feature/typos
c30b49954 fix typos in comments
56e77f096 Deprecate options.soft_rate_limit and add options.soft_pending_compaction_bytes_limit
d6e1035a1 A new compaction picking priority that optimizes for write amplification for random updates.
de6958b2e Merge pull request #877 from yuslepukhin/fix_unnecessary_type_truncation
0991cee6c Merge pull request #815 from SherlockNoMad/CounterFix
49957f9a9 Prefer integer arithmetics   The code had conversion to double then casting to size_t   and then casting uint32_t which caused compiler warning (VS15).
0836d265c Merge pull request #876 from warrenfalk/wf_win_master
c6fedf2bf Add compaction_iterator and delete_scheduler tests to Windows build
ac8e56f05 db_bench: in uncompress benchmark, get Snappy size from compressed stream
9c227923c Merge pull request #788 from OpenChannelSSD/to_fb_master2
fa3dbf203 Merge pull request #853 from Vaisman/enable_C4267_warning
ad6aaf4fa Merge pull request #848 from SherlockNoMad/db_bench
56bbecc31 Merge pull request #867 from SherlockNoMad/CacheFix
188170fb4 Updating HISTORY.md
758dbec7f Fix fb-only build for gcc 4.8.1
774b80e99 Resubmit the fix for a race condition in persisting options
afc84731f Include ldb_tools and sst_dump_tools libraries in shared library
e5c5f2381 Support marking snapshots for write-conflict checking - Take 2
ea1192355 Upgrade to ZSTD 0.4.2
b60cb88c7 Update examples/rocksdb_option_file_example.ini
3d8bb2c89 Fix valgrind failure in IncreaseUniversalCompactionNumLevels
7af91d425 Merge pull request #873 from yuslepukhin/make_vs15_build
1d63c3d61 Revert "Support marking snapshots for write-conflict checking"
78de0c922 Fix up VS 15 build.  Fix warnings  Take advantage of native snprintf on VS 15
ec704aafd Support marking snapshots for write-conflict checking
770dea932 Fix occasional failure of DBTest.DynamicCompactionOptions
ebc2d490d Split histogram per OperationType in db_bench
f307036bd Revert "Fix a race condition in persisting options"
2fa3ed518 Fix a race condition in persisting options
f276c3a82 Fix valgrind failures in 3 tests in db_compaction_test due to new skiplist changes
291088ae4 Fix undeterministic failure of ColumnFamilyTest.DifferentWriteBufferSizes
3c2b995fb Merge branch 'master' of https://github.com/facebook/rocksdb into CacheFix
355fa9436 EstimatedNumKeys Counter Inaccurate
b2863017b Move posix threads into a library
3a98a7ae7 Replace malloc with new for LRU Cache Handle
a9ca9107b Fix db_universal_compaction_test
d3bb572da Build break fix.
b28b7c6dd Added callback notification when a snapshot is created
e8180f990 added public api to schedule flush/compaction, code to prevent race with db::open
19b1201b2 Merge pull request #865 from yuslepukhin/fix_db_table_properties_test
e0de7ef87 Avoid empty ranges vector with subsequent zero element access
a330f0b3b Fix incorrect merge in db/db_compaction_test.cc
bd7a49d44 Make DBCompactionTestWithParam::CompactionTrigger more deterministic
be006d288 fix LinkFile() undefined reference error
4687ced5d fix ToString() not declared error
bcd7bd122 Relax verification condition of DBTest.SuggestCompactRangeTest
f9103d9a3 DBTest.DynamicCompactionOptions: More deterministic and readable
0ad68518b Fix DBCompactionTestWithParam.CompactionTrigger in non-jemalloc build.
459c7fba3 Revert previous behavior of internal_key_skipped_count
481f9edb1 Fix CLANG build
d7421c22f Fixed some typos in the comments of rocksdb options file example
ef8ed3681 Fix DBTest.SuggestCompactRangeTest for disable jemalloc case
db320b1b8 DB to only flush the column family with the largest memtable while option.db_write_buffer_size is hit
4a009f917 Merge pull request #860 from SherlockNoMad/BuildFix
b4efaebff Fix ms version Appveyor build error
d27ea4c9e Initialize options.row_cache
72930485b Fix clang build
6bbfa1874 BackupDB to have a mode to use file size in file name
f3ea00bc8 Merge pull request #856 from ceph/wip-env
4cedd6b03 EnvWrapper: add ReuseWritableFile
33e0c9382 Reduce extra key comparision in DBIter::Next()
9a9d4759b InlineSkipList part 3/3 - new skiplist type that colocates key and node
520172954 InlineSkipList - part 2/3
78812ec6b InlineSkipList - part 1/3
ffb466da4 Merge pull request #855 from yuslepukhin/enable_3rdparty_override
10d257d64 Enable override to 3rd party linkage
41b32c605 Enable C4267 warning
c5b467306 Fix race condition that causes valgrind failures
efb01a055 Merge pull request #850 from yuslepukhin/enable_2015_build
81be49c75 Have a way for compaction filter to ignore snapshots
047bd22aa Build on Visual Studio 2015 Update 1
88e052772 Reduce moving memory in LDB::ScanCommand
890f44f46 Merge pull request #844 from yuslepukhin/enable_C4804_unsafe_bool
9d0b8f19d plain table reader: avoid re-read the same position for index and data in non-mmap mode
89bacb7e7 Enable MS Warning C4804 : unsafe use of type 'bool' in operation
d5239f870 build_tools/fbcode_config4.8.1.sh: upgrade versions of some dependencies
c4ebb66d6 Not to build forward_iterator_bench now
c342549d0 Merge pull request #841 from yuslepukhin/fix_test_rerun_logic
7cb1293b6 Fix log names when scheduling runs and reruns
51fce92e1 "ldb compact" should force bottommost level compaction
f83164120 Merge pull request #837 from yuslepukhin/rerun_concurrency_value
4159ab816 Merge pull request #839 from SherlockNoMad/memtableOption
6170fec25 Fix build broken by previous commit of "option helper refactor"
3a6643c2f Merge pull request #805 from SherlockNoMad/OptionHelperFix
189b3e03d Fix uninitilizeded SpecialEnv::time_elapse_only_sleep_
d5540e18e DBTest.MergeTestTime to only use fake time to be determinstic
bd7be035e Support Memtable Factory Parse in option_helper.cc
94e39e236 Exclude DBTest.FileCreationRandomFailure as a long running test   Increase concurrency to 18   Fix exclusion but in the ps script
4189c0f9a Fix Java Makefile
605a24d94 Block forward_iterator_bench under MAC and Windows
2a0510c9f Failed tests must be rerun with concurrency 1
9b8c9be0b Fix forward_iterator allocation of vector.
5cbb7e43e DBTest.MergeTestTime: relax counter upper bound verification
52e04b3d0 Merge pull request #833 from yuslepukhin/fix_win_build_after_lint
314f62194 Remove headers from the cc since they are in the module's header.
472c74006 Add necessary headers after cpplint rearranged includes
9bc9c93bd Move to version 4.3
3381e2c3e Handle multiple calls to DBImpl::PauseBackgroundWork() and DBImpl::ContinueBackgroundWork()
65a042921 Merge pull request #831 from yuslepukhin/remove_forward_iter_bench_win
ca5566d20 Fix clang build
4175472ad Merge pull request #832 from yuslepukhin/fix_forward_iter_outofbounds
cb9459f85 Fix empty vector write in ForwardIterator
a163cc2d5 Lint everything
8f01f2541 Remove forward_iter_bench from Win build.
dac5b248b UniversalCompactionPicker::PickCompaction(): avoid to form compactions if there is no file
d06b63e99 Fix Rocksdb lite build failure in forward_iterator_bench
7824444bf Reuse file iterators in tailing iterator when memtable is flushed
2ae4d7d70 Make sure that CompactFiles does not run two parallel Level 0 compactions
d781da816 Add CheckOptionsCompatibility() API to options_util
2391b459b Merge pull request #824 from yuslepukhin/try_ci_tests_on_daily
2ab3e2df2 Fix a build break so tests can run
247c49a40 Merge branch 'master' into try_ci_tests_on_daily
935d1495c Run tests imporvements   Add sequential rerun for any failed tests. Add env_test case.   Limit concurrency   Allow to specify individual tests   Take $Limit into account when displaying number of tests
5ac16300b Fixed valgrind error in options_util_test
6ce42dd07 Don't merge WriteBatch-es if WAL is disabled
56245ddcf Fixed DBCompactionTest.SkipStatsUpdateTest
e11f676e3 Add OptionsUtil::LoadOptionsFromFile() API
e78389b55 Fixed build failure of RocksDBLite test on options_file_test.cc
e114f0abb Enable RocksDB to persist Options file.
7ed2c3e45 Merge pull request #823 from yuslepukhin/fix_off_t_type
7f59e33b1 Make CI build debug/optimized
ae2dfe404 Try running db_test during integration build
720af2269 Merge branch 'fix_off_t_type' of https://github.com/yuslepukhin/rocksdb into fix_off_t_type
5270b33bd Make use of portable `uint64_t` type to make possible file access   in 64-bit.
631863c63 track WriteBatch contents
505accda3 remove constexpr from util/random.h for MSVC compat
5421c9728 Make use of portable `uint64_t` type to make possible file access   in 64-bit.
b81b43098 Switch to thread-local random for skiplist
75a8bad2a Merge pull request #821 from yuslepukhin/continue_windows_warnings
986230b8c Revert "Fix TSAN build for fbcode"
f3ca28ab0 Correct the comment of GetApproximateMemoryUsageByType
838676c17 Revert "Adding new table properties"
7c86d5049 Enable C4305 'identifier' : truncation from 'type1' to 'type2'
85a2ce9c1 Enable C4702 unreachable code
62aa1b1b7 Enable C4200 warning   nonstandard extension used : zero-sized array in struct/union
5b9ce1a32 Merge pull request #820 from yuslepukhin/enable_compiler_warnings
20f57b171 Enable Windows warnings C4307 C4309 C4512 C4701   Enable C4307 'operator' : integral constant overflow   Longs and ints on Windows are 32-bit hence the overflow   Enable C4309 'conversion' : truncation of constant value   Enable C4512 'class' : assignment operator could not be generated   Enable C4701 Potentially uninitialized local variable 'name' used
8be568a9c Adding new table properties
2b42000f4 incorrect batch group size computation for write throttling
c745f1d2c Fix TSAN build for fbcode
fe789c5f2 Document SingleDelete
e89e5b253 Merge pull request #818 from yuslepukhin/improve_test_concurrency
ae7940b62 Fix regression failure in PrefixTest.PrefixValid
3277d172b Improve concurrency when running tests   PowerShell seems to have a hard time when a flood of async tasks is   scheduled at the same time.   I speculated that WaitForMultipleObjects() in Windows can only take up   to 64 process handles and if you want to handle more than you should write   some additional code which can be sub-optimal. I.e to implement Wait-Job -Any.   I decided to test that suggestion and introduced a $Concurrency parameter with a default value of 62.   So in the new version the script fires up up to $Concurrency value   and wait for anything to complete before starting any more processes.   This improved matters greatly. Individual tests against ramdrive now   run in 8 minutes and all of the 200+ db_tests run in 9 minutes with concurrency   values of 8-16. About 48 is required to load a CPU on my box running against HD   but that does not improve running times much.
c8e01ef98 Delete test iterators
9d50afc3b Prefix-based iterating only shows keys in prefix
14c6e1a04 Add write_stress to RocksDB Legocastle runs
db3f5e494 Update HISTORY.md
042fb053f Fix clang
2419f435a Merge pull request #816 from SherlockNoMad/GeoDBTestFix
2e4540991 Fix appveyor build failure
183cadfc8 Add OptionsSanityCheckLevel
dba5e0074 Fixed the compile error in RocksDBLite in memory_test.cc
7d7ee2b65 Add Memory Insight support to utilities
3ecbab004 Add GetAggregatedIntProperty(): returns the aggregated value from all CFs
93a966722 Merge branch 'master' of github.com:facebook/rocksdb
c9aef3c41 Add RocksDb/GeoDb Iterator interface
f31442fb5 Merge pull request #803 from SherlockNoMad/SkipFlush
dcc898b02 Merge pull request #812 from yuslepukhin/fix_windows_warnings
df7ed91ef Fix white space at end of line
a0163c068 Do not disable compiler warnings:   C4101 'identifier' : unreferenced local variable   C4189 'identifier' : local variable is initialized but not referenced   C4100 'identifier' : unreferenced formal parameter   C4296 'operator' : expression is always false
279c8e0cd Merge pull request #811 from OverlordQ/unused-variable-warning
affd83369 Fix introduced in 2ab7065 was reverted by 18285c1.
db68a2c09 Merge pull request #806 from yuslepukhin/signed_unsigned_warning
ccc8c10c0 Move skip_table_builder_flush to BlockBasedTableOption
eaaf081d1 Do not suppress C4018 'expression' : signed/unsigned mismatch   The code compiles cleanly for the most part. Fix db_test.   Move debug file to testutil library.
ff4499e29 Update DB::AddFile() to have less restrictions
9ac88c855 Merge branch 'master' of https://github.com/facebook/rocksdb into OptionTestFix
84992d647 Option Helper Refactoring
11c71a365 db_bench: --compaction_pri default should be rocksdb::Options().compaction_pri
335e4ce8c options_test: fix a bug of assertion
66a3a87ab Merge pull request #797 from SherlockNoMad/optionHelper
550af4ee6 Fix Travis Build Error
a6dd0831d Add Option to Skip Flushing in TableBuilder
2872e0c8c Clean and expose CreateLoggerFromOptions
296c3a1f9 "make format" in some recent commits
6388e7f4e Merge pull request #798 from yuslepukhin/readahead_buffermanagement
f4cbb90c4 Merge pull request #799 from yuslepukhin/fix_random_generator_compile
1277a48f1 Fix 80 character limit issue.
ee2c3236d Fix compilation problem on Windows.   char is not a valid template parameter for std::uniform_int_distribution   according to the standard. Replacing with int should be just fine.
b69b9b624 Support PlainTableOption in option_helper
c97667d9f Fix RocksDB lite build for write_stress
0d720dfc1 Use the correct variable when fetching table properties.
4b66d9534 Write stress test
47414c6cd Move include/posix/io_posix.h to util/io_posix.h
2889df84c Revert "Avoid to reply on ROCKSDB_FALLOCATE_PRESENT in include/posix/io_posix.h"
28c8758a3 Merge pull request #795 from yuslepukhin/fix_mocktable_id
5c8f2ee78 Fix MockTable ID storage   On Windows two tests fail that use MockTable:   flush_job_test and compaction_job_test with the following message:   compaction_job_test_je.exe : Assertion failed: result.size() == 4,   file c:\dev\rocksdb\rocksdb\table\mock_table.cc, line 110
72d6e758b Fix WritableFileWriter::Append() return
d0a18c284 Merge pull request #786 from aloukissas/unused_param
c37223c08 Avoid to reply on ROCKSDB_FALLOCATE_PRESENT in include/posix/io_posix.h
6fbc4f9f3 Implement smart buffer management.   introduce a new DBOption random_access_max_buffer_size to limit   the size of the random access buffer used for unbuffered access.   Implement read ahead buffering when enabled.   To that effect propagate compaction_readahead_size and the new option   to the env options to make it available for the implementation.   Add Hint() override so SetupForCompaction() call would call Hint()   readahead can now be setup from both Hint() and EnableReadAhead()   Add new option random_access_max_buffer_size support   db_bench, options_helper to make it string parsable   and the unit test.
d6219e4d9 Mac build break caused by include/posix/io_posix.h not declearing errno,
beb69d451 Merge pull request #765 from PraveenSinghRao/wal_filter
ab0f3b964 crash_test to trigger some less frequent crash point more frequently
7beb743cf Merge pull request #778 from Vaisman/master
4ce117c4d Merge branch 'master' into wal_filter
32cdec634 Fail recovery if filter provides more records than original and corresponding unit-test, fix naming conventions
44d4057d7 Avoid some includes in io_posix.h
2adad23a1 Fix unused parameter warnings.
b0980ff74 Fix unused parameter warnings.
bc898c5f8 Fix unused parameter warnings.
138876a62 Merge pull request #746 from ceph/wip-recycle
581f20fd8 Add LITE tests to Legocastle
3d56d868c Merge remote-tracking branch 'upstream/master'
d69111114 include/posix/io_posix.h should have a once declartion
a6962edf8 Merge pull request #783 from yuslepukhin/remove_test_conditional_compilation
3c750b59a No need to #ifdef test only code on windows
8c11c5dee Merge pull request #768 from OpenChannelSSD/to_fb_master2
6e6dd5f6f Split posix storage backend into Env and library
980a82ee2 Fix a bug in GetApproximateSizes
d0d13ebf6 fix bug in db_crashtest.py
01a41af0a Merge remote-tracking branch 'upstream/master'
5678c05d8 Use DEBUG_LEVEL=0 in make release and make clean
ac25fe6b9 Merge pull request #779 from yuslepukhin/optimize_windows_build
e154ee086 Do not build test only code and unit tests in Release builds   Test code errors are currently blocking Windows Release builew   We do not want spend time building in Release what we can not run   We want to eliminate a source of most frequent errors when people   check-in test only code which can not be built in Release.   This feature will work only if you invoke msbuild against rocksdb.sln   Invoking it against ALL_BUILD target will attempt to build everything.
cd3286fae Error while cmake by building from zip-archive
e3d4e1407 DBCompactionTestWithParam.ManualCompaction to verify block cache is not filled in manual compaction
033c6f1ad T7916298, bug fix
7717ad1af Adding artifacts to stress_crash CI job
0bf656b90 Don't spew warnings when flint doesn't exist
6d6776f6b Log more information for the add file with overlapping range failure
7951b9b07 make field order match initialization order
90228bb08 Merge pull request #771 from maximecaron/patch-1
2938c5c13 merge upstream changes
e3b1d23d3 Bump version to 4.2
a7b2bedfb log_{reader,write}: recyclable record format
4e07c99a9 Fix iOS build
0c59691dd Handle multiple batches in single log record - allow app to return a new batch + allow app to return corrupted record status
32c291e3c Merge branch 'master' of github.com:facebook/rocksdb into T7916298
4575de5b9 #7916298: merge tools/db_crashtest2.py into tools/db_crashtest.py
5c727de6a Merge pull request #777 from yuslepukhin/fix_win_build_uint
cfaa33f9a Update transaction iterator documentation
2f680ed09 Make index same type as auto deduced uint32_t
09f853550 uint is a not a datatype on windows.
ec1f8354a Fix the default assignment of DEBUG_LEVEL in Makefile
f18acd887 Fixed the clang compilation failure
4104e9bb6 log_reader: introduce kBadHeader; drop wal mode from ReadPhysicalRecord
9c33f64d1 log_reader: pass in WALRecoveryMode instead of bool report_eof_inconsistency
718805210 db_test_util: add recycle_log_files to set of tested options
3ac13c99d log_reader: pass log_number and optional info_log to ctor
5830c699f log_writer: pass log number and whether recycling is enabled to ctor
666376150 db_impl: recycle log files
d666225a0 db_impl: disable recycle_log_files if WAL archive is enabled
543c12ab0 options: add recycle_log_file_num option
1bcafb62f env: add ReuseWritableFile
e1a09a770 Implementation for GetPropertiesOfTablesInRange
ad471453e Allow GetProperty to report the number of currently running flushes / compactions.
277dea78f Add more kill points
a98fbacfa Moving memtable related files from util to a new directory memtable
8f143e03f Add ClearSnapshot()
f9ba79ecd crash_test to trigger fail points other than file appending more frequently
2f2de338c Run ROCKSDB_LITE tests in travis
680156ca6 crash_test to run with data sync on
e1a5ff857 Allow users to disable some kill points in db_stress
d306a7ea8 Merge pull request #773 from yuslepukhin/update_requirements
bb64d6da4 Disabling TSAN crash test
8c2fe68fd Update 4 is required for building with MS Visual Studio 13
952ad994a Fix db_test under ROCKSDB_LITE
6d730b4ae Block tests under ROCKSDB_LITE
5eee1ef2d Merge pull request #770 from Vaisman/master
63e507c59 Move ldb and sst_dump from utils to tools.
dae49e829 Make DBTest.ReadLatencyHistogramByLevel more robust
92060b215 Fix build error using Visual Studio 12
9f7413502 Error while cmake by building from zip-archive
b81b2ec25 Fix benchmarks under ROCKSDB_LITE
e587dbe03 Move manual_compaction_test.cc from util to db
666fb5df4 Remove DefaultCompactionFilterFactory.
d662b8dab Merge pull request #766 from PraveenSinghRao/lockfix
f55d3009c Make db_test_util compile under ROCKSDB_LITE
29a47cd2b Include the time unit in the comment of perf_context timers
2b925ccb4 Correct the comments in db/internal_stats.h
35ad531be Seperate InternalIterator from Iterator
91c041e57 move debug variable under ifndef
198ed5898 Merge pull request #760 from jwlent55/use-static-library-header-files
cc4d13e0a Put wal_filter under #ifndef ROCKSDB_LITE
385b41600 Merge pull request #764 from dmittendorf/fix-java-static-packaging
7062d0ea6 Make perf_context.db_mutex_lock_nanos and db_condition_wait_nanos only measures DB Mutex
1fe78a407 Fix tests failing in ROCKSDB_LITE
a6efefef7 Fix format specifiers
f7b2a7b40 Fix format specifiers
7f58ff7c3 Remove db_impl_debug from release build
1ddd91cd2 Fixed packaging of java crossbuild jar by forcing all compiled binaries to be output to the java/target directory.  The uber crossbuild jar is then assembled within the java/target directory.
eb2417855 merge from master
c64ae05b1 Move TEST_NewInternalIterator to NewInternalIterator
59a0c219b Adding log filter to inspect and filter log records on recovery
0be50ed12 Merge pull request #763 from PraveenSinghRao/lockfix
a1d37602a Fixing mutex to not use unique_lock
9e819d096 Modify the way java static builds are done so that: 1) There is no need to download and install the compression libraries twice just to get access to their header files during the compile phase. 2) Ensure that the compression library headers files used during the compile phase are the same ones used to build the static library that is linked into the library.
f1fdf5205 Clean up dependency: Move db_test_util.* to db directory
237994409 Fixed an incorrect replace of const value in util/options_helper.cc
0bb8ea56b [RocksDB Options File] Add TableOptions section and support BlockBasedTable
4a7970d75 Modify the way java static builds are done so that: 1) There is no need to download and install the compression libraries twice just to get access to their header files during the compile phase. 2) Ensure that the compression library headers files used during the compile phase are the same ones used to build the static library that is linked into the library.
c4366165e Merge pull request #759 from jwlent55/statically-load-compression-libraries
fa4b5b3db Fix for the travis build caused by my previous commit
3d07b815f Passing table properties to compaction callback
64546af83 Adding parser to CI jobs
def74f876 Deferred snapshot creation in transactions
c5f3707d4 DisableIndexing() for Transactions
776bd8d5e Pass column family ID to table property collector
5a7222782 Ensure that the compression libraries are statically linked into dynamic libraries included in the Java jar.  Also build the linux libraries using the portable flag to fix a problem with the linux32 build and improve the general portability of the RocksDB dynamic libraries. ==> linux32: util/crc32c.cc:318:39: error: ‘_mm_crc32_u64’ was not declared in this scope
e61d9c148 Make DBTest.AggregatedTableProperties more deterministic
b77eb16ab New Manifest format to allow customized fields in NewFile.
6732a5765 Merge pull request #756 from viveknelamangala/master
a52888ed0 Install snappy headers to standard locations using yum, so that build_tools/build_detect_platform sets -DSNAPPY flag to g++ . Current jars of rocksdb do no have snappy compression avaliable .
831101b5f Make it harder for users to run debug builds in production
000836a88 CompactionFilter::Context to contain column family ID
3a0bf873b Change RocksDB version to 4.1
77318ee14 Enable crash CI jobs
9803e0d81 compaction_filter.h cleanup
51fa7ecec Bytes read/written from cache statistics
f925208ff Create Makefile target unity_test
a065cdb38 bloom hit/miss stats for SST and memtable
40cdf797d Fix compile error on platforms without fallocate()
77e4ad7ce Fix compile failure on Travis
4049bcde3 Added boolean variable to guard fallocate() calls
aadf49fe6 Travis shouldn't fail when clang-format suggests improvements
d80ce7f99 Compaction filter on merge operands
726d9ce1a Disabling unity
026750265 Support for LevelDB SST with .ldb suffix
5855cdb6d Merge pull request #750 from yuslepukhin/fixup_build_options
7bbe10c01 Merge pull request #751 from yuslepukhin/return_noerror
e95b703b7 Mmap reads should not return error if reading past file
25c58a204 Add shared_linked DEBUG flag, remove port from among the include directories.
60b1c0a70 Fix to CI job definition
9babaeed1 Update dump_tool and undump_tool to accept Options
eb5b637fb Fix condition for bottommost level
9eaff629e Make corruption_test more robust
bf19dbff4 Fix valgrind - Initialize done variable
5c7bf56d3 [RocksDB Options] Support more options in RocksDBOptionParser for sanity check.
115427ef6 Add APIs PauseBackgroundWork() and ContinueBackgroundWork()
a39897369 Adding features to CI job description
a47bf325c Merge pull request #748 from yuslepukhin/improve_test_runs
65324a16d Improve test running scripts   Introduce proper command line arguments so we can control the script   Add appveyor support   Add an ability to run all other (non db_tests) test executables in parallel   Use .NET HashSet instead of empty valued hashtable to improve the looks   TODO: Some of the tests do not use GTests and need to improve log parsing
7e4ee4231 Merge pull request #743 from edsrzf/amalgamation
7a23e4d8c New amalgamation target
e9a6808c4 Merge pull request #745 from yuslepukhin/test_appveyor_baseline
9320ffd67 Improve CI build and fix Windows build breakage   Is there a way to enforce CMake additions for internal changes that seem to come   w/o a PR?
03b08ba9a Return MergeInProgress when fetching from transactions or WBWI with overwrite_key
c29af48d3 Add max_file_opening_threads to db_bench
da1cf8a9b Add a missing check for deprecated options in options_helper.cc
5a51fa907 Fix accidental object copy in transactions
1e73b11af Better handling of deprecated options in RocksDBOptionsParser
a8b8295d1 Fixed a compile warning in options_test.cc under clang
74b100ac1 RocksDB Options file format and its serialization / deserialization.
75134f756 Merge pull request #741 from yuslepukhin/test_appveyor_baseline
2e7506d82 Improve CI build and build switches   Add an optimized build config switch for faster test runs   Change compiler options to introduce more opitmizations and be more inline with MS internal switches.   Make appveyor build to utilize all the avaiable cores on the VM (parallel)   Introduce new appveyor configuration for daily test runs as it would take too long   to run db_test after each checkin even in paralell.   With some exclusions we make it in 38 minutes. We currently fail to install ramdisk during the build.   Add a powershell script to faicilitate paralell run for db_test cases.
1eff1834b Remove non-existing functions. Closes #680
16d1ba700 Clear SyncPoint Trace in DeleteSchedulerTests
30f74fa96 Make CompactionJobStatsTest.UniversalCompactionTest more robust
afe0dc539 SingleDelete support for Transactions
a263002a3 Fixed a tsan warning in db_stress.cc
e4861e7d6 Fixed a compile error in util/arena.h
0fdb4f168 Fixed a compile warning in util/arena.cc when hugetlb is not supported.
94ac8826c Merge pull request #737 from mlin/readonly-syncwal
60fa9cf0b Override DBImplReadOnly::SyncWAL() to return NotSupported. Previously, calling it caused program abort.
7df348b40 Minor fix to CI job definition
dac3f22b7 Fix the test failure
63e0f8679 Fixed a bug which causes rocksdb.flush.write.bytes stat is always zero
7ee445dd6 Fix the compile warning
174e2be5f Merge pull request #735 from jsteemann/fix-potential-leak-in-allocate
aa58958d3 prevent potential memleaks in Arena::Allocate*()
25fd743d7 Fix SingleDelete support in WriteBatchWithIndex
b6aa3f962 Fixed a memory leak issue in DBTest.UnremovableSingleDelete
7b7b5d9f1 [minor] Reuse SleepingBackgroundTask
e01f32cc3 Parameterizing email id
c58bac701 Fix valgrind failure due to memory leaks
a70d08ec0 Fix the bug of using freed memory introduced by recent plain table reader patch
628216fc1 Simplifying valgrind testing
4805fa0ea Remove ldb HexToString method's usage of sscanf
f03b5c987 Add experimental DB::AddFile() to plug sst files into empty DB
3fdb6e523 Fixed old lint errors in db/filename.cc
b349d2278 Fixed old lint errors in db/filename.h
df34aea33 PlainTableReader to support non-mmap mode
d746eaad5 RandomAccessFileReader should not inherit RandomAccessFile
03dd8f3ca Fixing punit job description
d0c31641d Internal stats WAL file synced to match meaning of the stats of the same name
48b4497f7 Merge pull request #730 from yuslepukhin/fix_write_batch_win_const_expr
a6c22e3e4 Disabling parallel test CI job
489a3e95d Re-work to support size_t max constant for 32/64-bit.
ff57c6511 [RocksJava] Fix test failure of InfoLogLevelTest
f1b9f804e Add a mode to always pick the oldest file to compact for each level
5e8f0a66d Use port::constant for std::muneric_limtis<>::max()
2754ec999 Fix Windows constexpr issue and '#ifdef' column_family_test in Release.
dd2e1eeb3 Disabling log running jobs
199744f4c Merge pull request #728 from jsteemann/fix-missing-include-header
0e65693f1 Merge pull request #727 from jsteemann/micro-optimization
4d6eb52d1 Fix to CI jobs.
3bcc072d2 Added more CI jobs
669b892f9 add missing header required for std::function
624ef456d fixed formatting. thanks @4tXJ7f for pointing me at `make format`
bbb18c827 removed unused variable of type Status, fixed indentation
470483335 pass input string to WriteBatch() by const reference
5ec129971 key_ cannot become nullptr, so no check is needed for that
834b12a8d made Size() function const because it does not modify data
1fc16cb2e Fix clang-format on Travis
e244bdf39 Merge pull request #725 from adamretter/fail-failed-java-test
7d937a090 Exit with non-zero status if one or more Java tests fail
1b598213a Check formatting in Travis
c7fba8029 Fix non-deterministic failure in backupable_db_test
014fd55ad Support for SingleDelete()
f35560d00 Merge pull request #723 from jsteemann/fix-typos
f8b770a94 fixed typos
51e1c1125 Do not flag error if file to be deleted does not exist
a5e312a7a Improving condition for bottommost level during compaction
9aca7cd6d DB::Open() to flush info log after printing DB pointer
cecd903ad Fix gflags build in Travis script
16934d495 Fix wrong constants in db_test_util
2e8e7eb39 Fix the verbosity issue in Java makefile
f21c7415a Change the log level of DB start-up log from Warn to Header.
9566342d2 Build gflags from source for Travis
3ebf11ed1 Adding the increment for a counter for a number of WAL syncs
81a61d75d Skipped tests shouldn't be failures [part 2]
1b7ea8ce8 Skipped tests shouldn't be failures
5ba3297d0 Add compaction time to log output
31a27a360 Callback for informing backup downloading added
d93a9f2aa [travis CI] Run ulimit -n 2000
2b683d497 Add DBOption.max_subcompaction to option dump
0e50a3fcc Merge issue with D46773
a7e80379b LogAndApply() should fail if the column family has been dropped
2819a1db3 Minor fix to CI job definition
df22e2fb7 Relax memory order for faster tickers
488607317 Adding Slice::difference_offset() function
5ce63e30e Merge pull request #720 from AMDmi3/fix-constant-overflow
925babc76 Merge pull request #721 from AMDmi3/printf-size_t
f171faa24 Fix printf format for size_t
4b0b0201c Fix `integer overflow in expression' error
f3170b6f6 DBImpl::FindObsoleteFiles() shouldn't release mutex between getting min_pending_output and scanning files
e467bf0de Fix valgrind error
7cb314b9e Skip some tests in ROCKSD_LITE
0bfe0573e Add gflags dependency to Travis script
5de807ac1 Add options.hard_pending_compaction_bytes_limit to stop writes if compaction lagging behind
7143242d1 Fix compaction_job_stats under ROCKSDB_LITE
592f6bf78 Merge pull request #716 from yuslepukhin/refactor_file_reader_writer_win
e2d6011f4 Minor fix to CI job definition
ad0d70ca1 Relax asserts in arena_test
03ddce9a0 Add counters for L0 stall while L0-L1 compaction is taking place
a3fc49bfd Transactions: Release Locks when rolling back to a savepoint
ddc8b4499 Address code review comments both GH and internal  Fix compilation issues on GCC/CLANG  Address Windows Release test build issues due to Sync
9f3a66a93 Improvements to CI jobs
7db1471cc Minor fix to CI job
c67d20689 Fixed arena_test failure due to malloc_usable_size()
34cedaff6 Initialize variable to avoid warning
aeb461268 Add counters for seek/next/prev
45e9e4f0b Refactor NewTableReader to accept TableReaderOptions
ddb950f83 Fixed bug in compaction iterator
30e82d5c4 Refactor to support file_reader_writer on Windows.   Summary. A change https://reviews.facebook.net/differential/diff/224721/   Has attempted to move common functionality out of platform dependent   code to a new facility called file_reader_writer.   This includes:   - perf counters   - Buffering   - RateLimiting
af7cdbf64 Run full test suite in Travis
c25f6a85b Removed __unused__ attribute
6db0a939d Fix DBCompactionTest failure with parallel L0-L1 compactions
8aa1f1519 Refactored common code of Builder/CompactionJob out into a CompactionIterator
41bce0586 CI job improvements
95ffc5d2b Correct ASSERT_OK() in ReadDroppedColumnFamily
3c37b3ccc Determine boundaries of subcompactions
112664408 Relaxing consistency detection to include errors while inserting to memtable as WAL recovery error.
abc7f5fdb Make DBTest.ReadLatencyHistogramByLevel more robust
ac9bcb55c Set max_open_files based on ulimit
4cbd2f9aa Merge pull request #714 from facebook/travisformac
d0df54d1f Run travis tests on OS X
a55e5a52a Merge pull request #711 from facebook/testtravis
2b676d5bb Upgrade travis to new architecture
c66d53fee Fixed minor issue in CompressionTypeSupported()
f3f2032c4 Release RocksDB 4.0.0
44b6e99e1 update max_write_buffer_number_to_maintain docblock
aa6eed0c1 Transaction stats
25dbc579f Update HISTORY file for transactions
52386a1e7 Minor fix to sandcastle jobs commands
b5b2b75e5 better tuning of arena block size
342ba8089 Make DBTest.OptimizeFiltersForHits more deterministic
e17e92ea1 Relaxed assert in forward iterator
5e94f68f3 TransactionDB Custom Locking API
0ccf2db38 Fixed broken build due to format specifier
6bdc484fd Added Equal method to Comparator interface
7a31960ee Tests for ManifestDumpCommand and ListColumnFamiliesCommand
778cf4449 Adding email notification.
3a0df7f16 Fixed comparison in ForwardIterator when computing hint for GetNextLevelIndex()
91f3c9079 Fix case when forward iterator misses a new update
ff1953c89 Merge pull request #707 from dkorolev/master
7b463e657 Fixed a typo in INSTALL.md
d9f42aa60 Adding a verifyBackup method to BackupEngine
50dc5f0c5 Replace BackupRateLimiter with GenericRateLimiter
20ef64cae Moving jobs to use gcc-4.9
0f1aab6c1 Add SetLockTimeout for Transactions
14456aea5 Fix compile
76f286cc8 Optimize bloom filter cache misses
0e6e5472e Fixed a compile warning in rocksjni/loggerjnicallback.cc
b8a962d4f Adding commands for few more CI jobs.
8a2d59a35 Add Cloudera's blog post to USERS.md
3c9cef1ee Unified maps with Comparator for sorting, other cleanup
3e0a672c5 Bug fix: table readers created by TableCache::Get() doesn't have latency histogram reported
b42cd6bed Remove the need for LATEST_BACKUP in BackupEngine
0f763db20 Merge pull request #705 from yuslepukhin/rate_limiter_fix
20c44fefb t6913679: Use fallocate on LOG FILESS
f14c3363e Make WinEnv::NowMicros return system time   Previous change for the function   555ca3e7b7 (diff-bdc04e0404c2db4fd3ac5118a63eaa4a)   made use of the QueryPerformanceCounter to return microseconds values that do not repeat   as std::chrono::system_clock returned values that made auto_roll_logger_test fail.
aad0572f8 Fixed the build issue of rocksdbjavastaticrelease
5508122ed Fix a perf regression in ForwardIterator
b72200777 Fix listener_test when using ROCKSDB_MALLOC_USABLE_SIZE
40cd91b7f Fixed compile warning in rocksdbjava
90415cfeb Fixed a compile warning in linux32 environment.
9d6503f88 Fix arena_test test break using glibc-2.17
77a28615e Support static Status messages
18db1e469 better db_bench options for transactions
0be260523 Merge pull request #702 from PraveenSinghRao/remove_spurious
8b689546b Add Subcompactions to Universal Compaction Unit Tests
c6d870ffb Merge branch 'arcpatch-D45741'
57b3a8773 Adding sandcastle determinator for RocksDB
3d78eb66b Arena usage to be calculated using malloc_usable_size()
effd9dd1e Fix deadlock in WAL sync
64f07deb8 remove spurious compression definitions
72a9b73c9 Removed unnecessary checks in DBTest.ApproximateMemoryUsage
cb164bfc4 Do not delete iterators for immutable memtables.
7a0dbdf3a Add ZSTD (not final format) compression type
e2db15efd Merge pull request #701 from PraveenSinghRao/usewinapi_notcruntime
e853191c1 Fix DBTest.ApproximateMemoryUsage
7c916a5d3 Merge pull request #699 from OpenChannelSSD/to_fb_master
0886f4f66 Helper functions to support direct IO
7e327980a Remove usage of C runtime API that has file handle limitation
8ef0144e2 Add argument --show_table_properties to db_bench
1fb2abae2 ColumnFamilyOptions serialization / deserialization.
5f4166c90 ReadaheadRandomAccessFile -- userspace readahead
16ebe3a2a Mmap reads should not return error if reading past file
d286b5df9 DBIter to out extra keys with higher sequence numbers when changing direction from forward to backward
3795449c9 Fix DBTest.GetProperty
a7834a129 Merge pull request #698 from yuslepukhin/address_noexcept_windows
9ccf1bd3e Correct the comment for GetProperty() API.
fbe2c05f5 s/NOEXCEPT/ROCKSDB_NOEXCEPT
6924d7582 Address noexcept and const integer lambda capture   VS 2013 does not support noexcept.    Complains about usage of ineteger constant within lambda requiring explicit capture.
2f8d71ec0 Moving sequence number compaction variables from SubCompactionState to CompactionJob
bab9934d9 Fix build failure caused by bad merge.
4d28a7d8a Add a whitebox test for deleted file iterators.
249fb4f88 Fix use of deleted file iterators with incomplete iterators
53b88784d Add throttling to multi-threaded backups
09d982f9e Fix compact_files_example
6996de87a Expose per-level aggregated table properties via GetProperty()
86d6c3cde Fix Windows build
20d1e547d Common base class for transactions
205083297 Fixing race condition in DBTest.DynamicMemtableOptions
e46bcc08b Remove an extra 's' from cur-size-all-mem-tabless
4ab26c5ad Smarter purging during flush
4c81ac0c5 Fix benchmark report script
b6def58f7 Changed 'num_subcompactions' to the more accurate 'max_subcompactions'
c85296846 db_iter_test: add more test cases for the data race bug
9130873a1 Add options.new_table_reader_for_compaction_inputs
07d2d3416 Add a counter about estimated pending compaction bytes
41a0e2811 Improve defaults for benchmarks
a203b913c Fixed a rare deadlock in DBTest.ThreadStatusFlush
962aa6429 Merge pull request #695 from yuslepukhin/address_windows_build
5bf890762 More indent adjustment.
e2a9f43d6 Adjust indent
6e9a260b0 Merge branch 'address_windows_build' of https://github.com/yuslepukhin/rocksdb into address_windows_build
1cac89c9b Address windows build issues  Intro SubCompactionState move functionality  =delete copy functionality  #ifdef SyncPoint in tests for Windows Release builds
f25f06ddd Address windows build issues   Intro SubCompactionState move functionality   =delete copy functionality   #ifdef SyncPoint in tests for Windows Release builds
027ca5b2c Total SST files size DB Property
b604d2562 Removing unused variables to fix build
1b114eed4 Free file iterators for files which are above the iterate upper bound to Improve memory utilization
3fd70b05b Rate limit deletes issued by DestroyDB
df79eafcb Introduce GetIntProperty("rocksdb.size-all-mem-tables")
888fbdc88 Remove the contstaint that iterator upper bound needs to be within a prefix
137c37667 Removing variables used only in assertions to prevent build error
b47cc5851 Bounding Number of Subcompactions
e58e1b18e Make tailing iterator show new entries in memtable.
9ec957159 DBOptions serialization and deserialization
b2df20a89 Make HashCuckooRep::ApproximateMemoryUsage() return reasonable estimation.
601b1aaca Fixing Failed Assertion in Subcompaction State Diff
f0da6977a [Parallel L0-L1 Compaction Prep]: Giving Subcompactions Their Own State
f32a57209 Simplify querying of merge results
72613657f Measure file read latency histogram per level
b7198c3af reduce db mutex contention for write batch groups
603b6da8b Add options.compaction_measure_io_stats to print write I/O stats in compactions
dc9d5634f Change master to 3.14
b78c8e07d Merge pull request #689 from msb-at-yahoo/add-tools-target
9f0dd2229 Add a 'tools' target.
463720712 Add test case to repro the mispositional iterator in a low-chance data race case
3bd9db420 [Cleanup] Remove RandomRWFile
c3466eab0 Have Transactions use WriteBatch::RollbackToSavePoint
0db807ec2 Transaction error statuses
c2f2cb021 Pessimistic Transactions
c2868cbc5 Use manual_compaction for compaction_job_test
6b2d57039 Fix Windows build by adding snapshot_impl to CMakeLists
e61fafbe7 Fixed clang-build error in util/thread_local.cc
cee1e8a08 Parallelize LoadTableHandlers
4249f159d Removing duplicate code in db_bench/db_stress, fixing typos
a03085b55 Fix linters on non-fb machines
1ae27113c reduce comparisons by skiplist
b47d65b31 Fixed Segmentation Fault in db_stress on OSX.
a1581eca8 Modernize RocksDB linters
a9dcc0a63 Fix clang build
2cf0f4f47 Adding wal_recovery_mode log message
68f934355 Better CompactionJob testing
22dcaaff3 More accurate time measurement for delete_scheduler_test
0a7ea582c Add auto-build manifest for appveyor
0093271ee Merge pull request #685 from flandr/fix-tls-build
ac04a6cfb Fix OSX + Windows build
16ea1c7d1 simple ManagedSnapshot wrapper
257ee895f Fixed memory leaks
254c4fb88 In HISTORY.md Switch unreleased notes to 3.13
40f893f4a Fix delete_scheduler_test valgrind error
6a4aaadcd Avoid type unique_ptr in LogWriterNumber::writer for Windows build break
d7314ba75 Fixing endless loop if seeking to end of key with seq num 0
48e6e9aa8 Add util/delete_scheduler_impl.cc to CMakeLists.txt
c7742452e Add Statistics.getHistogramString() to print more detailed outputs of a histogram
29b028b0e Make DeleteScheduler tests more reliable
fca88f8e1 valgrind_check to exit on test failures
7d364d0d9 Fix build failure
960d936e8 Add function 'GetInfoLogList()'
7ccd1c80a Add two unit tests for SyncWAL()
3ae386eaf Add statistic histogram "rocksdb.sst.read.micros"
8ecb51a7e "make commit-prereq" should clean up rocksjava properly
9aec75fbb Enable DBTest.FlushSchedule under TSAN
bd2fc5f5f Fix TSAN for delete_scheduler_test
8e01bd114 Fix misplaced position for reversing iterator direction while current key is a merge
c46507102 Removing duplicate code
e06cf1a09 [wal changes 3/3] method in DB to sync WAL without blocking writers
5dc3e6881 Update Tests To Enable Subcompactions
c45a57b41 Support delete rate limiting
102ac118b Update JAVA-HISTORY.md for v3.13
3a1d4e6c9 Merge pull request #670 from skunkwerks/fix_osx_shared_library_names
f5d072ae6 Fixed RocksJava test failure of shouldSetTestCappedPrefixExtractor
f39cbcb0a Merge pull request #654 from adamretter/remove-emptyvalue-compactionfilter
f0b5bcc7b add support for capped prefix extractor in java
18ba58a94 Upgrading jemalloc from 3.6.0 to the latest for fbcode+gcc 4.8.1
ce21afd20 Expose the BackupEngine from the Java API
b0d12a135 Merge pull request #569 from adamretter/travis-java-api
241bb2aef Make DBCompactionTest.SkipStatsUpdateTest more stable.
3424eeb1e Polish HISTORY.md
cf3e05304 crash_test cleans up directory before testing if TEST_TMPDIR is set
24daff6d7 Fix a typo and update HISTORY.md for NewCompactOnDeletionCollectorFactory().
14d0bfa42 Add DBOptions::skip_sats_update_on_db_open
e2a3bfe74 First half of whitebox_crash_test to keep crashing the same DB
2e73bd4ff crash_test to put DB under TEST_TMPDIR
1205bdbce crash_test to cover simply cases
d5c0a6da6 Merge branch 'master' of github.com:facebook/rocksdb
2d41403f4 Made change to fix the memory leak
92f7039ee fix memory corruption issue in sst_dump --show_compression_sizes
be8621ffa Fix compile warning in compact_on_deletion_collector in some environment
26894303c Add CompactOnDeletionCollector in utilities/table_properties_collectors.
20b244fcc Fix CompactFiles by adding all necessary files
87df6295d Make SuggestCompactRangeNoTwoLevel0Compactions deterministic
40c64434d Parallelize L0-L1 Compaction: Restructure Compaction Job
47316c2d0 dump_manifest supports DB with more number of levels
bd852bf11 Fixed typos in db_stress
544be638a Fixing fprintf of non string literal
193dc977e Fixing dead code in table_properties_collector_test
05d4265a2 Merge branch 'master' of github.com:facebook/rocksdb
4be6d4416 Compression sizes option for sst_dump_tool
8161bdb5a WriteBatch Save Points
7bfae3a72 tools/db_crashtest2.py should run on the same DB
d06c82e47 Further cleanup of CompactionJob and MergeHelper
e95c59cd2 Count number of corrupt keys during compaction
221a94a5f Another attempt at adding the Java API and tests to the travis build
1bdfcef7b Fix when output level is 0 of universal compaction with trivial move
6a82fba75 Add missing hashCode() implementation
f73c80143 Fixing Java tests.
14f413760 Correct the comment of DB::GetApproximateSizes
8279d4197 Merge pull request #667 from yuslepukhin/fix_now_microsec_win
eb8e3b4c7 Fix shared library names on OSX
555ca3e7b Fix WinEnv::NowMicrosec   * std::chrono does not provide enough granularity for microsecs and periodically emits     duplicates   * the bug is manifested in log rotation logic where we get duplicate    log file names and loose previous log content   * msvc does not imlement COW on std::strings adjusted the test to use     refs in the loops as auto does not retain ref info   * adjust auto_log rotation test with Windows specific command to remove     a folder. The test previously worked because we have unix utils installed     in house but this may not be the case for everyone.
82f148ef9 Fix test DBCompactionTest.PartialCompactionFailure undeterministic failure
6002801e0 Abandon ROCKSDB_WARNING
4922af6f8 fixed DBTest.GetPropertiesOfAllTablesTest and DBTest.GetUserDefinedTablaProperties flakiness
3bf9f9a83 cleaned up PosixMmapFile a little
fe09a6dae [wal changes 2/3] write with sync=true syncs previous unsynced wals to prevent illegal data loss
06aebca59 Report live data size estimate
85ac65536 Tests to avoid to use TMPDIR directly
f09d45690 Merge pull request #664 from yuslepukhin/add_tests_fix_sanity
31b35c902 Add missing tests, fix db_sanity  Add heap_test, merge_helper_test  Fix uninitialized pointers in db_sanity_test that cause SIGSEV when DB::Open fails in case compression is not linked.
66a3cbc54 Merge pull request #663 from yuslepukhin/fix_windows_build_refactor
ac5e441ad Fix windows build after refactoring   Missing and duplicate files in CMake   Missing definition of port::Crash
02b635fa3 Fix undeterministic failure of DBTest.GetPropertiesOfAllTablesTest
3dbf4ba22 RangeSync not to sync last 1MB of the file
7219088cd Move general compaction tests from db_test.cc to db_compaction_test.cc
0adecd9f4 Add db_inplace_update_test back to Makefile
064294081 Improved FileExists API
9f1de9518 Revert Makefile
6867fb19c Revert "Add missing db_log_iter_test in the test list"
443c6646b Move remaining universal compaction tests from db_test.cc to db_universal_compaction_test.cc
7462286d3 Move in-place-update related tests from db_test.cc to db_inplace_update_test.cc
03467bdd4 Add missing db_log_iter_test in the test list
331954ab8 Fixed DBTestUniversalManualCompactionOutputPathId test
a75f23eb8 Relax assertions in unit DropWrites to be more permissible
ee80432ff db_bench add an option of --universal_allow_trivial_move
58b4209e0 Trigger non-trivial compaction in fault_injection_test
59eca2cc9 Make memenv_test runnable in ROCKSDB_LITE
aa8ac6445 Skip unsupported tests in ROCKSDB_LITE
ce9712d34 Make mock_env_test runnable in ROCKSDB_LITE
c06d1d839 Make merge_test runnable in ROCKSDB_LITE
144d2910d Block backupable_db_test_lite in ROCKSDB_LITE
0d1d9aeeb Block plain_table_db_test in ROCKSDB_LITE
4853e228e Make table_test runnable in ROCKSDB_LITE
f0fe9126f Fix compile for write_callback_test in ROCKSDB_LITE
cf6a7bebc Block cuckoo table tests in ROCKSDB_LITE
20922c4a5 Make compaction_picker_test runnable in ROCKSDB_LITE
6e9fbeb27 Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env
5ec829bc4 Cleaning up CYGWIN define of fread_unlocked to port
26ca89319 Block document_db_test in ROCKSDB_LITE
35ca59364 Don't let flushes preempt compactions
79373c372 Fix ROCKSDB_WARNING
74c755c55 Added JSON manifest dump option to ldb command
a96fcd09b Deprecate CompactionFilterV2
1d20fa9d0 Fixed and simplified merge_helper
aede5cd8e Merge pull request #656 from qinzuoyan/fb-master
d730c3677 Merge pull request #657 from yuslepukhin/ensure_clean_public_headers
ac2b9367f Fix a typo in variable
415c47323 Merge after rebasing
d1a457181 Ensure Windows build w/o port/port.h in public headers
6c0c8dee7 Fix data loss after DB recovery by not allowing flush/compaction to be scheduled until DB opened
e4af3bfb2 Test for compaction of corrupted keys
91bf1b80e Java facility to use the RemoveEmptyValueCompactionFilter
3d00271e4 The ability to specify a compaction filter via the Java API
62dec0e2b RemoveEmptyValueCompactionFilter - A compaction filter which removes entries which have an empty value
c5bca5319 Fix compile on Mac
487bba434 extend temp str buffer size
247690fe3 Ensure Windows build w/o port/port.h in public headers
84c3577af fix append bug in DumpDBFileSummary()
43e982562 Fix mongo build -take 2
d8263d958 Unbreak mongo build
12c5528a8 Bump to RocksDB 3.13
81d072623 move convenience.h out of utilities
beb19ad0d Fixing delete files in Trivial move of universal compaction
c61396069 Build fix.
2c8de0eca Update --help message in db_bench.
6b2d44b2f Refactoring of writing key/value pairs
e1c99e10c Replace std::priority_queue in MergingIterator with custom heap, take 2
9a6a0bd8c Style fix in compaction_job.cc
e94c510c3 Make ldb_test not depend on compression
ddad40e93 Fixed nullptr deref and added assert
1bc8eb877 make coverage should execute sequentially
801df912a Move UniversalCompaction related db-tests to db_universal_compaction_test.cc
3ca6b2541 Move TailingIterator tests from db_test.cc to db_test_tailing_iterator.cc
ce829c77e Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc
c3f98bb89 Move CompactionFilter tests in db_test.cc to db_compaction_filter_test.cc
0936362a7 Block SyncPoint in util/db_test_util.h in released Windows mode.
05e194158 Merge pull request #639 from cleaton/setMaxTableFileSize
fc2b71d9c Merge pull request #655 from adamretter/java-make-resolve-maven
8a9fca261 Better error handling in BackupEngine
18d5e1bf8 Remove db_impl_readonly dependency on utilities
9d22a9737 Resolve Java test dependencies from local maven repo if present
49640bd82 Allow write_batch_test to run with ROCKSDB_LITE
a9c510951 Deprecate purge_redundant_kvs_while_flush
5aea98ddd Deprecate WriteOptions::timeout_hint_us
ae29495e4 Avoid manipulating const char* arrays
ab137af4b Partial cleanup of CompactionJob
1879d9370 Add ldb_test.py back to `make check`
b10cf4e2e Move DynamicLevel related db-tests to db_dynamic_level_test.cc
e290f5d3c Block reduce_levels_test in ROCKSDB_LITE
04d201fa0 Block spatial_db_test in ROCKSDB_LITE
49f42ad03 Move global static functions in db_test_util to DBTestBase
625467a08 Move reusable part of db_test.cc to util/db_test_util.h
e8e8c9049 fix compile for optimistic_transaction_test under ROCKSDB_LITE
8bca83e5d Add tombstone information in CompactionJobStats
f9728640f "make format" against last 10 commits
76d3cd328 Fix public API dependency on internal codes and dependency on MAX_INT32
5fd11853c Print Fast CRC32 support information in DB LOG
a6e38fd17 Fix a uncleaned counter in PerfContext::Reset()
e41cbd9c2 Merge pull request #646 from yuslepukhin/ms_win_port
4cab5ebec Merge branch 'ms_win_port' of https://github.com/yuslepukhin/rocksdb into ms_win_port
296de4ae6 Address review comments  Rule of five: add destructor  Add a note to COMMIT.md for 3rd party json.
041b6f95a perf_context: report time spent on reading index and bloom blocks
d08ba9f0c Merge branch 'ms_win_port' of http://vstfbing:8080/tfs/Bing/_git/repo.RocksDB into ms_win_port
805fe84ba Conditional use of third-party libraries  Committed by Alexander Zinoviev <alexander.zinoviev@me.com> 7/9/2015 2:42:41 PM
5555cc500 Improve build system
c903ccc4c Merge from github/master
54d124a38 Conditional use of third-party libraries
7189e90c2 Fix a noisy unit test.
1f4d56570 Add db_bench flag to set cache_index_and_filter_blocks
5c7913233 Revert the changes related to Options, as requested to seperate them into a different patch.
d8586ab22  All of these are in the new code added past 3.10      1) Crash in env_win.cc that prevented db_test run to completion and some new tests      2) Fix new corruption tests in DBTest by allowing a shared trunction of files. Note that this is generally needed ONLY for tests.      3) Close database so WAL is closed prior to inducing corruption similar to what we did within Corruption tests.
4bed00a44 Fix function name format according to google style
e2e3d84b2 Added multi WAL log testing to recovery tests.
5219226d3 Merge branch 'ms_win_port' of http://vstfbing:8080/tfs/Bing/_git/repo.RocksDB into ms_win_port
ef4b87f1b Commit both PR and internal code review changes
95f4c2bcb Conditional use of 3rd-party libraries
4f56632b1 Fix occasional failure in compaction_job_test
411c8e3d1 Build fail fix
b7a2369fb Revert "Replace std::priority_queue in MergingIterator with custom heap"
c0b23dd5b Enabling trivial move in universal compaction
d8e3e766f Fixed a bug in test ThreadStatusSingleCompaction
57d216ea6 Remove assert(current_ == CurrentReverse()) in MergingIterator::Prev()
59b50dcef Update HISTORY.md for Listener
4ce5be425 fixed leaking log::Writers
685582a0b Revert two diffs related to DBIter::FindPrevUserKey()
e12b40399 Initialize threads later in constructor
58d7ab3c6 Added tests for ExpandWhileOverlapping()
155ce60da Fix compaction_job_test
b6655a679 Replace std::priority_queue in MergingIterator with custom heap
e25ee32e3 Arena needs mman header for mmap
d2f0912bd Merge the latest changes from github/master
35cd75c37 Introduce InfoLogLevel::HEADER_LEVEL
acee2b08a Fixed endless loop in DBIter::FindPrevUserKey()
218487d8d [wal changes 1/3] fixed unbounded wal growth in some workloads
feb99c31a Merge remote-tracking branch 'origin' into ms_win_port
e70115e71 Fix unity build by removing anonymous namespace
4159f5b87 Prepare 3.12
a69bc91e3 Multithreaded backup and restore in BackupEngineImpl
9dbde7277 Merge remote-tracking branch 'origin' into ms_win_port
03d433ee6 [RocksJava] Fixed test failures
326da912d Add string.h to Histogram as we init the array out of curly braces
ca2fe2c1b Address GCC compilation issues
19e13a595 Fix header inclusion
18285c1e2 Windows Port from Microsoft
c00948d5e [RocksJava] Fix test failure of compactRangeToLevel
05e283196 Allocate LevelFileIteratorState and LevelFileNumIterator from DB iterator's arena
436ed904d Add rpath option to production builds for 4.8.1 toolchain
b0f1927db Increasing timeout for drop writes.
ec70fea4c Fix a comparison in DBIter::FindPrevUserKey()
501591c42 Make column_family_test runnable in ROCKSDB_LITE
91cb82f34 Merge branch 'master' of github.com:facebook/rocksdb
09f5a4b48 set -e in fb_compile_mongo.sh
6199cba99 Fix race in unit test.
0a019d74a Use malloc_usable_size() for accounting block cache size
4cbc4e6f8 Call merge operators with empty values
619167ee6 Fix mac compile
472e64d39 Improve fb_compile_mongo.sh
c9cd404bc Make flush check for shutdown
4fb09c687 Updating SeekToLast with upper bound
dadc42976 Reproducible MongoRocks compile with FB toolchain
62a8fd154 Make stringappend_test runnable in ROCKSDB_LITE
48da7a9ca Improve the comment for BYTES_READ in statistics.
72cab8895 Block redis_test in ROCKSDB_LITE
dec2c9f56 Make table_properties_collector_test runnable in ROCKSDB_LITE
0b1ffe2e1 Remove -Wl,--no-as-needed flag when making shared_lib in OSX and IOS
674b1181c Bottommost level compaction option
782a1590f Implement a table-level row cache
de85e4cad Introduce WAL recovery consistency levels
530534fce Fix trivial move merge
7015fd81c Add read_nanos to IOStatsContext.
7160f5d80 Fix broken gflags link
dda74111a add setMaxTableFilesSize Options unit test
d62b6ed83 add setMaxTableFilesSize to JNI interface
e1d3c7dbe Fixing valgrind error in checkpoint_test
3bdec09cb Remove ldb_tests.py from make check until it is working again.
15325bf55 First version of rocksdb_dump and rocksdb_undump.
04251e1e3 Add wal files to Checkpoint for multiple column families.
18cc5018b Fix memory leaks in PinnedUsageTest
bf03f59c1 Disable CompressLevelCompaction() if Zlib is not supported
df719d496 Make autovector_test runnable in ROCKSDB_LITE
4d6d47688 Block geodb_test in ROCKSDB_LITE
71b438c4a Remove unused target --- compactor_test
eade498bd Block utilities/write_batch_with_index in ROCKSDB_LITE
760e9a94d Fail DB::Open() when the requested compression is not available
69bb210d5 Add Cache.GetPinnedUsageUsage()
4eabbdb7e Skip bottommost level compaction if possible
4b8bb62f0 Don't dump DBOptions for each column family
176f0bedc Merge branch 'master' of github.com:facebook/rocksdb
bb1c74ce1 Fixed a bug of CompactionStats in multi-level universal compaction case
a66b8157d Merge branch 'master' of github.com:facebook/rocksdb
f06be62fd Replace %llu with format macros in ParsedInternalKey::DebugString())
2dc3910b5 Add --benchmark_write_rate_limit option to db_bench
12e030a99 Use CompactRangeOptions for CompactRange
c89369f57 Move dockerbuild.sh to build_tools/
4716ab4d1 Merge pull request #638 from HolodovAlexander/master
25d600569 Clean up InstallSuperVersion
1369f015e Only initialize the ThreadStatusData when necessary.
1a08d0beb Block c_test in ROCKSDB_LITE
40f562e74 Allow GetApproximateSize() to include mem table size if it is skip list memtable
d59d90bb1 db_bench periodically writes QPS to CSV file
46296cc86 Cygwin build not to use -fPIC
bee8d033f Removed two unused macros in iostats_context
5fec96387 Fixed false alarm of size comparison in compaction_job_stats_test
cccd2199a Revert skip bottommost compaction
20f2b5425 Skip bottom most level compaction if no compaction filter
7842920be Slow down writes by bytes written
a84df655f Don't let two L0->L1 compactions run in parallel
d6ce0f7c6 Add largest sequence to FlushJobInfo
ab455ce49 fix clang build
3eddd1abe Add Env::GetThreadID(), which returns the ID of the current thread.
73faa3d41 Handling edge cases for ReFitLevel
bffaf0a8b Merge pull request #631 from mkhq/patch-1
821cff114 Re-generate WriteEntry on WBWIIterator::Entry()
8b7be1808 Updated OS X instructions, replace homebrew/dupes with homebrew/versions
d03f11090 Link all libraries when building shared libraries
75222d130 Revert "Fix compile"
47f1e7212 Merge pull request #630 from rdallman/c-wb-logdata
51440f83e Fix compile
4949ef08d Re-generate WriteEntry on WBWIIterator::Entry()
735df6655 C: add WriteBatch.PutLogData support
e409d3d74 Make "make all" work for CYGWIN
62c3a9579 Add test for iteration+mutation of WBWI
d9b3338eb Add Yahoo's blog post about Sherpa to USERS.md
75d7075a8 Print info message about files need compaction for debuging purpose
406a5682e Fix hang when closing a DB after doing loads with WAL disabled.
d8c8f08c1 GetSnapshot() and ReleaseSnapshot() to move new and free out of DB mutex
643bbbf08 Use nullptr for default compaction_filter_factory
21f2492ac Fix CYGWin release build
f02ce0c65 Fix ASAN errors in c_test
133130a4f Merge pull request #625 from rdallman/c-slice-parts-support
de4d172d0 Merge pull request #622 from rdallman/c-multiget
6df589b44 Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files
2e764f06e [API Change] Improve EventListener::OnFlushCompleted interface
7322c7401 Revert incorrect commit
31e60e2a7 Unlock mutex in ReFitLevel
7647df8f9 Fixed the tsan failure in util/compaction_job_stats_impl.cc
b2785472c Fix compile
3ce3bb3da Allowing L0 -> L1 trivial move on sorted data
bb808eadd Changed the CompactionJobStats::output_key_prefix type from char[] to string.
0b3172d07 Add EventListener::OnTableFileDeletion()
211a195d4 C: add MultiGet support
5dc174e11 C: add support for WriteBatch SliceParts params
2d0b9e5f0 Fix compile on darwin
3af668ed1 Fix DBTest.MigrateToDynamicLevelMaxBytesBase slowness with valgrind
408cc4b8e Revert "Merge pull request #621 from rdallman/c-slice-parts-support"
78382d4ba Merge pull request #621 from rdallman/c-slice-parts-support
ca8b85ac0 better document max_write_buffer_number_to_maintain
0483dab2a Remove a TODO that has been done
8afafc278 Fix compile warning in db/db_impl
fe5c6321c Allow EventListener::OnCompactionCompleted to return CompactionJobStats.
3083ed212 Fixed heap-use-after-free error in compaction_job_test.cc
8d8d4e45b Fixed ROCKSDB_LITE compile error due to the missing of TableFileCreationInfo
ab946af08 Fix a compile warning in listener_test.cc
fc8382127 Add EventListener::OnTableFileCreated()
898e803fc Add a stats counter for DB_WRITE back which was mistakenly removed.
ac81130fa Fix Bug: CompactRange() doesn't change to correct level caused by using wrong level
ec7a94436 more times in perf_context and iostats_context
4266d4fd9 Allow users to migrate to options.level_compaction_dynamic_level_bytes=true using CompactRange()
d333820ba Removed DBImpl::notifying_events_
495ce6018 Fixed compile warning in compact_files_example.cc
2ecac9f96 add rocksdb::WritableFileWrapper similar to rocksdb::EnvWrapper
a187e66ad Merge pull request #617 from rdallman/wb-merge-sliceparts
16c197627 Fixed db_stress
4c181f08b Fix compile on darwin
bc7a7a400 fix LITE build
832271f6b Fixed a compile warning in db_stress in NDEBUG mode.
dc9d70de6 Optimistic Transactions
d5a0c0e69 Fixed a compile warning in db_stress
ebfdb3c7f Fixed a compile error in ROCKSDB_LITE
9ffc8ba02 Include EventListener in stress test.
a3da59022 Decrease number of jobs in make release
21cd6b7ad C: add support for WriteBatch SliceParts params
a0635ba3f WriteBatch.Merge w/ SliceParts support
c81535103 Support saving history in memtable_list
ec4ff4e99 Rename EventLoggerHelpers EventHelpers
672dda9b3 [API Change] Move listeners from ColumnFamilyOptions to DBOptions
3ab8ffd4d Compaction now conditionally boosts the size of deletion entries.
a81ac2412 Merge pull request #615 from rdallman/master
6d299b70b Fixed a bug in EventLoggerHelpers::LogTableFileCreation
a0580205c Removed an unused private variable in db_impl.h
328ad902a update an import path to fit in with the rest of the kids
9c38ce1d0 C: extra bbto / noop slice transform
8d26799fe Merge pull request #614 from arschles/docker
32198343f fix typo in c_simple_example
6116ccc23 moving dockerfile to root
d90cee9fd adding docker build script and dockerfile
ea6d3a8ac Don't skip last level when calculating compaction stats
5c224d1b7 Fixed two bugs on logging file deletion.
dc81efe41 Change the log-level of DB summary and options from INFO_LEVEL to WARN_LEVEL
687214f87 Ensure ColumnFamilyOptions.num_levels >= 2 when level compaction is used.
2abb59268 Avoid logging under mutex in DBImpl::WriteLevel0TableForRecovery().
309a9d076 Run tests sequentally if J=1
7fee8775a Allow EventLogger to directly log from a JSONWriter.
7a3577519 Don't artificially inflate L0 score
4cb4d546c Set stats_dump_period_sec to 600 by default
e2c1d4b57 [Public API Change] Make DB::GetDbIdentity() be const function.
eaf61ba9f Minor text correction
f16c0b289 Merge pull request #613 from DerekSchenk/DerekSchenk-patch-issue-606
d1a978ae3 Rename JSONWritter to JSONWriter
3e0817541 Add LDFLAGS to Java static library
812c461c9 Dump db stats in WARN level
b588505a7 Update HISTORY.md for GetThreadList() update.
944043d68 Add --wal_bytes_per_sync for db_bench and more IO stats
d5de04d20 Update history for 3.11
08b6b3796 FORCE_GIT_SHA
04feaeebb Fix comparison between signed and usigned integers
4a855c079 Add an option wal_bytes_per_sync to control sync_file_range for WAL files
b0fdda4ff Allow flushes to run in parallel with manual compaction
74f3832d8 Fixed compile errors due to some gcc does not have std::map::emplace
0c8017dba Remove duplicated code
fb5bdbf98 DBTest.DynamicLevelMaxBytesCompactRange: make sure L0 is not empty before running compact range
6fa708512 CompactRange skips levels 1 to base_level -1 for dynamic level base size
84a9c6a53 add comment
eeb44366b C api: human-readable statistics
3f0867c0f Allow GetThreadList to report Flush properties.
a66f643e9 Use a better way to initialize ThreadStatus::kNumOperationProperties.
7413306d9 Take a chance on a random file when choosing compaction
8c52788f0 Use version defined in Makefile in rocksdb_build_git_sha
5aad88129 DBTest.DynamicLevelMaxBytesBase2: remove an unnecesary check
ec43a8b9f Universal Compaction with multiple levels won't allocate up to output size
714fcc067 Make ThreadStatus::InterpretOperationProperties take const uint64_t*
bc68bd5a1 db_bench to support rate limiter
df1f87a88 Fixed compile error in db/column_family.cc
14431e971 Fixed a bug in EventListener::OnCompactionCompleted().
dbd95b753 Add more table properties to EventLogger
b5881762b Reset parent_index and base_index when picking files marked for compaction
711465cce API to fetch from both a WriteBatchWithIndex and the db
3996fff8a Fix clang build - add override
d97813906 SuggestCompactRange() is manual compaction
2fe24e4b5 Don't treat warnings as error when building rocksdbjavastatic
beda81c53 rm -rf in make clean
50eab9cf3 Fix BackupEngine
962f8ba33 Bump to 3.11
37bbd3185 Merge pull request #601 from adamretter/jdb-bench
77a5a543a Allow GetThreadList() to report basic compaction operation properties.
65fe1cfbb Cleanup CompactionJob
df4130ad8 fix crashes in stats and compaction filter for db_ttl_impl
7ea769487 Fix flakiness in column_family_test
a2c4cc756 Don't treat warnings as error when building release
9aa011fa3 Optimize GetRange Function
36a740889 Fix UNLIKELY parenthesis
9bdbaab94 Update USERS.md
2ab7065af build: avoid unused-variable warning
88044340c Add Size-GB column to benchmark reports
d2346c2cf Fix hang with large write batches and column families.
b6b100fe0 Remove iter_refresh_interval_us
197f01b7b Bugfix remove deprecated option use which was removed in previous commit 019ecd19329ee895284e9b040df0ffe4c08b35d8
aa094e8ea Fix conversion from nano-seconds to milli-seconds and seconds
dddceefe5 Fix clang build
d4540654e Optimize GetApproximateSizes() to use lesser CPU cycles.
fd96b5540 Making GetOptions() comment better (#597)
7246ad34d Don't compact bottommost level in SuggestCompactRange
7f47ba0e2 Fix possible SIGSEGV in CompactRange (github issue #596)
aba005c44 Merge pull request #585 from fyrz/RocksJava-RemoveDeprecatedStuff
d6f39c5ae Helper function to time Merges
a087f80e9 Add scripts to run leveldb benchmark
1bb4928da Include bunch of more events into EventLogger
3db81d535 Fix memory leak in cache_test introduced in the previous commit
4961a9622 Fix build
93ab1473d Merge pull request #593 from charsyam/feature/type-1
6ede020dc fix typos
3d1af4ae6 Don't preinstall jemalloc in Travis
242f9b4c2 Fix CLANG build issue introduced in previous commit
794ccfde8 Task 6532943: Rocksdb - SetCapacity() can dynamically change cache capacity if feasible
98a44559d Build for CYGWIN
d01bbb53a Fix CompactRange for universal compaction with num_levels > 1
e003d3864 Abstract out SetMaxPossibleForUserKey() and SetMinPossibleForUserKey
fd7a35731 Enable open source users to use jemalloc (github issue #438)
aa14670b2 Add an assertion in CompactionPicker
2dc421df4 Implement DB::PromoteL0 method
9bf40b64d Print max score in level summary
397b6588b options.paranoid_file_checks to read all rows after writing to a file.
283a04296 Set --seed per test
618d07b06 Making PreShutdown tests more reliable.
0a91bca5d test: avoid vuln-inducing use of temporary directory
6e359419f Add rpath for production builds
78dbd087d Improve benchmark scripts
6a5ffee0c Fix gflags Makefile
108a927f0 Merge pull request #589 from coderplay/patch-1
a58fd7427 Update USERS.md
d85d08c7b One last fix to Makefile
2db96dca1 Fix make install when there is no shared lib
7d136994c Get rid of error output
79c1b021a Fix Makefile
742fa9e31 Fix compile with two gflags
79c21ec0c skip ioctl-using tests when not supported
6059bdf86 Add experimental API MarkForCompaction()
acf8a4141 maint: use ASSERT_TRUE, not ASSERT_EQ(true; same for false
b5400f90f Kill dead code
48b0a045d Speed up reduce_levels_test
00c2afcd3 Fix bug in ExpandWhileOverlapping()
019ecd193 [RocksJava] Remove deprecated methods
98ef21d2f Merge pull request #584 from pshareghi/rocksdb-3.10-falloch
5b7131c75 [RocksJava] Removed deprecated skipLogErrorOnRecovery methods.
566f65271 [RocksJava] Removed deprecated ColumnFamilyDescriptor methods
6997aa0b6 Merge pull request #582 from fyrz/RocksJava-Fix-RateLimiter
582c4b0f7 [RocksJava] Fix RateLimiter Tests in 3.10
6cfb2150d Merge pull request #581 from vladb38/patch-3
d71e8f7ad Update USERS.md
debaf85ef Bug of trivial move of dynamic level
12d7d3d28 Fix and Improve DBTest.DynamicLevelCompressionPerLevel2
a1271c6c6 Fix build break introduced by new SyncPoint interface change
fcb206b66 SyncPoint to allow a callback with an argument and use it to get DBTest.DynamicLevelCompressionPerLevel2 more straight-forward
281db8bb6 Temporarily disable test CompactFilesOnLevelCompaction
e8808b912 Added falloc.h in build_detect_platform
1983fadcb assert(sorted) in vector rep
9da874801 Get benchmark.sh loads to run faster
9b983befa Fix flakiness of WalManagerTest
d41a565a4 Don't do O(N^2) operations in debug mode for vector memtable
08be1803e Fix bad performance in debug mode
0a0501c8d Add Xfunc to makefile
e7ad14926 Fix flakiness in FIFOCompaction test (github issue #573)
abb405227 Kill benchharness
894e9f745 Update Patent Grant.
590fadc40 Fix compile warning on CLANG
47b874398 Make Compaction class easier to use
753dd1fdd Fix valgrind issues in memtable_list_test
697380f3d Repairer documentation improvement.
2f66d7f92 Add LinkedIn back to USERS.md
0feeee643 Fix memtable_list_test
7b9581bc3 Fixed xfunc related compile errors in ROCKSDB_LITE
fabc11569 MemTableList tests
9741dec0e Fix a compile error in ROCKSDB_LITE in db/db_impl.cc
465b25ca9 "make commit-prereq" doesn't really build ROCKSDB_LITE
d2a056241 Fix a compilation error in ROCKSDB_LITE in db/internal_stats.h
316ec80bf fault_injection_test: add a test case to cover log syncing after a log roll
ed229a0de Fixes for readcache-flashcache
91df4e969 Remove use of whole-archive to include jemalloc
84c5bd7eb Add thread-safety documentation to MemTable and related classes
ee9bdd38a Script to check whether RocksDB can read DB generated by previous releases and vice versa
2b019a151 Enabling checksum in repair db as it should have been.
b1bbdd791 Create EnvOptions using sanitized DB Options
edbb08b5d Fix Makefile
199313dc3 build: create .o files specifically for java-related targets
b118238a5 Trivial move to cover multiple input levels
e7adfe690 Fix formatting of USERS.md
4e7543dcf Add USERS.md
58346b9e2 Log writer record format doc.
db6569cd4 Fix the compilation error in flashcache.cc on Mac
cba592001 build: don't use a glob for java/rocksjni/*
c66483c13 Fix github issue #563
de22c7bd1 Integrate Jenkins with Phabricator
f12614070 Fix TSAN build error of D36447
824e64634 Adding another NewFlashcacheAwareEnv function to support pre-opened fd
5e067a7b1 Clean up compression logging
e3ee98b38 run 'make check's rules (and even subtests) in parallel
a45e7581b Avoid naming conflict of EntryType
3be82bc89 Add p99.9 and p99.99 response time to benchmark report, add new summary report
953a885eb A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge
d2a92c13b avoid returning a number-of-active-keys estimate of nearly 2^64
a7ac6cef1 Fix level size overflow for options_.level_compaction_dynamic_level_bytes=true
089509b84 db_test: clean up sync points in test cleaning up
afbafeaea Disallow trivial move if compression level is different
d0695f3e2 Fix crash caused by opening an empty DB in readonly mode
51c8133a7 Fix make unity build compiler warning about "stats" shadowing global variable
df71c6b9e Script to trigger jenkins test
38a01ed1b Update COMMIT.md
76d63b452 Fix one non-determinism of DBTest.DynamicCompactionOptions
b23bbaa82 Universal Compactions with Small Files
2511b7d94 Makefile minor cleanup
1bd70fb54 Add --stats_interval_seconds to db_bench
fd3dbef22 Clean up old log files in background threads
99ec2412e Make the benchmark scripts configurable and add tests
2158e0f83 Fix clang build
d61cb0b9d db_bench can now disable flashcache for background threads
1c47c433b build: always attempt to update util/build_version.cc
e018892bb Formalize the DB properties string definitions.
cfa576402 Make auto_sanity_test always use the db_sanity_test.cc of the newer commit.
e9fddb7a2 Merge pull request #560 from xiaoxichen/patch-1
bcd8a71a2 Fix interger overflow on i386 arch
030859eb5 Dump compression info on startup
3539e0644 Merge pull request #558 from aamihailov/master
a3e4b3248 fix compilation error (same as fix #284)
ff1ff7c62 TSAN: avoid new link failure with -pg
39d508e34 Add a missing section title in HISTORY.md
727684bf9 Fixed a typo in RocksDBSample.java
ca25e86ef build: cause the "check" rule to depend on $(PROGRAMS)
986bdc680 Merge branch 'master' of github.com:facebook/rocksdb
1e57f2bf2 Fix build
2d417e52d Update HISTORY.md for 3.10.0
cbd6a2073 Merge branch 'master' of github.com:facebook/rocksdb
211ca26ae Fixing build issue
2495f9396 Merge pull request #555 from pshareghi/staticLZ4
f06de5f23 Merge pull request #556 from fyrz/RocksJava-Maven-Fix
cd987c383 Fix compile error when NROCKSDB_THREAD_STATUS is not used.
8f104ae5e [RocksJava] Maven build fix
3d1a924ff Adding stats for the merge and filter operation
afc51649e Merge pull request #546 from fyrz/RocksJava-MemEnv
4806cc126 Added static lz4 support for roccksjavastatic
6284eef4c Merge pull request #545 from fyrz/RocksJava-Level-Compression
248c063ba Report elapsed time in micros in ThreadStatus instead of start time.
315abac94 Undeprecate GetLiveFiles()
a057bb2a8 Improve ThreadStatusSingleCompaction
868968c8e Merge branch 'master' of github.com:facebook/rocksdb
689391406 Make SSTDumpTest.GetProperties less noisy
8d8656243 Merge pull request #551 from fyrz/RocksJava-JavaDoc-Fix
f8c505b23 Merge pull request #550 from fyrz/RocksJava-Sample-Fix
46443bfa9 [RocksJava] Add missing JavaDoc annotations
864b7e88f [RocksJava] Java sample bugfix
2b2394cbb [RocksJava] DBBenchmark option for RocksMemEnv
fd8804f97 [RocksJava] Expose MemEnv in RocksJava Summary: In 3.10 the C++ code was extended with a MemEnv implementation. This is now also available in RocksJava.
004b89fba [RocksJava] Add compression per level to API
51da3aab4 Merge pull request #536 from fyrz/RocksJava-32-Bit-adjustment
b7e88cfb5 Merge pull request #543 from fyrz/RocksJava-Logger-Comment
38d286f14 Clean-up WAL directory before running db_test
28bc6de98 rocksdb: print status error message when (ASSERT|EXPECT)_OK fails
9405b5ef8 rocksdb: Remove #include "util/string_util.h" from util/testharness.h
220d0dff7 rocksdb: Remove #include "util/random.h" from util/testharness.h
b088c83e6 Don't delete files when column family is dropped
17ae3fcbc rocksdb: initial util/testharness clean up
39f4271be [RocksJava] Enhanced Logger comment
5615e23d8 [RocksJava] 32-Bit adjustments
836bcc2f0 Merge pull request #532 from fyrz/RocksJava-LevelCompactionDynamicLevelBytes
67d995808 rocksdb: fix make unity
52e0f3353 Clean up compactions_in_progress_
6b626ff24 rocksdb: change db_test::MultiThreadedDBTest as value parameterized test.
9720ea4de A build option to run through all check-in requirements.
0831a3599 Add a DB Property For Number of Deletions in Memtables
f7ed65464 Fix RocksJava test failure due to deprecation of table_cache_remove_scan_count_limit
51301b869 Enable dynamic changing of rate limiter's bytes_per_second
652db51a3 Fix compilation error in rocksjni/write_batch_test.cc
dfccc7b4e Add readwhilemerging benchmark
c345d1ee8 [RocksJava] Integrated changes for D34857
12350115d [RocksJava] Added LevelCompactionDynamicLevelBytes to Options
230e68727 Fix TSAN failue in env_test
155d468c5 Using chrono as a fallback
81345b90f Create an abstract interface for write batches
46214df4a Only run db_test in Travis
c88ff4ca7 Deprecate removeScanCountLimit in NewLRUCache
b4b69e4f7 rocksdb: switch to gtest
413e35273 Merge pull request #540 from dalgaaf/wip-da-fix-elif
969aa806b util/xfunc.h: fix #elif check for NDEBUG
87c7d49d6 util/env_posix.cc: fix #elif check for __MACH__
c86e5d7b9 stack_trace.cc: fix #elif check for OS_MACOSX
98c37fda5 Remove unused parameter in CancelAllBackgroundWork
9fd6edf81 rocksdb: Replace ASSERT* with EXPECT* in functions that does not return void value
d4d42c02e Fixed clang build in env.h
b2b308652 Speed up rocksDB close call.
a7aba2ef6 rocksdb: Add gtest
95344346a rocksdb: Small refactoring before migrating to gtest
bd4963e64 rocksdb: update reference to the latest version of clang dev in fbcode
56337faf3 Fix compaction IO stats to handle large file counts
eafa1bfc3 Merge pull request #529 from fyrz/RocksJava-Logger
ac03c4d51 Merge pull request #522 from fyrz/RocksJava-Optimize-Filters-For-Hits
c6967a1a5 Make RecordIn/RecordOut human readable
c8da67032 Stop printing per-level stall times.
04778a94c [RocksJava] OptimizeFiltersForHits
57f2a00c6 RocksJava - JNI Logger callback
814627af3 [RocksJava] Change log level at runtime in custom logger
a3bd4142f [RocksJava] Logging JNI callback
58878f1c6 Switch to use_existing_db=1 for updaterandom and mergerandom
12134139e Fixed the unit-test issue in PreShutdownCompactionMiddle
fd1b3f385 Fix the issue in PreShutdownMultipleCompaction
56c4a9c76 Fix compile warning in thread_status_util.h on Mac
417367c42 Fix SIGSEGV when not using cache
e25ff039c Prevent slowdowns and stalls in PreShutdown tests
f69071265 Speed up db_bench shutdown
c1b3cde18 Improve the robustness of ThreadStatusSingleCompaction
8c12426c9 Fix the deadlock issue in ThreadStatusSingleCompaction.
b16ead531 DBTest.DynamicLevelCompressionPerLevel should not run without snappy support
a5e60bafc Fix a typo / test failure in ThreadStatusSingleCompaction
cb2c91850 Don't run some tests is snappy is not present
c594b0e89 Allow GetThreadList() to report operation stage.
2623b2cf6 Include chrono
52d8347a9 EventLogger
756532daf Merge pull request #524 from fyrz/RocksJava-Test-Fix
47a2b3a40 Merge pull request #534 from fyrz/RocksJava-Fix-BrokenJacocoReport
9d22a1f13 Allow negative Wnew
2c84303aa Merge pull request #528 from fyrz/RocksJava-NativeLibraryLoader
2dc636f62 [RocksJava] Fix broken jacoco report
f210b0f6c [RocksJava] Fix NativeLibraryLoader
3ebebfccd Prevent xxhash symbols from polluting global namespace
53996149d Removing unnecessary kInlineSize
b411d0603 Prevent stalls in preshutdown tests
1d43bc41f Fixing segmentation fault in db_bench
e9de8b65a Change the way options.compression_per_level is used when options.level_compaction_dynamic_level_bytes=true
2b785d76b Fixed a bug where CompactFiles won't delete obsolete files until flush.
6f5579868 Fixed a compile error in db_bench in mac.
05d92efa7 Add convenience.cc to src.mk
2884b100b db_bench: Better way to randomize repeated read keys in -read_random_exp_range
284be570c Provide a mechanism to inform Rocksdb that it is shutting down
2ddf53b2c Get OptimizeFilterForHits work on Mac
480b28476 Fix make_new_version.sh
89597bb66 Allow GetThreadList() to report the start time of the current operation.
37921b499 db_bench: Add Option -read_random_exp_range to allow read skewness.
485ac0dbd Add rate_limiter to string options
e126e0da5 Single threaded tests -> sync=0 Multi threaded tests -> sync=1 by default unless DB_BENCH_NO_SYNC is defined
dc4532c49 Add --thread_status_per_interval to db_bench
34c75e984 fix-up patch: avoid new link error
ebc647de8 build: fix missing dependency problems
492f6d27e Fix a segfault in fbson under Mac OS X compiler
0d13bbe27 RocksJava] Fix ColumnFamily tests
67533809f [RocksJava] Fixed CompactionTest
1b7b997b8 [RocksJava] Remove MaxValue from Statistics
f862b3812 [RocksJava] Fix cleanup in tests
a01b59259 [RocksJava] DefaultColumnFamily Memory Fix
22c73d15b [RocksJava] Fix ColumnFamily tests
694988b62 Fix a bug in stall time counter.  Improve its output format.
b8d23cdcb Revert chrono use
db0373934 options.level_compaction_dynamic_level_bytes to allow RocksDB to pick size bases of levels dynamically.
f29b33c73 Add functionality to pre-fetch blocks specified by a key range to BlockBasedTable implementation.
c4bd03a97 Fix typo in log message
3cf7f353d Instrument memtable seeks
216a9e16f Fix compile
b9ff6b050 Fix a bug in ReadOnlyBackupEngine
afa8156af adding stdlib to fbson
f9c14a42e Fix compile on Mac
a9f0e2b5b Fix compile
e7c434c36 Add columnfamily option  optimize_filters_for_hits to optimize for key hits only
ba9d1737a RocksDB on FreeBSD support
4ba119df5 rocksdb: Update path to dev clang in fbcode
8984e5f84 Fix race in sync point.
03b432d4b rocksdb: Fix uninitialized use error
ccef2a766 Merge pull request #518 from fyrz/RocksJava-Native-Library-Loader-Fix
9fcf1a7b0 [RocksJava] RocksJava Testcases
03bbf718c Return fbson
62247ffa3 rocksdb: Add missing override
1e06a4068 Support builds for MongoDB+RocksDB
b74ad6632 Merge pull request #508 from fyrz/RocksJava-Final-Patch
d9f4875e5 Disable pre-fetching of index and filter blocks for sst_dump_tool.
182b4ceac Limit key range to number of keys, not number of writes
a360bb61b Merge pull request #516 from fyrz/RocksJava-Update-Statistics-To-3.10
8c7684474 [RocksJava] Updated TickerTypes and Histogram to 3.10
7f0c77cb3 [RocksJava] Integrated changes from D33417
819e787bb [RocksJava] Final usage correction
5139e678b Upgrade compiler in Travis
4ade89962 Fix compile error on MacOS.
ace3d8506 Revert "Unused managed iterator"
7b8f348e5 Attempt at fixing travis issue
d85993998 Merge pull request #506 from fyrz/RocksJava-Raw-Use
30e93c9b9 Merge pull request #505 from fyrz/RocksJava-Redundant-Modifier
217854dc4 Introduce DISABLE_WARNING_AS_ERROR in Makefile
bd339a979 Unused managed iterator
174a79c99 LevelDb include guard replaced with #pragma once
6fdda8ac4 rocksdb: changes to support 'make analyze' in Jenkins
96d989f70 catch config errors with L0 file count triggers
62f7a1be4 rocksdb: Fixed 'Dead assignment' and 'Dead initialization' scan-build warnings
5636c4a80 Verbose build in travis
a047409ae Fixed a bug in the test case
4f514a53d build: enable more compiler warnings
a2b911b63 inputs: restore "const" attribute removed by D33759
1b4082581 mark as unused some variables with cpp-derived names
c6d54b503 fix erroneous assert: cast kBlockSize (of type unsigned int) to "int"
aa5d8e6d9 table_test.cc: add missing 5th arg in TestArgs initializer
c37937a9c maint: remove extraneous "const" attribute from return type
9283c7afd build: remove always-true assertions
06a766de5 Adding Flush to AutoRollLogger
92416fa7f Fix mac build
96ab15d30 GetOptionsFromString + fixes to block_based_table_options
73711f956 rocksdb: Fix scan-build bug 'Memory leak' in db/db_bench.cc
98870c7b9 rocksdb: Fix scan-build memory warning in table/block_based_table_reader.cc
a42324e37 build: do not relink every single binary just for a timestamp
d45a6a400 Add rocksdb.num-live-versions: number of live versions
11581b741 build: abbreviate AR command, too
b8ac71ba1 Revert "Fbson to Json"
7ce1b2c19 Fbson to Json
7d817268b Managed iterator
b4b8c25a5 build: factor out AM_LINK command (trivial)
dc885c6e9 build: make "make" output readable by default
a37b46ae1 build: fix Makefile inconsistencies (trivial)
55277c328 build: remove unused rules: rocksdb_shell, DBClientProxy_test
3ad6b794c rocksdb: Fix 'Division by zero' scan-build warning
12753130e Remove ThreadStatusMultiCompaction test
f0c36da6e Add thread_status_util_debug.cc back
daebb1f91 build: running "make" with no arguments must not "uninstall"
e60bc99fe Allow GetThreadList to reflect flush activity.
b9a0213cd build: fix unportable Makefile syntax
4e4b85784 rocksdb: Fix scan-build 'Called C++ object pointer is null' and 'Dereference of null pointer' bugs
b3fd16226 build: remove unportable use of sed in favor of $(CXX)'s -MT
e7ea51a8e Introduce job_id for flush and compaction
6a0e737d9 [RocksJava] Raw use of parametrized class
439701270 [RocksJava] Redundant access-modifier in interfaces
2d62e8051 Merge pull request #504 from fyrz/RocksJava-Flush-Correction
eaf39568e [RocksJava] FlushOptions Correction
5d1151deb Added simple monitoring script to monitor overusage of memory in db_bench
5f00af457 DBTest.DestroyDBMetaDatabase: create DB directories if not exists
68af7811e Remember whole key/prefix filtering on/off in SST file
fd5970b45 Merge pull request #503 from weiweisd/master
513ad866b modify double type euqal compare in json_document.cc
933973dc2 Merge pull request #495 from fyrz/RocksJava-CF-Name-Byte-Handling
5e8e453d5 [RocksJava] Integrated changes from D33165
677d02427 [RocksJava] CF Name shall handle bytes correctly
d1cafc089 Merge pull request #1 from facebook/master
6d6305dd7 Perf Context to report DB mutex waiting time
863009b5a Fix deleting obsolete files #2
1851f977c Added RocksDB stats GET_HIT_L0 and GET_HIT_L1
91ac3b206 Print DB pointer when opening a DB
bee4e5124 Merge pull request #492 from fyrz/logger-logv-virtual
cfe8837e4 Switch logv with loglevel to virtual
aaceef363 Fix formatting
ee4aa9a0e Merge pull request #481 from mkevac/backupable
82faa377a added simple example for db restore from backup
d090330c8 fixed c_simple_example and added some comments
965130830 renamed backup to backup_and_restore in c_test for clarity
bbb52b21f Merge pull request #483 from adamretter/restructure-java-build
7e50ed8c2 Added some more wrappers and wrote a test for backup in C
218c3ecea Fix std::cout data race
8f679c290 Merge branch 'master' of github.com:facebook/rocksdb
da9cbce73 Add Header to logging to  capture application level information
2a979822b Fix deleting obsolete files
8e83a9d31 Add a missing field for STATE_MUTEX_WAIT to global_state_table
6f1013035 Fix DestroyDB
7de4e99a8 Revert "Fix wal_dir not getting cleaned"
9a52e06a0 Add GetID to ColumnFamilyHandle
181191a1e Add a counter for collecting the wait time on db mutex.
f36d394ae Fix wal_dir not getting cleaned
53ae09c39 db_test: fix a data race in SpecialEnv
fe9f69119 Fix fault_injestion_test
b37f5ffc7 Put db_test back to TESTS in Makefile
108470e96 Fix stack trace on mac
3e53760fc Fix compaction_picker_test
e39f4f6cf Fix data race #3
e63140d52 Get() to use prefix bloom filter when filter is not block based
678503ebc Add utility functions for interpreting ThreadStatus
4d98e2935 rocksdb: Enable scan-build static analysis
756e1f151 Remove unused util/thread_event_info.h
dad98dd4a Changes for supporting cross functional tests for inplace_update
9898f6398 Divide test DBIteratorTest.DBIterator to smaller tests
829363b44 Options::PrepareForBulkLoad() to increase parallelism of flushes
b04408c47 Fix unity build
8d3819369 NewIteratorWithBase() for default column family
2c2d5ab7e Fix compile warning in util/xfunc.h
0b8dec717 Cross functional test infrastructure for RocksDB.
868bfa403 Merge pull request #488 from ekg/master
9900f3821 Merge pull request #484 from fyrz/RocksJava-Release-MD-change
e6eaf938c remove old debugging message (#487)
f33f3955e Moved Java Samples main classes into samples/src/main/java
ad325517f Update test lib versions and maven plugin versions
d6187d07b Maven can now build a standard project layout
157768890 Moved Java Benchmark main classes into benchmark/src/main/java
dd8d5471e Adjustment to NativeLibraryLoader to allow native library to be loaded from either java.library.path or from extracting from the Jar. Means that the test in the build do not need to rely on the Jar, useful when creating similar builds (and executing tests) from Maven
353db6dae Moved Java main classes into src/main/java
98cb501bc Moved Java test classes into src/test/java
7479a62a7 Release.md - Remove version change instrcution
4a4e4279f Update HISTORY-JAVA.md
384cb6619 Merge pull request #480 from fyrz/RocksJava-Deprecate-SkipLogError
ca52a67cf [RocksJava] Deprecate setSkipLogErrorOnRecovery
114d21878 Merge pull request #479 from fyrz/RocksJava-Snapshot-Sequence-Number
cb5c3159f [RocksJava] Snapshot - GetSequenceNumber
ea189b320 Merge pull request #474 from fyrz/RocksJava-GetUpdatesSince
391f85fc8 [RocksJava] Incorporated changes for D32151
68cd93b87 [RocksJava] GetUpdatesSince support
caedd40dd [RocksJava] Adjusted auto pointer
b39006e3d [RocksJava] enable/disable File deletions
9a456fba2 [RocksJava] GetUpdatesSince support
d3a736761 Merge pull request #482 from fyrz/RocksJava-TTL-Fix
939bb3659 [RocksJava] Fix ColumnFamily name alloc in TTL DB
86e2a1eee Allow creating backups from C
db9ed5fdb Unaddressed comment in previous diff. Change only in code comments.
5917de0ba CappedFixTransform: return fixed length prefix, or full key if key is shorter than the fixed length
6c6037f60 Expose Snapshot's SequenceNumber
2fd8f750a Compile MemEnv with standard RocksDB library
173c52a97 Fix build on older compilers -- emplace() is not available
d07fec3bd make DBTest.SharedWriteBuffer to pass MockEnv
4bdf38b16 Disable FlushSchedule when running TSAN
e84299c76 Fix bug recently introduced in MemFile::Lock()
e5aab4c2b Fix data race in HashLinkList
2113ecd3c Merge pull request #472 from fyrz/RocksJava-Cleanup
10af17f3d fault_injection_test: add a unit test to allow parallel compactions and multiple levels
0c4d1053d Fix data race #5
cc0d8be01 [RocksJava] Integrated review comments (D32145)
5257c9c42 Merge pull request #452 from robertabcd/backupable-mem
560ed402b [minor] fprintf to stderr instead of stdout in test
551a41df3 Merge pull request #476 from alabid/alabid/add-to-simple-example
d2a2b058f fault_injection_test: to support file closed after being deleted
f8f040ccc Updated .gitignore to ignore *~ files and example object files
e8bf2310a Remove blob store from the codebase
ea7d0b943 Added WriteBatch block to simple_example.cc
d6c7300cc Fixed a compile warning in clang in db/listener_test.cc
f9758e012 Add compaction listener.
e919ecedf SuperVersion::Unref() to use sequential consistency to decrease ref counting
4c49fedaf Use ustricter consistency in thread local operations
1b43ab58d fault_injection_test: add more logging and makes synchronization slightly stronger
ca2b00277 [RocksJava] Cleanup portal.h & tests
f8dc5c459 [RocksJava] Add missing test to Makefile
b3c133148 [RocksJava] Removed todo comment in portal.h
7ffcc457f [RocksJava] Cleanup portal.h
c4fb83441 Update the comment for the removal of mac-install-gflags.sh
be8f0b12e Rename DBImpl::log_dir_unsynced_ to log_dir_synced_
c1de6c42a fault_injection_test: add a test case to drop random number of unsynced data
d888c9574 Sync WAL Directory and DB Path if different from DB directory
58f34edfc Fix valgrind
f1c886247 Fix data race #1
b08b2fe73 Merge pull request #471 from fyrz/RocksJava-Fix-NativeLibraryLoader
e61f38e5a [RocksJava] Fix native library loader
26b50783d Fix assert in histogramData
42189612c Fix data race #2
f5a839835 Fix archive WAL race conditions
43ec4e68b fault_injection_test: bring back 3 iteration runs
c2e8e8c1c Fix two namings in fault_injection_test.cc
b4c13a868 fault_injection_test: improvements and add new tests
a52dd0024 Fix ASAN failure with backupable DB
910186c27 Return the build with 4.8.1
401d4205e Add thread sanitizer
b068f0a67 Upgrade our compilers
a76d92862 Merge pull request #466 from fyrz/RocksJava-Support-ReadOptions-Iterator
bef7821f0 [RocksJava] ReadOptions support in Iterators
3b494a610 Make options_test runnable on ROCKSDB_LITE
912c52e82 Merge pull request #465 from fyrz/RocksJava-BlockBasedTable-FormatVersion
cd4c07197 Update HISTORY.md for GetThreadStatus() support on compaction.
46a7048dc Reduce false alarm in ThreadStatusMultipleCompaction test
aed028698 Merge pull request #462 from fyrz/RocksJava-JNI-allocation-correction
e5df90f5d Fix comment (minor)
dd53428f8 Incorporated review comments
908258a4f [RocksJava] BlockBasedTableConfig 3.10
2efe22849 [RocksJava] Incorporated changes for D31809
4e48753b7 Sync manifest file when initializing it
e204a5a16 [RocksJava] ColumnFamily name JNI correction
96264784d [RocksJava] ColumnFamily name JNI correction
ae82849bc Fix build failure
423dee841 Abort db_bench if Get() returns error
206237d12 DBImpl::CheckConsistency() shouldn't create path name with double "/"
5e98e5324 Merge pull request #458 from fyrz/RocksJava-TTLDB-Support
4ffe0be41 [RocksJava] Integrated changes for D31449
e82856754 [RocksJava] Integrated changes from D31449
859c54a03 [RocksJava] TTL-Support
5ff8aec4d [RocksJava] TTL Support
ca47da9e6 [RocksJava] TTL-Support
1190ebe5a Merge pull request #461 from fyrz/RocksJava-DirectSlice-Fix
ea25ff715 [RocksJava] Integrated proposed simplificiation
d68e83c35 [RocksJava] DirectSlice String termination fix
0ddf5f73e memenv: normalize file path
4d9d5955a Merge pull request #464 from fyrz/RocksJava-Various-Fixes
ceaea2b72 Adding prefix_extractor string config parameter
3d628f8f2 Update format_version comment
155bec4cf fallocate also tests FALLOC_FL_KEEP_SIZE
c75c02e7a [RocksJava] WriteBatchWithIndexTest fix
c787fb50b [RocksJava] JavaDoc errors in Java8
b229f970d Remove Compaction::ReleaseInputs().
f2ddb8b45 Fix for bug where GeoDB accesses key after next modification of iterator
d10f1de2b Ported LevelDB's fault_injection_test
2bb059007 Change db_stress to work with format_version == 2
9ab5adfc5 New BlockBasedTable version -- better compressed block format
2355931c6 Merge pull request #450 from adamretter/writebatch-with-index
3d246c89c Abstract duplicate code on key and value slice objects into generic methods
2d0dd8db3 Implement WBWIRocksIterator for WriteBatchWithIndex in the Java API
de678b288 Abstractions for common iterator behaviour
e01acb3a0 Test for WriteBatchWithIndex#newIterator()
56f24941a Simplify the Java API by permitting WriteBatchWithIndex to be provided straight to RocksDB#write
95d5f9848 Test for RocksDB#write(WriteBatchWithIndex)
ef5b34dee Implement WriteBatchWithIndex in the Java API
c6e554561 Abstractions for common write batch behaviour
be905491b Test for WriteBatchWithIndex#newIteratorWithBase(org.rocksdb.RocksIterator)
2241e3f4d Extract the interface for a RocksIterator
a8cfa7ace Extract the interface for a WriteBatch
45e43b81d Adds support for db->DefaultColumnFamily() to the Java API
516a04267 Add LZ4 compression to sanity test
2ccc54301 Merge pull request #460 from neutronsharc/master
2a7bd0ea4 Remove duplicated method declarations in C header.
bb128bfec More accurate message for compaction applied to a different version
96b8240bc Support footer versions bigger than 1
53f615df6 Fix clang build
02b30202c Merge pull request #455 from Andersbakken/stdlib_fix
2159484dd Remove two unnecessary blank lines in db/db_test.cc
e7dd88c57 Merge pull request #441 from fyrz/RocksJava-ColumnFamilyDescriptor-Alignment
d2c018fd5 Make ThreadStatusMultipleCompaction more robust.
bf9aa4dfc Improve GetThreadStatus to avoid false alarm in some case.
c91cdd59c Allow GetThreadList() to indicate a thread is doing Compaction.
402c1152a Fix c_simple_example
a9ea65d65 Build with clang 3.5 on Linux.
23ad5f401 [RocksJava] Incorporated changes for D30525
0aab1005f [RocksJava] ColumnFamilyDescriptor alignment with listColumnFamilies
15d2abbec Fix build issues
abb9b95ff Move compression functions from port/ to util/
9132e52ea DB Stats Dump to print total stall time
93b35c299 Merge pull request #453 from fyrz/SimpleCExampleSigSegv
628a67b00 Reduce memory footprint in backupable db.
ef3901642 Fixed memory issue in c_simple_example
b89d58dfa :%s/build_config/make_config
242b9769c Memtablerep Benchmark
73ee4feba Add comments about properties supported by DB::GetProperty() and DB::GetIntProperty()
2dca48f55 Merge pull request #451 from StanislavGlebik/document_db_improvement
4b57d9a82 Fixed negative numbers comparison in DocumentDB
9ef59a09a VersionSet::AddLiveFiles() to assert current version is included.
4d16a9a63 VersionBuilder to optimize for applying a later edit deleting files added by previous edits
7731d51c8 Simplify column family concurrency
07aa4e0e3 Fix compaction summary log for trivial move
9d5bd411b benchmark.sh won't run through all tests properly if one specifies wal_dir to be different than db directory.
62ad0a9b1 Deprecating skip_log_error_on_recovery
fa0b126c0 Fix corruption_test -- if status is not OK, return status -- during recovery
d7b4bb62a Fail DB::Open() on WAL corruption
9619081d9 Merge pull request #449 from robertabcd/improve-backupable
49376bfe8 Fix errors when using -Wshorten-64-to-32.
a8c5564a9 Do not issue extra GetFileSize() calls when loading BackupMeta.
caa1fd0e0 Improve performance when loading BackupMeta.
e9ca35815 Fix CLANG build for db_bench
bf287b76e Add structures for exposing thread events and operations.
a801c1fb0 db_bench --num_hot_column_families to be default off
2067058a6 Dump routine to BlockBasedTableReader (valgrind)
ddc81440d db_bench to add an option as number of hot column families to add to
a944afd35 Fixed a compile error in db/db_impl.cc on ROCKSDB_LITE
7ea7bdf04 Dump routine to BlockBasedTableReader
ae508df90 Clean up compile for c_simple_example
b62300961 Fix compile of compact_file_example
ded26605f Merge pull request #444 from adamretter/java-api-fix
98490bccf Fix the build on Mac OS X
4d9972974 Merge pull request #443 from behanna/master
5045c4394 add support for nested BlockBasedTableOptions in config string
d232cb156 Fix the build with -DNDEBUG.
45bab305f Move GetThreadList() feature under Env.
4fd26f287 Only execute flush from compaction if max_background_flushes = 0
0acc73881 Speed up FindObsoleteFiles()
d8c4ce6b5 Merge pull request #442 from alabid/alabid/fix-example-typo
949bd71fd fix really trivial typo
f8999fcf3 Fix a SIGSEGV in BackgroundFlush
ade4034a9 MultiGet for DBWithTTL
fdb6be4e2 Rewritten system for scheduling background work
a3001b1d3 Remove -mtune=native because it's redundant
e27c84522 Merge pull request #437 from fyrz/RocksJava-SliceTests-Fixes
1fed1282a [RocksJava] Incorporated changes D30081
5b9ceef01 [RocksJava] JavaDoc correction
5fbba60b6 [RocksJava] Incorporated changes D30081
b0230d7e0 [RocksJava] Incorporate additions for D30081
b015ed0ca [RocksJava] Slice / DirectSlice improvements
4d422db01 Merge pull request #430 from adamretter/increase-parallelism
04c4e4969 Merge pull request #411 from fyrz/RocksJava-RangeCompaction
62d19b7b5 Merge pull request #427 from haneefmubarak/c-examples
28424d734 style fixes in c example
7198ed5a2 Handle errors during pthread calls
91c58752f error detection and memory leaks in c example
25f70a5ab Avoid unnecessary unlock and lock mutex when notifying events.
7661e5a76 Move the file copy out of the mutex.
17e84f215 Rudimentary test cases for setIncreaseParallelism
eda0dcdd9 Exposed IncreasedParallelism option to Java API as setIncreasedParallelism
efc94ceb2 [RocksJava] Incorporated changes for D29283
69188ff44 [RocksJava] CompactRange support
48adce77c [RocksJava] CompactRange support
153f4f071 RocksDB: Allow Level-Style Compaction to Place Files in Different Paths
06eed650a Optimize default compile to compilation platform by default
cef6f8439 Added 'dump_live_files' command to ldb tool.
7ab1526c0 Add an assert and avoid std::sort(autovector) to investigate an ASAN issue
74b3fb6d9 Fix Mac compile errors on util/cache_test.cc
d7a486668 Improve scalability of DB::GetSnapshot()
ee95cae9a Modifed the LRU cache eviction code so that it doesn't evict blocks which have exteranl references
0ab0242f3 VersionBuilder to use unordered set and map to store added and deleted files
e93f044d9 add range scan test to benchmark script
cb82d7b08 Fix #434
046ba7d47 Fix calculation of max_total_wal_size in db_options_.max_total_wal_size == 0 case
1b7fbb9e8 Update HISTORY.md for release 3.9
635c61fd3 Fix problem with create_if_missing option when wal_dir is used
2871bc7bc Merge pull request #422 from fyrz/RocksJava-Quality-Improvements
8c5781666 Add -fno-exceptions flag to ROCKSDB_LITE.
1f04066ca Add DBProperty to return number of snapshots and time for oldest snapshot
6436ba6b0 Provide mechanism to restart tests from previous error
d84b2bade Replace exception by abort() in dummy HdfsEnv implementation.
9260e1ad7 Bump version to 3.9
8f4e1c1c9 Remove the compability check on log2 OS_ANDROID as it's already blocked by ROCKSDB_LITE
c4a7423c1 Replace runtime_error exception by abort() in thread_local
a94d54aa4 Remove the use of exception in WriteBatch::Handler
a5d4fc0a2 Fix compile warning in db_stress
1a8f4821a Replace exception by assertion in autovector
97c194088 Fix compile warning in db_stress.cc on Mac
5f719d720 Replace exception by setting valid_ = false in DBIter::MergeValuesNewToOld()
c0dee851c Improve formatting, add missing newlines
815f638cd Fix java build
32a0a0384 Add Moved(GB) to Compaction IO stats
a14b7873e Enforce write buffer memory limit across column families
3e684aa68 Integrated changes from D29571
37d73d597 Fix linters
a15169f2e Fixed a Lint problem
b7f9e644c [RocksJava] Quality improvements
e002a6122 [RocksJava] Comparator tests for CF
335e6ad5c [RocksJava] Remove obsolete dbFolder cleanup
b036804ac RocksJava - FindBugs issues
9a632b4a9 Merge pull request #429 from fyrz/RocksJava-MacOSX-strip-fix
b42667506 [RocksJava] MacOSX strip support
e463cb0bc Merge pull request #424 from eile/master
91d898163 Tweak Makefile for building on BG/Q
c6f31a289 minor memory leak in C example
703ef66a8 Merge pull request #426 from fyrz/RocksJava-Restore-PrecisionFix
ac4ed1e30 fix examples/makefile for C example
d7f5ccb0c add c example to makefile and fix "make clean"
9c34d5e36 fix type in C simple example
0a9a7e753 added C version of simple_example
bcf908689 Block Universal and FIFO compactions in ROCKSDB_LITE
67cb7ca75 [RocksJava] Fixed MacOS build of RocksJava
b8136a7d2 Merge pull request #398 from fyrz/RocksJava-CreateCheckPoint
533592a27 Merge pull request #401 from fyrz/RocksJava-Sigsegv-MergeOperatorName
73d72ed5c Block ReadOnlyDB in ROCKSDB_LITE
e47f0fa9e Merge pull request #425 from adamretter/macosx-clean-fix
ff0cb90d1 Do not delete Java Fatal Error Log, developers may still want these for reference
2a792cd30 There will also be a librocksdbjni-osx.jnilib.dSYM folder on MacOSX builds to be deleted
beb74c14c Fix travis-build error
a486352e0 Merge pull request #423 from zerebubuth/c_iterate_upper_bound
26109d487 Store upper bound `Slice` with the same lifetime as the `ReadOptions` so that we can provide a pointer to it.
a97314219 Fix compile error in ROCKSDB_LITE
9d5019327 Replace log2 by implementing Log2 in options_builder
805bac6d2 Add test for upper bounds on iterators using C interface.
f193deea3 [RocksJava] Addressed comments in D28971
94f70a86b [RocksJava] Incoroporated changes for D29013
a280af2a5 [RocksJava] Sigsegv fix for MergerOperatorByName
fcc2dfd9f [RocksJava] Support for stored snapshots
274ba6270 Block internal_stats in ROCKSDB_LITE
4f2e8bab5 Merge pull request #421 from fyrz/RocksJava-PrecisionFix
c4765dc10 [RocksJava] Fix precision problem in rocksjni
14788e181 Merge pull request #420 from rdallman/add-wal
88dd8d889 c api: add max wal total to opts
7e608e2fe Block plain_table_index.cc in ROCKSDB_LITE
13de000f0 Add rocksdb::ToString() to address cases where std::to_string is not available.
90ee85f8e Improve listener_test to avoid possible false alarm
2946e37a0 remove unreliable test in db/cuckoo_table_db_test.cc
9c7ca65d2 free builders in VersionSet::DumpManifest
7530c75ab Merge pull request #413 from saghmrossi/master
d699d7034 Make RocksDB compile without gflags
325722149 Fixes valgrind error in GetSnapshotLink. Free checkpoint now.
ada3d7873 Merge pull request #415 from fyrz/RocksJava-Makefile
569853ed1 Fix leak when create_missing_column_families=true on ThreadStatus
c4b65f70f [RocksJava] Makefile correction
141018016 Make arena use hugepage if possible
3a40c427b Fix db_bench on CLANG mode
9222a2d02 Fixed iOS build caused by GetThreadList feature.
aa31fc506 Improve listener_test by ensuring flushes are completed before assert.
7ec71f101 Provide default implementation of LinkFile, don't break the build
cd278584c Clean up StringSplit
d84069995 Fix mac compile
4f882924d Merge pull request #404 from fyrz/RocksJava-Backup-Restore-3.8
4b63fcbff Add enable_thread_tracking to DBOptions
bafce6197 first rdb commit
9e285d423 Added CompatibleOptions for compatibility with LevelDB Options
353307758 Add IOS_CROSS_COMPILE to macro guard for GetThreadList feature.
eecdebe65 Fixed the destruction order of static variables in ThreadStatusImpl.
004f416b7 Moved checkpoint to utilities
beabc6879 Fixed ~ThreadStatusImpl().
faa8d32be [RocksJava] Integrated changes from D29019.
3d78c7a8c [RocksJava] Lint adjustments
d7529b2de [RocksJava] Cleanup Backupable implementations
fa703efb2 [RocksJava] Improved BackupableDBTest
24fdc4741 [RocksJava] Backupable/Restorable DB update 3.8.0
9972f969e [RocksJava] BackupableDBOptions alginment + 3.8
fbc42a093 Fixed -Werror=unused-but-set-variable in thread_status_impl
a564be715 Fix asan error in thread_status_impl.cc
7165d1886 Fix clang compile error
d0c5f28a5 Introduce GetThreadList API
1fd1aecb3 Merge pull request #409 from fyrz/RocksJava-Make-cleanup
91c8dcefc [RocksJava] Strip library in publish
e7fcaa4d9 [RocksJava] JavaDoc is executed too often
2cd1794e4 [RocksJava] Make cleanup - Clean Target
be005e17b fix clang compilation
5e69f19c4 Merge pull request #405 from fyrz/RocksJava-Convenient-Options
55a344872 Merge pull request #408 from fyrz/Missing-include
9e9a83baf Missing header in build on CentOS
91ccc8ebe [RocksJava] Integrated changes in D29025
5249d0db5 [RocksJava] Convenience methods for Options
8d3f8f969 remove all remaining references to cfd->options()
1e4a45aac remove cfd->options() in DBImpl::NotifyOnFlushCompleted
517c28994 Options helper supports k, m, g, and t unit suffixes
c46c2be8d Merge pull request #397 from fyrz/RocksJava-GetIntProperty
8efd4bb42 [RocksJava] Improved comments in RocksDB class
5529c1ad1 [RocksJava] GetIntProperty in RocksDB
db59eeb61 Merge pull request #406 from fyrz/Build-Fix
e97f014b9 [RocksJava] JavaDoc corrections - Java8
98e59f981 Fixed a bug which could hide non-ok status in CompactionJob::Run()
ec24bd4e6 Merge pull request #402 from adamretter/bugfix-native-library-loader
d3c4a0f4a Improve the comment in InfoLogLevelTest.java
a77e97c53 Merge pull request #396 from fyrz/RocksJava-LogLevel
585c759cf Make sure to use the correct Java classloader for loading the RocksDB Native Library
c3915abba Minor tidyup and use Java 7 for file copying
a122a42bb Merge pull request #399 from fyrz/RocksJava-Version-to-3.8
b8d5e3f08 [RocksJava] MVN Build reads version from version.h
23295b74b Clean job_context
0ce38fe98 Fix signed/unsigned compile
e7960c03a Don't parallelize the build in travis
84af2ff8d Clean job context in DeleteFile
8a1bcc39c [RocksJava] Bump version to 3.8 in rocksjni.pom
5c04acda0 Explicitly clean JobContext
4947a0674 [RocksJava] Incorporated review comments D28947
07cd3c42a [RocksJava] LogLevel support in Options
26dc5da96 Fix compaction_job_test
5f583d2a9 Merge pull request #394 from lalinsky/cuckoo-c
353303a76 Merge pull request #380 from fyrz/RocksJava-Junit-Framework
3f9c95a51 [RocksJava] Minor lint correction
e46450da6 [RocksJava] Rebased + integrated CF tests
cd82beb0c [RocksJava] Merged in latest changes.
b6abab8b7 [RocksJava] Merged & rebased to HEAD
74057d6d2 [RocksJava] Improved tests within RocksJava
628e39e97 [RocksJava] Integrated review comments from D28209
a4b28c1ae [RocksJava] Extended Testcases
36f3a0bb8 [RocksJava] Integrated review comments from adamretter in D28209
b09268695 [RocksJava] Extended testcases
9bec23c41 [RocksJava] Test-framework integration
f617135d5 [RocksJava] Testcase improvements
1fe7a4c62 [RocksJava] Test-framework integration
04ca7481d Fix build
6c1b040cc Provide openable snapshots
9be338cf9 CompactionJobTest
c9fd03ec5 Update docs for NewAdaptiveTableFactory
e6c3cc657 Add very basic tests to make sure the C cuckoo table options compile and run
7fe247080 Update HISTORY.md for RocksJava
c44a29278 Add cuckoo table options to the C interface
136b8583b Merge pull request #395 from lalinsky/fix-env-test
94fa542f8 Update HISTROY.md for 3.8 release
a177742a9 Make db_stress built for ROCKSDB_LITE
746cfaac5 Relax the block count check on deallocation in env_test
f822129b3 Add a unit test for behavior when merge operator and compaction filter co-exist.
4161de92a Fix SIGSEGV
373c665ed Fix broken test in 31b02d.
772bc97f1 No CompactFiles in ROCKSDB_LITE
1d1a64f58 Move NeedsCompaction() from VersionStorageInfo to CompactionPicker
cd0980150 Add concurrency to compacting SpatialDB
3c92e5233 Fix include
25f273027 Fix iOS compile with -Wshorten-64-to-32
fa50abb72 Fix bug of reading from empty DB.
31b02dc21 Improve Backup Engine.
1033db29f Merge pull request #390 from fyrz/RocksJava-Cleanup
9a03da773 Merge pull request #375 from fyrz/RocksJava-ColumnFamilyOptions-Extension-3.6
d50c68e3a [RocksJava] JavaDoc cleanup warnings with Java8
079d942ea [RocksJava] Code-cleanup + Java7 warnings removed
9a255b95f [RocksJava] Sample and Default value
9d2ba2136 [RocksJava] Incorporated review comments
fa9cfc65f [RocksJava] Integrated Review comments from yhchiang in D28023
75010d208 [RocksJava] ColumnFamily custom Options API extension
0345c2156 [RocksJava] Extend Options with ColumnFamilyOptions implementation ColumnFamilyOptions implementation with tests [RocksJava] Extended ColumnFamilyTest
975949522 Fixed clang compile error in version_builder_test
581141935 Fixed GetEstimatedActiveKeys
1f621e6ab Fix additional -Wshorten-64-to-32 errros
767777c2b Turn on -Wshorten-64-to-32 and fix all the errors
113796c49 Fix NewFileNumber()
625e162c6 Merge pull request #393 from fyrz/RocksJava-Flush
fc6fcbab9 [RocksJava] Flush functionality
8e5547f64 [RocksJava] Makefile restructured
26f0a78b0 Merge pull request #363 from adamretter/write_batch-iterate
35c8c814e Make ForwardIterator::status() more efficient
d88568c68 Move -Wnon-virtual-dtor to c++ flags
c7ee9c3ab Fix -Wnon-virtual-dtor errors
746252197 Merge pull request #384 from msb-at-yahoo/compaction-filter-2-empty-changed-values
dd726a59e Bump Version Number to 3.8
4a3bd2bad Optimize usage of Status in CompactionJob
bcdb9671c Fix build
d904fbbb0 Addresed comments from code review https://reviews.facebook.net/D27567
eeb9cf6c4 Test for WriteBatchHandler
8fb4751d5 Iterator support for Write Batches
00211f9c5 Fix SIGSEGV in db_stresS
a4a2bfd6b Merge pull request #391 from Liuchang0812/fixmake
856059094 Merge pull request #389 from Liuchang0812/master
01a770637 remove unused target
dc3410463 Merge branch 'fix-example'
e7620536c fix make static_lib error
543df158c Expose sst_dump functionality as library call.
e3d3567b5 Get rid of mutex in CompactionJob's state
344edbb04 Fixed the shadowing in db/compaction.cc and include/rocksdb/db.h
b8b390342 Fixed compile error in db/db_impl.cc
b622ba5d6 Fixed compile error in db/flush_job.cc
642ac9d8a Fixed compile error in db/compaction.cc and db/compaction_picker.cc
68effa034 Fix -Wshadow for tools
844786189 Fixed -WShadow errors in db/db_test.cc and include/rocksdb/metadata.h
28c82ff1b CompactFiles, EventListener and GetDatabaseMetaData
5c9309053 Turn on -Wshadow
31342c400 Fix implicit compare
a0f887c9e Fix compile
53af5d877 Redesign pending_outputs_
ec101cd49 Correctly test both compaction styles in CompactionDeletionTriggerReopen
8d87467bb Make PartialCompactionFailure Test more robust again.
64d302d30 make DropWritesFlush deterministic
cd5c0925a Merge pull request #387 from fyrz/RocksJava-WShadow-Fix
c4bf07c24 [RocksJava] -WShadow improvements
e526b7140 Make PartialCompactionFailure Test more robust.
0c2be0de3 Turn on -Wshadow for travis
5fd33d26f Turn off -Wshadow
9f20395cd Turn -Wshadow back on
c02338a69 update HISOTRY.md for new release
367a3f9cb Improve DBTest.GroupCommitTest: artificially slowdown log writing to trigger group commit
b52b144c7 Merge pull request #386 from EugenePig/java8
ac95ae1b5 Make sure WAL is synced for DB::Write() if write batch is empty
59d549798 suppress JDK8 errors for #385
ea18b944a Add db_bench option --report_file_operations
2ea1219eb Fix RecordIn and RecordDrop stats
e4211d10c Apply InfoLogLevel to the logs in util/env_hdfs.cc
76f6c7c7c CompactionFilterV2: eliminate an often unnecessary allocation.
29a9161f3 Note dynamic options in options.h
fd24ae9d0 SetOptions() to return status and also add it to StackableDB
b1267750f fix the asan check
83bf09144 Bump verison number to 3.7
da5daa061 Replace some ASSERT_TRUE() asserts in DBTest.DynamicMemtableOptions and DynamicCompactionOptions with more specific ones
b0cda4a11 DBTest.DynamicMemtableOptions to use single background compaction
8810850dd Apply InfoLogLevel to the logs in db/compaction_job.cc
71783f652 Merge pull request #377 from fyrz/RocksJava-KeyMayExist
614bbcbe2 Merge pull request #374 from fyrz/RocksJava-DBOptions-Extension-3.6
d8e119663 Apply InfoLogLevel to the logs in db/version_set.cc
2a019f1d0 Apply InfoLogLevel to the logs in db/wal_manager.cc
469d474ba Apply InfoLogLevel to the logs in db/db_impl.cc
ac6afaf9e Enforce naming convention of getters in version_set.h
09899f0b5 DB::Open() to automatically increase thread pool size if it is smaller than max number of parallel compactions or flushes
636e57b52 Fix coverage script
30ca3752b Revamp our build tools
051c67ff5 Merge pull request #378 from baotiao/master
d0e7e49ae Merge pull request #379 from fyrz/RocksJavaBuildFix
94e31ac22 [RocksJava] Extend Options with DBOptions implementation [RocksJava] Included DBOptionsTest and refactored OptionsTest
b060d3006 [RocksJava] Build fix after options refactoring
9fd65e566 add make clean in examples makefile
8e79ce68c Revert "Fix lint errors and coding style of ldb related codes."
45a612f99 Revert "Fix incorrect fixing of lint errors in ldb_cmd.cc"
27129c739 [RocksJava] KeyMayExist w/o ColumnFamilies
86905e3cb Move VersionBuilder logic to a separate .cc file
74eb4fbe9 CompactionJob
8ddddd62d Fix incorrect fixing of lint errors in ldb_cmd.cc
46c14c666 Fix #258. benchmarkharness -- make bm_min_usec uint
72cb7cf20 Add fsync / corrupt simulation to env_mem
0e526eb9d introduce TestMemEnv and use it in db_test
8db24f4b3 exclude mock test file from MOCK_SOURCES
5594d446f unfriend DBImpl and InternalStats from VersionStorageInfo
82e3ae540 fix c_test
bc9f36fd5 Fix lint errors and coding style of ldb related codes.
c645250ee CompactionStats to support larger value of RecordIn and RecordDrop
f7e6c856a Fix BaseReferencedVersionBuilder's destructor order
c76dcb44d fix
b452dede5 fix
29d83cc33 temporarily remove -Wshadow
c1a924b9f Move convenience.h to /include
7e01d1202 Add support for in place update for db_stress
9f7fc3ac4 Turn on -Wshadow
98849a35f Apply InfoLogLevel to the logs in table/block_based_table_reader.cc
4d2ba38b6 Make VersionBuilder unit testable
2b1f23dca Apply InfoLogLevel to the logs in db/db_iter.cc
ccaf1aa7c Merge pull request #372 from fyrz/RocksJava-CF-Merge-Hardening
a29118ffc Merge pull request #355 from fyrz/RocksJava-Options-Refactoring-3.6
85b04ca76 [RocksJava] Review comments - reformatted MergeTest
df7abb4e8 [RocksJava] Integrated code review comments
171be0ed5 Merge with ColumnFamilies & Hardening CFHandle
39464a990 [RocksJava] Options Refactoring 3.6
0f7f3b860 Check InfoLogLevel earlier in Log functions.
73605d917 Apply InfoLogLevel to the logs in util/db_info_dumper.cc
fda592d90 Merge pull request #356 from fyrz/RocksJava-TableOptions-3.6
c73d13bb8 [RocksJava] Integrate review comments from yhchiang
b011e201f Integrated review comments by ankgup87
2c1bd8846 BlockBasedTableConfig & PlainTableConfig enhancements
e770d6175 Merge pull request #371 from dlezama/master
41af0f56b Fix build break because of unsigned/signed mismatch
c5db7f260 Fix CompactionPickerTest.Level1Trigger2
37e9b6370 Apply InfoLogLevel to the logs in utilities/ttl/db_ttl_impl.h
217cc217d Apply InfoLogLevel to the logs in table/meta_blocks.cc
635905481 WalManager
fd95745a5 Fix compile error in table/plain_table_index.cc
c3dd0f75d comparator_db_test to cover more irregular comparators
6afafa369 Apply InfoLogLevel to the logs in utilities/merge_operators/uint64add.cc
e7ad69b9f Apply InfoLogLevel to the logs in table/plain_table_index.cc
bbd9c5345 Apply InfoLogLevel to the logs in table/block_based_table_builder.cc
065766b8d DynamicCompactionOptions: relax the check bound a little
5c82a8837 Add a test in compaction_picker_test to test the max score
86de2007b Add ComparatorDBTest to test non-default comparators
17be187ff dummy var to suppress compiler warning/error
c2999f54b Revert "tmp"
76d1c28e8 Make CompactionPicker more easily tested
01e6f8509 Apply InfoLogLevel to the logs in db/transaction_log_impl.h
082e49ba8 Apply InfoLogLevel to the logs in db/repair.cc
c4b468000 Apply InfoLogLevel to the logs in db/flush_job.cc
34d436b7d Apply InfoLogLevel to the logs in db/column_family.cc
cda9943f9 Apply InfoLogLevel to the logs in db/compaction_picker.cc
7b3a618f9 Apply InfoLogLevel to the logs in db/db_filesnapshot.cc
2d4fe048f remove dead code
9ab013236 tmp
76d54530d minor - remove default value for ChangeFilterOptions() and ChangeCompactionOptions()
44f0ff31c use fallocate(FALLOC_FL_PUNCH_HOLE) to release unused blocks at the end of file
97451f837 add an env var ROCKSDB_TESTS_FROM to control where to start from a list of tests
e130e88bc DBTest: options clean up - part 4
34f3c5a20 DBTest: options clean up - part 3
cdc7230e4 DBTest: options clean up - part 2
5a921b895 DBTest: options clean up - part 1
c9c935923 Move the check to the beginning of the loop in VersionEdit::EncodeTo()
2110e43a5 Remove an unnecessary include file in version_edit.cc
412b7f85b Include atomic in mock_table.h
c08285334 Include all the mocks
abac3d647 TableMock + framework for mock classes
fb3f8ffe5 Improve the robustness of PartialCompactionFailure test again.
60fa7d136 Improve the robustnesss of PartialCompactionFailure test.
3772a3d09 Fix the bug where compaction does not fail when RocksDB can't create a new file.
c49dedbe0 Merge pull request #366 from fyrz/RocksJava-Backup-Restore-Improvements
a39e931e5 FlushProcess
efa2fb33b make LevelFileNumIterator and LevelFileIteratorState anonymous
f7c973069 [RocksJava] Integrated review comments
7e12ae5a2 [RocksJava] - BackupInfos & Restore-/BackupableDB enhancements
eb357af58 unfriend ForwardIterator from VersionSet
f981e0813 unfriend ColumnFamilyData from VersionSet
834c67d77 rename FileLevel to LevelFilesBrief / unfriend CompactedDBImpl
a28b3c438 unfriend UniversalCompactionPicker,LevelCompactionPicker and FIFOCompactionPicker from VersionSet
5187d896b unfriend Compaction and CompactionPicker from VersionSet
75d7e2c37 Merge pull request #369 from fyrz/Small-Fix
db52419cf Merge pull request #290 from vladb38/master
45e756f04 [RocksJava] Minor correction to the previous pull request merge
f94f1a97d Merge pull request #368 from fyrz/RocksJava-Hardening-RocksIterator
b08c39e14 [RocksJava] RocksIterator: Assert for valid RocksDB instance & documentation
b680033e6 Include atomic in env_test
56ef2caaa [RocksJava] - Hardening RocksIterator
c1c68bce4 remove atomic_pointer.h references
7c303f0e7 Include atomic
179c23021 Merge pull request #351 from fyrz/RocksJava_Snapshot_Support
48842ab31 Deprecate AtomicPointer
f37048ad1 Merge pull request #367 from fyrz/FixBrokenJavaBuild
679a9671f RocksJava Fix after MutableCFOptions change.
714c63c58 db_stress for dynamic options
f1841985e dynamic inplace_update options
a04929aa4 fixed conflict in java/Makefile
a1bae76c8 Integrated changes due to review bei ankgup87
b8ce52648 [RocksJava] Support Snapshots
bc3bc4bc2 Merge pull request #357 from fyrz/JavaTest-Fix
42f0eaceb Merge pull request #354 from fyrz/RocksJava-memtables-3.6
965d9d50b Fix timing
001ce64dc Use chrono for timing
240ed0cd7 Fix uninitialized parameter caused by D24513
724fba2b3 Improve the log in Universal Compaction to include more debug information.
720c1c056 fix erro during merge
b794194ad Remove java build from travis
122f98e0b dynamic max_mem_compact_level
1fee591e7 comments for DynamicCompactionOptions test
574028679 dynamic max_sequential_skip_in_iterations
bd4fbaee3 Fixed cross platform build after introducing Java-7 dependencies
1eb545721 Fix incorrectly merged Java - Makefile
9aa9668a8 [RocksJava] Memtables update to 3.6
4b1786e95 Fix SIGSEGV when declaring Arena after ScopedArenaIterator
05b2e60dd Merge pull request #362 from adamretter/travis-jcheck
9383922cc Added java tests to travis build
90f156402 Fix CompactBetweenSnapshots
2717422e2 Merge pull request #361 from adamretter/fix-yosemite-build
3b5fe3a1f Correct the log message in VersionEdit
c584d2b53 Fix for building RocksDB Java on Mac OS X Yosemite
2a8e5203d db_bench: --batch_size used for write benchmarks too
d755e53b8 Printing number of keys in DB Stats
839c376bd fix table_test
0fd985f42 Avoid reloading filter on Get() if cache_index_and_filter_blocks == false
e11a5e776 Improve the comment of util/thread_local.h
6398e6a6a Fix DeleteFile() + enable deleting files oldest files in level 0
0d3145198 Merge pull request #231 from adamretter/master
a6fb7f312 Fix code review comments raised in https://reviews.facebook.net/D22779
c63494fb6 Tests for ComparatorOptions, Comparator and DirectComparator, and by proxy we also exercise Slice and DirectSlice
5e2527411 Fix code style problems identified by lint
25641bfc9 Fix to memory dealocation when creating a slice from a byte buffer
fc12cb83f Add locking to comparator jni callback methods which must be thread-safe
d6fe8dacc Feature - Implement Java API for Comparator and Slice. Allows use of either byte[] or DirectByteBuffer for accessing underlying data.
700f6ec3f Ignore IntelliJ idea project files and ignore java/out folder
5bfb7f5d0 db_bench: seekrandom can specify --seek_nexts to read specific keys after seek.
ff8f74c20 remove checking lower bound of level size
2dd9bfe3a Sanitize block-based table index type and check prefix_extractor
dbcfe27d6 Merge branch 'master' of https://github.com/facebook/rocksdb
6c6691864 Speed up DB::Open() and Version creation by limiting the number of FileMetaData initialization.
5db9e7664 Fix Mac compile error: C++11 forbids default arguments for lambda expressions
f4363fb81 Fix DynamicMemtableOptions test
8f01bf80c Merge pull request #353 from fyrz/FixBloomfilterRocksJava
ee80fb4b4 Total memtables size counter
c12f571d3 Fix mac compile, second try
d2e60f5ce Fix mac compile
274dc81c9 fix build failure
d6c8dba72 Log MutableCFOptions in SetOptions
4d5708aa5 dynamic soft_rate_limit and hard_rate_limit
065a67c4f dynamic disable_auto_compactions
dc50a1a59 make max_write_buffer_number dynamic
6a150c011 ldb: support --fix_prefix_len
bafbc23ba Filters getting disposed by System.gc before EOL
ca250d71a Move logging out of mutex
5cc9adf5b WriteBatchWithIndex's Iterator bug of SeekToFirst() and SeekToLast()
7658bcc1e Merge pull request #352 from fyrz/OptionsTestMergeProblem
1b97934a2 Options correction
2ef3ed86f Integrated feedback from ankgup87
a40ce219b Adding merge functions to RocksDBJava
cc6c883f5 Stop stopping writes on bg_error_
4942f6efb Merge pull request #350 from fyrz/reenable_accidentally_disabled_test
ee28f431d With the last commit a Test was accidentally disabled. This commit solves this.
b5dd7eed6 Merge pull request #319 from fyrz/column_families_java
18004d2f2 [RocksJava] Column family support
5908d08a0 Merge pull request #336 from fyrz/32BitRocksJavaBug
9e5f2a952 Merge pull request #344 from fyrz/java-doc-enhancements
4f5a68725 32-Bit RocksJava resolution for jlong overflows
16d2ebdbc Minor adjustment to prevent two warnings
70294c911 JavaDoc improvements on RocksJava
833357402 WriteBatchWithIndex supports an iterator that merge its change with a base iterator.
4f65fbd19 WriteBatchWithIndex's iterator to support SeekToFirst(), SeekToLast() and Prev()
f441b273a WriteBatchWithIndex to support an option to overwrite rows when operating the same key
3ead857a0 Fixed Mac compile error in util/options_test.cc
5a7618634 Fixed compile error on Mac: default arguments for lambda expressions
cd0d581ff convert Options from string
f18b4a484 minor update to benchmark script
b7d3d6ebc db_bench: set thread pool size according to max_background_flushes
c5f54a8f3 Merge pull request #339 from fyrz/bloomFilterSupportNewFormat
ced612957 Improved JavaDoc
5e43155b3 RocksJava should support not only BlockBased Bloomfilter
1d525891b Update HISTORY for 3.6
88edfd90a SkipListRep::LookaheadIterator
6a443309d Merge pull request #342 from fyrz/java_makefile_fix
4f272408c RocksJava Makefile includes incorrect paths to version.h
f78b832e5 Log RocksDB version
25f6a852e add db_test for changing memtable size
daab6dc51 Merge pull request #318 from criccomini/master
63eade401 Fix error introduced by merge
ba882972f Merge pull request #340 from nbougalis/nullderef
6bcff9dc2 Merge pull request #341 from fyrz/arc-lint
d6169954b Removed code which prevents `arc lint` from working properly.
b87db0715 Avoid dereferencing a null field
1a1b95347 Merge pull request #270 from tdfischer/check-with-unity
e107b6b2a Merge pull request #337 from fyrz/cross-platform-jdb_bench
70e401a2e Merge pull request #338 from fyrz/rm_obs_code_write_batch_with_index_test
1e5a52815 update release readme
d44871e80 fix java doc directory in git ignore
2a4d6e796 merge master to resolve merge conflicts
6b2c1d962 make publish jni jars depend on release jni jars
1c7c76476 Replaced obsolete comparator with builtin variant.
22c64be43 Cross platform fix for Java benchmark shell script.
81828592b Merge pull request #335 from fyrz/version-script-cross-platform
69d4c5123 Cross-platform fix version.sh
fcd13a77a Merge pull request #334 from fyrz/JavaDoc-Cleanup
bfb0246f5 Merge pull request #331 from fyrz/findbug_issues
05204bb11 Lint changes
da8ff9ff8 Fixed Findbugs issues
a5757ff3c Listing of changes
df3373fbf [Java] Fix compile error on DbBenchmark.java
4eb5a40f7 [Java] Fixed link error on library loading on Mac.
56dfd363f Fix a check in database shutdown or Column family drop during flush.
0e516a75d Fix lint errors in java/rocksjni/options.cc
8ea232b9e Add number of records dropped in compaction summary
f4086a88b perf_context.get_from_output_files_time is set for MultiGet() and ReadOnly DB too.
e869fc6a8 remove proper javadoc directory
c1273533b Merge pull request #333 from nbougalis/cleanups
99744e0c4 bump version to 3.6
45d526e22 singular javadoc directory
378f321da merge master to resolve merge conflicts
a1d3f0d2b don't fail if javadocs diretory doesn't exist
c832f1644 add not about updating pom version and rename pom to be unversioned
a213971d8 Don't return (or dereference) dangling pointer
2a1add673 use proper major/minor/micro version rather than hard coding 3.5.0
2d72f7807 update release docs in java
8322cf000 use javadoc instead of javadocs
079a612b5 Fix build_tools/version.sh
2e8012498 add javadoc and sources targets for sonatype
df08a2d03 add single rocksdbjni pom
deefcf476 make fat jar unclassified to satisfy sonatype
fd2545c80 add maven publication target and instructions
1e9af10ef Merge pull request #330 from fyrz/setFilterPolicyForBBTConfig
017354177 FilterTest
d410b39d5 BlockBasedTableConfig Filter policy support RocksJava
0908ddcea Don't keep managing two rocksdb version
d0916f452 add major minor micro version to java jars
d6987216c Merge pull request #327 from dalgaaf/wip-da-SCA-20141001
25888ae00 Merge pull request #329 from fyrz/master
89833e5a8 Fixed signed-unsigned comparison warning in db_test.cc
fcac705f9 Fixed compile warning on Mac caused by unused variables.
b3343fdea resolution for java build problem introduced by 5ec53f3edf62bec1b690ce12fb21a6c52203f3c8
187b29938 ForwardIterator: update prev_key_ only if prefix hasn't changed
5ec53f3ed make compaction related options changeable
d122e7bcf Update INSTALL.md
9d6f38086 backupable_db_test.cc: pass const string param by reference
8ff0b4095 document_db_test.cc: pass const string param by reference
177caca42 ttl/ttl_test.cc: pass const string param by reference
4a171882d db/version_set.cc: remove unnecessary checks
bf3bfd044 util/cache_test.cc: use static_cast over C-Style cast
86e29f033 document_db.cc: remove unused variable
28a6e3158 table/block_based_table_builder.cc: remove unused variable
091153493 db/db_test.cc: remove unused variable
6b6cedbb1 table/format.cc: reduce scope of some variables
55652043c table/cuckoo_table_reader.cc: pass func parameter by reference
5abd8add7 db/deletefile_test.cc: remove unused variable
d6483af87 db/db_test.cc: reduce scope of some variables
44cca0cd8 db/db_iter.cc: remove unused variable
986dad025 Merge pull request #324 from dalgaaf/wip-da-SCA-20140930
3a0d498a3 rebase master
8ee75dca2 db/memtable.cc: remove unused variable merge_result
0fd8bbca5 db/db_impl.cc: reduce scope of prefix_initialized
676ff7b1f compaction_picker.cc: remove check for >=0 for unsigned
e55aea551 document_db.cc: fix assert
d517c8364 in_table_factory.cc: use correct format specifier
b14037556 ttl/ttl_test.cc: prefer prefix ++operator for non-primitive types
43c789c8f spatialdb/spatial_db.cc: use !empty() instead of 'size() > 0'
0de452ee9 document_db.cc: pass const parameter by reference
4cc8643ba util/ldb_cmd.cc: prefer prefix ++operator for non-primitive types
af8c2b2d9 util/signal_test.cc: suppress intentional null pointer deref
33580fa39 db/db_impl.cc: fix object handling, remove double lines
873f1356a db_ttl_impl.h: pass func parameter by reference
855845714 ldb_cmd_execute_result.h: perform init in initialization list
063471bf7 table/table_test.cc: pass func parameter by reference
93548ce8f table/cuckoo_table_reader.cc: pass func parameter by ref
b8b7117e9 db/version_set.cc: use !empty() instead of 'size() > 0'
8ce050b51 table/bloom_block.*: pass func parameter by reference
53910ddb1 db_test.cc: pass parameter by reference
68ca53416 corruption_test.cc: pass parameter by reference
726ac5bca shrink vagrant commands to single line
a2f98ef61 fix tabs in Makefile
7506198da cuckoo_table_db_test.cc: add flush after delete
1f963305a Print MB per second compaction throughput separately for reads and writes
ffe3d490d Add an instruction about SSE in INSTALL.md
0b923f0f9 add centos 5.6 build instead of ubuntu.
ee1f3ccb0 Package generation for Ubuntu and CentOS
f0f795549 Fixing comile errors on OS X
99fb613e5 remove 2 space linter
b2d64a486 Fix linters, second try
747523d24 Print per column family metrics in db_bench
56ebd4087 Fix arc lint (should fix #238)
637f89179 Merge pull request #321 from eonnen/master
827e31c74 Make test use a compatible type in the size checks.
fd5d80d55 CompactedDB: log using the correct info_log
2faf49d5f use GetContext to replace callback function pointer
6a64ea617 add note about java 7
983d2de2d Add AUTHORS file. Fix #203
c4519c777 fix mis-named jar in JNI loader
abd70c5e1 Merge pull request #316 from fyrz/ReverseBytewiseComparator
2dc6f62bb handle kDelete type in cuckoo builder
8b8011a68 Changed name of ReverseBytewiseComparator based on review comment
b8e26615a since we're not sharing folders with the vm, copy built .so files and jars back to host system.
389edb6b1 universal compaction picker: use double for potential overflow
4e735bb7f Rsync files to VM rather than sync folders, since sync folders was causing clock skew and confusig make.
82a8f43cc Document RELEASE.mdgit status
9db13987b Update RocksDB's Java bindings to support multiple native RocksDB builds in the same Jar file. Cross build RocksDB for linux32 and linux64 using Vagrant. Build a cross-platform fat jar that contains osx, linux32, and linux64 RocksDB static builds.
534048426 Built-in comparator(s) in RocksJava
d439451fa delay initialization of cuckoo table iterator
94997eab5 reduce memory usage of cuckoo table builder
c6275956e improve memory efficiency of cuckoo reader
581442d44 option to choose module when calculating CuckooTable hash
fbd2dafc9 CompactedDBImpl::MultiGet() for better CuckooTable performance
3c6800610 CompactedDBImpl
f7375f39f Fix double deletes
21ddcf6e4 Remove allow_thread_local
fb4a492cc Merge pull request #311 from ankgup87/master
611e286b9 Merge branch 'master' of https://github.com/facebook/rocksdb
0103b4498 Merge branch 'master' of ssh://github.com/ankgup87/rocksdb
1dfb7bb98 Add block based table config options
cdaf44f9a Enlarge log size cap when printing file summary
7cc1ed7f0 Merge pull request #309 from naveenatceg/staticbuild
ba6d660f6 Resolving merge conflict
51eeaf65e Addressing review comments
fd7d3fe60 Addressing review comments (adding a env variable to override temp directory)
cf7ace886 Addressing review comments
0a29ce539 re-enable BlockBasedTable::SetupForCompaction()
55af37075 Remove TODO for checking index checksums
3d74f0997 Fix compile
53b003995 Fix release compile
d0de413f4 WriteBatchWithIndex to allow different Comparators for different column families
57a32f147 change target_file_size_base to uint64_t
5e6aee432 dont create backup_input if compaction filter v2 is not used
49b5f94c5 Merge pull request #306 from Liuchang0812/fix_cast
787cb4db2 remove cast, replace %llu with % PRIu64
a7574d4fa Update logging.cc
7e0dcb953 Update logging.cc
57fa3cc5b Merge pull request #304 from Liuchang0812/fix-check
cd44522a9 Merge pull request #305 from Liuchang0812/fix-logging
6a031b6a8 remove unused variable
4436f17bd fixed #303: replace %ld with % PRId64
7a1bd057f Merge pull request #302 from ankgup87/master
423e52cd4 Merge branch 'master' of https://github.com/facebook/rocksdb
bfeef94d3 Add rate limiter
ed9a2df8c fix unity build
32f2532a0 Print compression_size_percent as a signed int
976caca09 Skip AllocateTest if fallocate() is not supported in the file system
3b897cddd Enable no-fbcode RocksDB build
f44594743 RocksDB: Format uint64 using PRIu64 in db_impl.cc
e17bc65c7 Merge pull request #299 from ankgup87/master
b93797abc Fix build
adae3ca1f [Java] Fix JNI link error caused by the removal of options.db_stats_log_interval
90b8c07b4 Fix unit tests errors
51af7c326 CuckooTable: add one option to allow identity function for the first hash function
035043559 Fixed a signed-unsigned comparison in spatial_db.cc -- issue #293
2fb1fea30 Fix syncronization issues
6f0964e37 Only run make unity on travis instead of make check
9ed1b49a2 Build unity build on make check
ff7689561 Remove some unnecessary constructors
feadb9df5 fix cuckoo table builder test
3c232e164 Fix mac compile
54cada92b Run make format on PR #249
27b22f13a Merge pull request #249 from tdfischer/decompression-refactoring
fb6456b00 Replace naked calls to operator new and delete (Fixes #222)
5600c8f6e cuckoo table: return estimated size - 1
a062e1f2c SetOptions() for memtable related options
e4eca6a1e Options conversion function for convenience
a7c209452 Merge pull request #292 from saghmrossi/master
4d0523452 Merge branch 'master' of github.com:saghmrossi/rocksdb
60a4aa175 Test use_mmap_reads
94e43a1df [Java] Fixed 32-bit overflowing issue when converting jlong to size_t
f9eaaa66e added include for inttypes.h to fix nonworking printf statements
f090575e4 Replaced "built on on earlier work" by "built on earlier work" in README.md
faad439ac Fix #284
49aacd8d2 Fix make install
acb9348ff [Java] Include WriteBatch into RocksDBSample.java, fix how DbBenchmark.java handles WriteBatch.
4a27a2f19 Don't sync manifest when disableDataSync = true
9b8480d93 Merge pull request #287 from yinqiwen/rate-limiter-crash-fix
28be16b1d fix rate limiter crash #286
04ce1b25f Fix #284
add22e351 standardize scripts to run RocksDB benchmarks
dee91c259 WriteThread
540a257f2 Fix WAL synced
24f034bf4 Merge pull request #282 from Chilledheart/develop
49fe329e5 Fix build issue under macosx
ebb5c65e6 Add make install
0352a9fa9 add_wrapped_bloom_test
9c0e66ce9 Don't run background jobs (flush, compactions) when bg_error_ is set
a9639bda8 Fix valgrind test
d1f24dc7e Relax FlushSchedule test
3d9e6f775 Push model for flushing memtables
059e584dd [unit test] CompactRange should fail if we don't have space
dd641b211 fix RocksDB java build
53404d9fb add_qps_info_in cache bench
a52cecb56 Fix Mac compile
092f97e21 Fix comments and typos
6cc12860f Added a few statistics for BackupableDB
0a42295a2 Fix SimpleWriteTimeoutTest
06d986252 Always pass MergeContext as pointer, not reference
d343c3fe4 Improve db recovery
6bb7e3ef2 Merger test
88841bd00 Explicitly cast char to signed char in Hash()
52311463e MemTableOptions
1d284db21 Addressing review comments
55114e7f4 Some updates for SpatialDB
171d4ff4a remove TailingIterator reference in db_impl.h
9b0f7ffa1 rename version_set options_ to db_options_ to avoid confusion
2d57828d0 Check stop level trigger-0 before slowdown level-0 trigger
659d2d50c move compaction_filter to immutable_options
048560a64 reduce references to cfd->options() in DBImpl
011241bb9 DB::Flush() Do not wait for background threads when there is nothing in mem table
a2bb7c3c3 Push- instead of pull-model for managing Write stalls
0af157f9b Implement full filter for block based table.
9360cc690 Fix valgrind issue
02d5bff39 Merge pull request #277 from wankai/master
88a2f44f9 fix comments
7c16e3922 Merge pull request #276 from wankai/master
823773837 replace hard-coded number with named variable
db8ca520b Merge pull request #273 from nbougalis/static-analysis
b7b031f42 Merge pull request #274 from wankai/master
4c2b1f097 Merge remote-tracking branch 'upstream/master'
a5d286307 typo improvement
9f8aa0939 Don't leak data returned by opendir
d1cfb71ec Remove unused member(s)
bfee319fb sizeof(int*) where sizeof(int) was intended
d40c1f742 Add missing break statement
2e97c3898 Avoid off-by-one error when using readlink
40ddc3d6c add cache bench
9f1c80b55 Drop column family from write thread
8de151bb9 Add db_bench with lots of column families to regression tests
c9e419ccb rename options_ to db_options_ in DBImpl to avoid confusion
5cd0576ff Fix compaction bug in Cuckoo Table Builder. Use kvs_.size() instead of num_entries in FileSize() method.
0fbb3facc fixed memory leak in unit test DBIteratorBoundTest
adcd2532c fix asan check
4092b7a0b Merge pull request #272 from project-zerus/patch-1
bb6ae0f80 fix more compile warnings
6d3144118 Merge pull request #271 from nbougalis/cleanups
0cd0ec4fe Plug memory leak during index creation
4329d74e0 Fix swapped variable names to accurately reflect usage
45a5e3ede Remove path with arena==nullptr from NewInternalIterator
5665e5e28 introduce ImmutableOptions
e0b99d4f5 created a new ReadOptions parameter 'iterate_upper_bound'
51ea88900 Fix travis builds
a4816269f Relax backupable rate limiting test
f7f973d35 Merge pull request #269 from huahang/patch-2
ef5b38472 fix a few compile warnings
2fd3806c8 Merge pull request #263 from wankai/master
1785114a6 delete unused Comparator
1b1d9619f update HISTORY.md
703c3eacd comments about the BlockBasedTableOptions migration in Options
4b5ad8865 Merge pull request #260 from wankai/master
19cc588b7 change to filter_block std::unique_ptr support RAII
9b976e34f Merge pull request #259 from wankai/master
5d25a4693 Merge remote-tracking branch 'upstream/master'
dff2b1a8f typo improvement
343e98a7d Reverting import change
ddb8039e3 RocksDB static build Make file changes to download and build the dependencies .Load the shared library when RocksDB is initialized
REVERT: 1fdd726a8 Hotfix RocksDB 3.5
REVERT: d67500a59 Add `make install` to Makefile in 3.5.fb.
REVERT: 4cb631aa0 update HISTORY.md
REVERT: cfd0946be comments about the BlockBasedTableOptions migration in Options

git-subtree-dir: src/rocksdb2
git-subtree-split: aead4041720f40e305e7f9d432875749d91d5a2d
2017-12-02 13:44:49 -05:00
Brad Chase
3d6ea6737c Update soci unity build 2017-12-02 13:44:12 -05:00
Brad Chase
dc5f40bf62 Add generated soci-config.h 2017-12-02 13:44:12 -05:00
Brad Chase
1882bbdae7 Update soci 2017-12-02 13:44:06 -05:00
Brad Chase
855681a40b Squashed 'src/soci/' changes from b2855dce5..79e222e3c
79e222e3c [Travis] Switch to latest Oracle downloader v2.0.3
3992cd8f6 [Travis] Try travis-oracle downloader hotfix
958f1b010 [Travis] Use only ORACLE_LOGIN_userid and ORACLE_LOGIN_pass
f03569ffe [Travis] Oracle has changed login form
682486db9 Merge pull request #607 from rocksolidwebdesign/master
e8ed51c37 use fully qualified soci::session to fix build on macosx sierra
c98c877bc [doc] Add note on factory objects not exported from DLLs
d0552e64a Catch unhandled exception in PostgreSQL ODBC unit test
643853322 Add missing -DWITH_POSTGRESQL=ON to scripts\build.bat
82867ced7 [docs] Clarify name of MySQL package with client library
a35851478 Complete scripts\built.bat with basic CMake options.
225d5a3a4 [CMake] Simplify warning messages
d90a7acfc [docs] Fix CMake options for SQLite 3
3d32f5253 Merge pull request #596 from mloskot/ml/macro-soci-override
5d980ee98 Merge pull request #600 from mloskot/ml/cmake-update-FindDB2
05ab0bcf6 Add SOCI_OVERRIDE macro as conditional C++11 override specifier
aa9744e0a [DB2] Add additional install paths to search for DB2 client
3069f242b Merge pull request #594 from mloskot/ml/cmake-postgresql-nosinglerow
a5bb763ab Merge pull request #599 from mloskot/ml/cmake-drop-cdash
466fdf32c [docs] Update name DB2 driver/client library name
a7f5240a1 [CMake] Add SOCI_POSTGRESQL_NOSINLGEROWMODE option (default OFF)
601a89227 [CMake] Remove CDash remains
fffe6d9c8 [CMake] Tidy up FindOracle.cmake formatting
6157f8d97 [CMake] Add /usr/lib/oracle/*/client${LIB_SUFFIX} to searched paths
bbc18d73d [firebird] Add Ubuntu firebird-dev package as dependency example
fd18f5c6c [CMake] Determine LIB_SUFFIX from SOCI_TARGET_ARCH_X64
c5f59c9d3 [Travis] Revert remove of any notifications to soci-devel mailing list
f7f2a6123 [Travis] Remove any notifications to soci-devel mailing list
6df733826 [docs] Add pages for Ada language bindings
41f288bf3 Merge pull request #595 from mloskot/ml/db2-null-dereference
62f4bf5c4 [db2] Fix null pointer dereference
340a72eda [CMake] FindPostgreSQL.cmake update lost POSTGRESQL_VERSION return
5da552537 [docs] Remove Ada pages from TOC due to building failure
5e0e70c95 [docs] Add copyright footer
ea0f6c11e [docs] Add languages/ada pages to TOC
2ab0fb901 Add CHANGELOG.md generated from GitHub
961368043 Add changelog.sh utility to generate CHANGELOG.md from GitHub
0621bd610 [Circle] Fix missing checkout in workflow
2c45f2bc5 [docs] Fix issues reported by markdownlint
46071f5f6 [docs] Replace HTML tables with Markdown tables
80a86c0eb Format compact table in README
14b693e64 [docs] Avoid markdownlint warnings about line length
13031a831 [docs] Fix README.md linting issues
2d575b8d7 [docs] Add link to master branch docs on docs.html
8427054b7 [Circle] Fix markdownlint invocation
cc75d6e09 [Circle] Fix global installation of markdownlint-cli
522c7bbb6 [docs] Add .markdownlint.json configuration file
498381595 [AppVeyor] Add build job with Visual Studio 2017 (#588)
fd37e3b80 [AppVeyor] Update badge after switch to organizational account
51d058f04 Repair VC8 compile of test applications (#593)
c2310cf54 [Circle] Add markdownlint and define lint-build-deploy workflow for docs
b9f4007e4 Fix __GNUC__ condition ignoring -Wmaybe-uninitialized
08fe54076 Ignored -Wmaybe-uninitialized in GCC all versions following 4.7
29be6f860 Merge pull request #592 from mloskot/ml/replace-alternative-tokens
a161e24c1 Replace C alternative boolean operator representations with primary tokens
07a23d8a2 [CircleCI] Disable gem. Requires separate docker w/ Ruby
df87eb1eb [CircleCI] Install markdownlint gem
92de7a387 [Vagrant] Fix bach if. Silence wget. Add logging.
6b9531179 [Vagrant] Update setup with primary VM and optionals.
4cb3b636f [Travis] Disable clang jobs to save on built times
4e98cd2cf [docs] Add MySQL 8.0.1 on Windows to tested versions
275965736 Add CMake options to disable/enable individual backends.
cba05c702 Add build.bat for convenience of Visual Studio users
67c479ed9 Merge pull request #590 from mloskot/ml/cmake-update-FindPostgreSQL
35d2df6f4 [cmake] Update FindPostgreSQL.cmake from CMake 3.9
4fdfbf817 [cmake] Improve testing of Abc_FOUND and ABC_FOUND variables
3f006d3ae Fix copy-paste typo error in README
bd3ee7a48 [CI] Fix webhook specs and events
d31d17be3 [CI] Fix webhook URLs and use default events
34ced08e3 Fix URLs for badges in README
1653bd0b7 Update badges in README
7a4704be3 [Travis] Update webhook for notifications on gitter.im/SOCI/soci
17167fd97 [AppVeyor] Update webhook for notifications on gitter.im/SOCI/soci
95e55689e [Circle] Update webhook for notifications on gitter.im/SOCI/soci
68b914b19 Add footer: BSL © Maciej Sobczak and contributors.
8bf913e9c [Circle] Add webhook to receive notification on Gitter chat
25b2ab9af Actually link the generated docs
4aa8d5ab9 Link docs generated for master as (development)
a40420e4c [Circle] Use API token for the badge
fc59a621c [Circle] Fix status badge
15eba50e8 Add CircleCI setup to build docs and deploy to SF.net
3fa0a58cb [EditorConfig] Add settings for YAML and CMake scripts
b36f437bc Merge branch 'more-xml'
b85ec90c6 Check that inserting malformed XML values fails
2b89a134a Enable XML/CLOB tests for PostgreSQL via ODBC
6e4732102 Add very simple XML and CLOB support to ODBC backend
fabb8f0f0 Make XML comparison in the test work for SQL Server too
11a8982d5 Add a helper function for copying strings in ODBC backend too
556f5d336 Add exchange_type_traits<> specializations for XML and CLOB
f1243acdf Add helper function for copying string to PostgreSQL backend
5e6f37e35 Document types used for XML and CLOB support in Firebird backend
bd236c8c8 Merge branch 'xml-firebird'
e6e1ad305 Merge pull request #585 from mloskot/ml/enable-oracle-for-pulls
0cadedaf5 Enable Oracle builds for pull requests
93e844d02 [Travis] Update Oracle installation setup
3a82540aa Also throw if unsupported type is used for vector "into" in DB2
0f1756c55 Throw if unsupported type is used for vector "into" parameter
3a7677425 Merge branch 'odbc-vec-into-null'
ff2464925 Merge branch 'oracle-fix-null-str'
023f4df03 Skip running Oracle tests for PR builds on Travis CI
df4039923 Fix bug with null strings in bull selections in ODBC and DB2
6ca9cd64a Add REQUIRE_NOTHROW() around SQL statements in the string test
63bcc8944 Add workaround for empty strings being null in Oracle
2d74f9aaa Merge pull request #579 from mloskot/ml/travis-oracle-restore
c3b943c87 Restore Oracle build job on Travis CI
9943459f4 Add support for XML type to Firebird backend too
059faf5d9 Add CLOB support to Firebird backend
85159db6f Refactor: move XML and CLOB tests to common code
4744c1705 Fix the build after the previous commit
201c89f0e Fixed wrong handling of errors in single-row mode for PostgreSQL.
e26d134b4 Merge pull request #577 from mloskot/ml/bin-to-scripts
6363537e6 Add Windows shell scripts with CRLF to .editorconfig
d38b21ca6 Move directory /bin to /scripts
316885a37 Merge pull request #576 from alexott/editorconfig
461a44585 Update .editorconfig with comments from review
2c7c312d7 Add support for EditorConfig
09c26314d Allow defining SOCI_POSTGRESQL_NOSINLGEROWMODE for PostgreSQL < 9
4b451971a Avoid using overloaded virtual methods in vector type backends
f6980d2ce Merge pull request #572 from alexott/coverity-fixes
23d2f2593 fix some of warnings from fresh Coverity report
25eb5bdb3 Initialize indISCHolder_ member in firebird_into/use_type_backend classes
042dc0fd9 Merge pull request #570 from mcencora/master
9d82ee8e7 Remove unnecessary namespace qualification
3db8f4371 cmake: export SOCI public include directories
e4ab099df Fix undefined sanitizer warnings
8fdb2e7ea Merge pull request #566 from mloskot/ml/mkdocs
80941a9d9 Update structure of SOCI 4.0.0 documentation
9c8191903 Fixed Makefile build for PostgreSQL (broken by other modifications).
e7db59c74 Merge pull request #557 from msobczak/master
f6d08a75f Merge pull request #564 from Arenoros/master
43229ecfe Fix types for SQLite3 backend
ed25eea2c Fix session::get_table_names() for sqlite backend
7a6559c18 Merge branch 'odbc-string-len'
c6aef12b1 Add test checking string length when using bulk insert
51c4c1e1a Fix the length of strings in DB2 vector operations
be4f26fea Fix the length of strings in ODBC vector operations
521a84085 Improve get_affected_rows() documentation
e7e7fb78a Return -1 from get_affected_rows() in ODBC backend if unknown
97a4728cb Remove loop that could be never executed from ODBC code
1a0eeb186 Fix harmless variable shadowing warnings
e7cdd4754 Fix loss of error message in ODBC backend when using vectors
06915e483 Add check for absence of unneeded truncation too
82038018c Merge branch 'refactor-parse-std-tm'
56bc9f9c5 Return int, not long, from parse10() helper
281f54f51 Check for negative date components in parse10()
f2e84c4bf Don't pass exception message to parse10() helper
a1e07e389 Avoid duplicate code for parsing dates in different backends
e2f1ca530 Fix libraries link order when linking tests statically
a3685fd09 Fix unused parameter warnings in Oracle backend code
488825079 Explicitly use Ubuntu Precise for Travis builds
638b89d1f Merge remote-tracking branch 'upstream/master'
138b29e43 Lazy initialization of the temporary LOB objects for Oracle.
a9ab0db16 Merge pull request #541 from ckaminski/master
f938c0133 Get rid of allowed failures in Travis CI configuration
52539f8ab Merge branch 'odbc-mysql-bug'
2aeb32872 Fix reading from unallocated memory in ODBC with MySQL
dc05c8e82 Use switch over exchange type in ODBC vector backend code
7bd491cf2 Fix check for clang in CMake configuration
702c214a9 Remove dummy stream insertion overload for boost::optional
1b416275d Remove Travis CI builds using Postgression
d7c986c6b Disable MySQL unit test currently always resulting in a crash
2f65f02e8 Define SOCI_SQLITE3_SOURCE in all SQLite3 backend sources
4f1fbfcd4 Suppress memory leak report in glibc NSS under Ubuntu
d91c2228c Add get_dummy_from_{table,clause}() methods
78bedc931 Consistently check for connection in the session object methods
cf9cfa90c Fix DB2 backend build after x_xmltype and x_longstring additions
4b642ac5c Fixed handling of BINARY_DOUBLE in dynamic row.
2d621e8f3 Fixed memleak when reusing into and use elements.
9bb0f70b6 Revert "Fix inserting strings longer than 8000 bytes with ODBC/MS SQL"
7c28bd6c3 Fix inserting strings longer than 8000 bytes with ODBC/MS SQL
b510e1a63 moved to latest Catch UT Framework (#544)
33ae81a71 Merge pull request #545 from snikulov/odbc_handle_defaults
92afc1f91 Use defaults for unsupported types in ODBC
7fba06984 disable autosetting of SOCI_CXX_C11 to OFF
fa0e5b5a2 Merge pull request #536 from ddowling/master
723230fcc Support the OCI_SUCCESS_WITH_INFO return code in get_error_details
f21578e2a Fix an uninitialised variable access when using the Oracle backend with a database type not in the standard accepted set.
a912148f9 Fix uninitialized procedure::gotData_
646efa070 Merge pull request #534 from mutcher/master
87509d52d Corrections in README.md
34c534722 Export struct sqlite3_soci_error from DLL on Windows.
840100102 Update .gitignore patterns for Visual Studio (Code)
dc87a2c58 Docs markdown cleanup.
e719c44bb Merge pull request #524 from ravselj/firebird_blob_bug
2edb58395 Firebird backend blob write bug fix.
0a88814c2 Added Firebird specific test indicating BLOB write bug.
6eb1a3e97 Fixed handling of empty long strings.
e7194837e Added empty_blob() and nvl() to portable utilities.
adae5ade8 Merge pull request #509 from msobczak/xml_clob
45eaff9e9 Added handling of big string in DDL.
9b2d48f3b Merge pull request #510 from mloskot/ml/add-missing-decl
8ae0f3962 Add missing declspec to properly export exception classes.
f7b11ca71 Fixed handling of dt_xml in sqlite3.
1655a195e Added bigstring (XML and CLOB) support.
81a48efaf Fixed missing space in DDL.
d68170494 Fixed handling of DDL alter column syntax.
1eb4c2935 Merge pull request #508 from msobczak/blob_offset
4b0dbab3e Added uniform offset for BLOB read/write operations.
c5e37d273 Merge pull request #503 from wisk/cmake-package-export
e082831ce export cmake package
ba9963779 travis-ci: Allows to fail for odbc and mysql
92c430669 Merge pull request #454 from snikulov/enable_asan
ec746bd1d Merge pull request #375 from ravselj/sqlite_load_one
88b331507 Merge pull request #487 from msobczak/bulkiterators
780a5c5d2 Corrected handling of vectors with user-defined types.
9e83a50c3 Merge pull request #494 from snikulov/fix_travis
da09c7c6b Fix travis-ci build
307d94caf Merge pull request #492 from snikulov/av_odbc_driver_name
12708b190 Merge pull request #491 from snikulov/av_mingw_fix
4a0ac2ba3 Fixed driver name and passwd for PostgreSQL ODBC driver on AppVeyor
783e8b089 Fixed MinGW build with proper URL
0142d057f Merge pull request #489 from snikulov/fix_383
0723512f3 fixed #383
8d2901675 Added unit tests and docs for bulk iterators.
2f8a7a05e Added bulk iterators for Oracle and PostgreSQL.
2fcad8b24 Merge pull request #486 from msobczak/failover
002f6a9a7 Added docs for failover.
3daef94c7 Added failover callback for Oracle and PostgreSQL.
a0b29bc94 Merge pull request #484 from msobczak/portable-DDL
c959c3812 Added Oracle unit tests for DDL and metadata.
2a8e8965e Added PostgreSQL unit tests for DDL and metadata.
cbee6429a Typo fixes and improved docs.
4244fae07 Merge pull request #482 from msobczak/PostgreSQL-singlerows
8d2a75baa Added support for portable DDL statements.
a0917b7c8 Fixed multiple definitions bug with column_info.
596c6762b Merge pull request #480 from msobczak/metadata
b70a4e8b2 Added singlerows mode for PostgreSQL.
816839514 Merge remote-tracking branch 'upstream/master' into metadata
30aca6230 Added docs for error categories.
f2b2f4bc0 Added more error codes for error categories in Oracle.
a05b61f1f Added docs and removed impractical function for metadata queries.
3c3fec623 Portable metadata queries.
cd26169ef Added (partial) handling of OCI_SUCCESS_WITH_INFO in Oracle.
21a062023 Merge pull request #479 from snikulov/fix_478
bb8aaca05 fixed #478
376362493 Merge pull request #474 from wdavilaneto/master
23967e281 Merge pull request #475 from snikulov/cov_fixes
ec65da5a5 Fixed uninited local std::tm structs
8cb99538c adding oracle remote connection exemple without tnsnames.ora
56379eb81 Merge pull request #471 from snikulov/fix_travis
25ac65fc5 Travis-CI: allowed failure for DB2 backend
8d8002b7b Merge pull request #470 from snikulov/fix_appvr
2895c7d06 Appveyour now using MySQL 5.7
d419e945a Update AppVeyor badge URL
5c14d49cc Remove info on binding char[] support.
57db62210 Added basic support for error categories.
a1d45503b Documented Oracle wallet authentication option.
cae51eec9 Added Oracle wallet authentication.
8c6b47977 Added NLS support for connection parameters in Oracle.
912649c2c Merge pull request #459 from shelomentsevd/master
8b70e0983 Documentation. exchange.md cleaned from typos
c80646c0d Merge pull request #455 from snikulov/msvc2012_fix
3470079d1 compilation fix for msvc2012
d27fba19a Merge pull request #453 from snikulov/catch_updates
36e8fc91b updated catch.hpp version to 1.3.3
56b0b02b8 added Address Sanitizer to expose memory errors
f26138913 Merge pull request #452 from vadz/cxx11
c07a42da0 Merge pull request #451 from vadz/config-vs-platform
ec7d2dfc3 Define cxx_details::auto_ptr<> to get rid of preprocessor checks
e06bfe96e Include soci/soci-platform.h instead of soci-config.h
6fbd8481e Avoid recursive inclusion between soci-{config,platform}.h
6caeb0b50 Merge pull request #449 from snikulov/appvr_up
78ea85ca1 fixed #447
ffc564b70 Use noexcept(false) for once_temp_type dtor with MSVS 2015 too
5e3813f50 Use C++11 deleted members for SOCI_NOT_{ASSIGNABLE,COPYABLE}
449e743b9 Merge pull request #446 from ArnaudD-FR/version
46513d877 CMake read version from version.h
a8ab4c88b Revert "CMake generates soci/version.h"
6f68a7f00 Clean up already implemented features from TODO.
38d405935 Vagrant: update usage and development workflow.
c2a387b2f Vagrant: correct DB2 database lookup for existing soci test db.
f856f7730 Vagrant: fix accidental syntax error
fb545a9b8 Vagrant: update usage documentation
bd4d11d34 Add #define SOCI_HAVE_CXX_C11 mapped from with CMake SOCI_CXX_C11 option.
e476b9d93 Vagrant: build with SOCI_CXX_C11=ON
c6deb5d7e Vagrant: script formatting [ci skip]
f4a47bfcb Vagrant: correct connection strings for tests
9b10e0ba2 Vagrant: set DB2 connection string for the backend tests.
a84dade10 CMake: add boost_message_value as counterpart of boost_report_value.
540d33272 Vagrant: add SOCI_HOST variables for VMs networking
d9e6d687a Merge pull request #438 from mloskot/db2-sqldriverconnect
da61b6379 DB2: document option db2_option_driver_complete
838b99b62 DB2: Add sample SQLDriverConnect connection string
5d1ab2c77 DB2: Switch session from SQLConnect ot SQLDriverConnect
ba8b1aa24 DB2: SQLExecute return code was not tested for SQL_NO_DATA.
837e88dec Vagrant: reorder booting to load DBs before dev env
8afea2f90 Vagrant: add SOCI_DB2_USER and SOCI_DB2_PASS variables
4a576a0c2 Vagrant: switch back to building from /vagrant share
cd167df2e Vagrant: enable host to db2 networking on port 50000
7903d8e1a Vagrant: document networking support
04a59095d Vagrant: increase vm memory to 1024M
98ae7b206 Vagrant: try to avoid re-creation of soci DB if it exists
48a9c9797 Vagrant: do not run tests during provision
29c429ce9 Merge branch 'master' of https://github.com/SOCI/soci
d6182bd9b Vagrant: add Avahi/MDNS to support resolving .local hostnames
c22bfbcc1 Merge pull request #436 from ArnaudD-FR/test_version
094e19c68 CMake generates soci/version.h
2104cc9a6 Merge pull request #435 from ArnaudD-FR/test_definition
6908e473f Specify definitions in soci-config.h
634b2bf84 Vagrant: update DB2 CLI download steps
137efc496 Vagrant: add DB2 CLI client to soci.vm
435547064 Vagrant: define SOCI_* env variables via /etc/profile.d
569e8ecea CMake: add DB2 CLI driver locations
521f83cc0 Vagrant: correct comment [ci skip]
e50025ea9 Vagrant: install zip [ci skip]
a3e3f9671 Merge branch 'master' of https://github.com/SOCI/soci
6043cdf54 Vagrant: Add VM provisioned with DB2 instance and sample database
3a5066d9b Travis: remove sourcing of common.sh as unnecessary
baad25922 Spell-check and formatting
111f5644d Preserve LF or CRLF for certain file types.
016687f03 Vagrant: explain debconf issue while installing Firebird
8a04c61c4 Merge pull request #434 from mloskot/master
01b9b48a3 Add Vagrant configuration
0ba242f51 Merge pull request #430 from ArnaudD-FR/sqlite3_reset
d219a5790 [SQLITE3] Add reset API to reset sqlite3 statement
fa708ad9f Merge pull request #428 from MonsieurNicolas/affectedRowFixSqlite
8cd3cf584 added test that reuses prepared statement
6af865fce reset count of affected rows (sqlite3)
149c53a43 Merge pull request #427 from snikulov/pr_coverity_init
0cc5c50ab - Fixed Coverity issues for SQLite3: 12644, 12645, 12646 - Fixed gcc/clang warns produced with -Wextra switch for SQLite3
9476a7a53 Merge pull request #426 from snikulov/pr_odbc_enable
86890bd2b Enabled mysql/postgresql - fixed #407
0232debc8 Merge pull request #425 from snikulov/cmake_win_cxx11
c2b37f0d2 moved SOCI_CXX_C11 define to common section
91fb559f0 Merge pull request #422 from snikulov/pr_374_fix
2af6c5efc Initialize rowid_backend-derived classes members in ctor
10e3c768a Avoid harmless Coverity dead code warning in MySQL code
4fd5c965a If prepare, execute, fetch... scenario is used and fetch() is called after gotData=false was already returned it begins another loop instead of returning false again indicating that all the data was already read. The following backend fix resolves this issue.
e07f634a7 Added additional basic functionality test from prepare, execute, fetch scenario.
5b96f8c69 Clang: fixed #374
4576731e6 Disable email notifications to soci-devel
35bde9728 Merge pull request #420 from jeking3/feature/postgresql-uuid-unit-test
ddde1f4c8 Merge pull request #421 from jeking3/bugfix/test-access-dsn-file-ignore
315b2cb5b add ignore of cmake generated dsn file for access
60abeb88c add a unit test for the postgresql uuid data type
b1f730cd8 Merge pull request #419 from snikulov/pr_valgrind
8f51034cf Travis-CI: added valgrind check
185593f91 Merge pull request #418 from hw-dwalter/master
b63e850cc fixed get_affected_rows() to pass test and fixed formatting
1ebc2661c Update .travis.yml to change coverity invokation branch
ae3c4d2d8 Merge pull request #414 from hw-dwalter/master
90097ec73 added test to demonstrate bug #221 and make it valid with all backends
b86a7b8ca Update README.md
6fe0a64bd Merge pull request #417 from snikulov/coverity_part_2
dff81b73e coverity: updated .travis.yml
8c0f35e21 Merge pull request #415 from snikulov/pr_413_rework
59d2cf016 Merge pull request #416 from snikulov/pr_travis_upd
1fa1a9413 Travis-CI: try add Coverity tool
63ef39126 AppVeyor: enable MSSQL ODBC connection
7aab87e8d Merge pull request #410 from snikulov/pr_appvr_mysql
d36b06e26 Appveyor: added build for MySQL backend
8a67dbc63 Merge pull request #409 from snikulov/pr_appvr_postgres
c0de792ae Appveyor: added PosgreSQL 9.4 backend
58975c6aa Merge pull request #408 from snikulov/pr_appvr_sqlite
d0f6dec5f Appveyor: Added SQLite to build
4d2e19533 Merge pull request #406 from snikulov/appvr_db_services
f7278ce54 Merge pull request #405 from snikulov/pr_odbc_test_reorg
629e64231 appvr: added db services to build workers
9d06e47a7 ODBC: test updates
14dfe5195 Appveyor: run ctest in test_script step
0530f6fec Avoid use of POSIX strptime function
68a335707 Add some variations for tests for use with indicators.
0d84fef30 Add std::tm checks to the use with indicators tests
985571234 Add tests for some variants with char* query
0b2837def Add test for query string as char*
3a49359bd Request CATCH to disable all the C++11 features.
24cbf26cd Merge pull request #403 from DraconPern/minorcmakefix
4fab0bb33 fixed typo
0e8223f89 Notify Gitter on every build
474c44443 Fix Gitter invalid token error
1eac4da5d Merge pull request #401 from snikulov/pr_appvr_boost
da59e623d Merge pull request #400 from snikulov/pr_boost_opt
c02d65340 Merge pull request #402 from ArnaudD-FR/master
09d6f9c81 Name struct to avoid clang error/warning (issue #374)
87e1c8384 Appveyor: added Boost libraries to build
063e75374 fix for #370
1fd43ebf6 Fix accidentally broken YAML syntax
87492930c Add Gitter integration webhooks for Travis CI and Appveyor
21ba1ddc8 Merge pull request #398 from snikulov/mingw_final
2060fa062 Merge pull request #399 from SOCI/snikulov-patch-1
c683042f6 Update README.md
59c49ed50 mingw: not use ms extension for MinGW G++ compilation
492d04249 Merge pull request #397 from snikulov/make_verbose
e3e093b2e cmake: generate verbose build
903590290 Merge pull request #395 from snikulov/av_warn_fix
d1fd85b45 connection-pool: moved common part out of #ifdef
6c8b6cd42 Switch Appveyor URL from snikulov/soci to mloskot/soci.
b6443a4a2 Restore using -pedantic but use -Wno-pedantic-ms-format too
8fe5d2a1d Initialise in/out pos variable
96783407e Update connection-pool.cpp
21357eda7 Merge pull request #391 from snikulov/pr_appveyor_init
443beaa0b Don't add "-fPIC" explicitly, CMake does it automatically
e607b1c73 Don't use "-pedantic" option when building with g++
56207063b initial integration with AppVeyor-CI
8fda2bb37 boost: disable autolink, because soci controls required libs itself
7747cab51 updated to latest Catch UT framework to work with MinGW with PR #496
bb79b6074 cmake: updated FindODBC.cmake to search right sqlext.h on Windows
64dfb79f7 Merge pull request #381 from ravselj/sqlite_bug
23d398994 sqlite3_close moved after sqlite3_errmsg is called.
d0f718350 Merge pull request #380 from dgrafe/subproject_integration_fixed
f2261c40a Build system: Replacing CMAKE_SOURCE_DIR with CMAKE_CURRENT_SOURCE_DIR

git-subtree-dir: src/soci
git-subtree-split: 79e222e3c2278e6108137a2d26d3689418b37544
2017-12-02 13:44:06 -05:00
Brad Chase
db066da443 Update sqlite to 3.21.0 2017-12-02 13:43:49 -05:00
seelabs
090d813065 Set version to 0.90.0-b1 2017-12-01 14:15:09 -05:00
Scott Schurr
722917e9a6 Improved error message on mistyped command [RIPD-1527]:
Previously if you mistyped the "submit_multisigned" command as
"submit_multisign", the returned message was "Internal error".  Not
very helpful.  It turns out this was caused by a small amount of
code in RPCCall.cpp.  Removing that code improves two situations:

1. It improves the situation with a mistyped command.  Now the
   command returns "Unknown method" and provides the string of
   the mistyped command.

2. The "transaction_entry", if properly entered in its command
   line form, would fire an assert.  That assert is now removed.

In the process, it was discovered that the command line form of
the "transaction_entry" command has not worked correctly for at
least a year.  Therefore support for that the command line form
of "transaction_entry" is added along with appropriate unit
tests.
2017-12-01 14:15:05 -05:00
seelabs
8a02b76d73 Fix snprintf warning 2017-12-01 14:15:05 -05:00
Brad Chase
3e483cfa4c Improve JSON logging 2017-12-01 14:15:05 -05:00
JoelKatz
c42ea14531 Relay untrusted validations/proposals from cluster members (RIPD-1549)
It is common for a validator operator to connect their validator
only to other nodes under their control, using clustering. Relaying of
untrusted validations and proposals must be unreliable to prevent
denial of service attacks. But currently, they are unreliable even
within a cluster.

With this change, a cluster member's decisison to relay (or originate)
a validation or proposal is honored by other cluster members. This
ensures that validators in a cluster will get reliable relaying to
hubs outside the cluster, even if other members of the cluster do not
have that validator on their UNL.
2017-12-01 14:15:05 -05:00
Edward Hennis
a30af2ac89 Disable slow Travis builds
* Changes to Travis seem to be making these builds slower, and are
  resulting in spurious timeouts. Remove the builds that fail more
  often than not until we complete our migration to another build
  system.
* Use CMake for unity builds, instead of the slow build.
2017-12-01 14:15:05 -05:00
Brad Chase
83dc45db62 Default account_tx RPC to validated ledger range (RIPD-1544) 2017-12-01 14:15:05 -05:00
Howard Hinnant
3757829f8e Convert more timing code to <chrono>
* LoadMonitor
* Entry
* Import
2017-12-01 14:15:05 -05:00
Scott Schurr
db17ae8997 Increase multisign test coverage 2017-12-01 14:15:05 -05:00
Howard Hinnant
cd19246937 Enable batch rpc processing
* Can be exercised from the command line with json2

* Rewrite Env::do_rpc to call the same code as
  rpc from the command line.  This puts rpc
  handling logic in one place.
2017-12-01 14:15:05 -05:00
Mike Ellery
d61436cca5 add deprecation warning flag (CMake) 2017-12-01 14:15:05 -05:00
Howard Hinnant
4e6c8d8b35 Remove use of deprecated behavior involving copy members
*  If any of the destructor, copy assignment or copy constructor
   are user-declared, both copy members should be user-declared,
   otherwise the compiler-generation of them is deprecated.
2017-12-01 14:15:05 -05:00
Howard Hinnant
16acba1636 Remove use of deprecated behavior involving copy members in beast
*  If any of the destructor, copy assignment or copy constructor
   are user-declared, both copy members should be user-declared,
   otherwise the compiler-generation of them is deprecated.
2017-12-01 14:15:05 -05:00
Mike Ellery
50b35e2090 Remove hardcoded ports in several unit tests:
Fixes: RIPD-1522
2017-12-01 14:15:05 -05:00
seelabs
e8d02c1333 Refactor FeatureBitset:
* Remove composite helper functions
* Add set difference and Bitset/uint256 operators
* Convert tests to use new feature bitset set difference operator
2017-12-01 14:15:05 -05:00
Scott Schurr
3523cee63d Minimize use of jtx::with_only_features (RIPD-1515):
In order to automatically run unit tests with newly created
amendments, prefer to start with jtx::supported_features() and
then subtract unwanted features.

These changes identified a few bugs that were hiding in
amendments.  One of those bugs, in FlowCross, is not yet fixed.
By uncommenting the test in CrossingLimits_test.cpp you can see
failures relating to that bug.  Since FlowCross is not yet
enabled on the network we can fix the bug at our convenience.
2017-12-01 14:15:05 -05:00
Edward Hennis
aa76632bb3 Add STAmount switchover support to tx queue:
* RIPD-1513
* New fix1513 amendment
2017-12-01 14:15:05 -05:00
Scott Schurr
feb7582aca Unit tests don't enable Tickets or SHAMapV2 by default:
Both Tickets and SHAMapV2 have been around for a while and don't
look like they will be enabled on the network soon.  So they are
removed from the supportedAmendments list.  This prevents Env
from automatically testing with Tickets or SHAMapV2 enabled,
although testing with those features can still be explicitly
specified.

Drive-by cleanups:

o supportedAmendments() returns a const reference rather than
  a fresh vector on each call.

o supportedAmendments() implementation moved from Amendments.cpp
  to Feature.cpp.  Amendments.cpp deleted.

o supportedAmendments() declared in Feature.h.  All other
  declarations deleted.

o preEnabledAmendments() removed, since it was empty and only
  used in one place.  It will be easy to re-add when it is needed.

o jtx::all_features_except() renamed to
  jtx::supported_features_except(), which is more descriptive.

o jtx::all_amendments() renamed to jxt::supported_amendments()

o jtx::with_features() renamed to with_only_features()

o Env_test.cpp adjusted since featureTickets is no longer
  automatically enabled for unit tests.
2017-12-01 14:15:04 -05:00
seelabs
07e3f81b76 Run unit tests in parallel 2017-12-01 14:15:04 -05:00
seelabs
6ff5d3734f Make unit tests repeatable 2017-12-01 14:15:04 -05:00
seelabs
5fe65c5906 Use unique ports for every unit test 2017-12-01 14:15:04 -05:00
Brad Chase
2c13d9eb57 Redesign CSF framework (RIPD-1361):
- Separate `Scheduler` from `BasicNetwork`.
- Add an event/collector framework for monitoring invariants and calculating statistics.
- Allow distinct network and trust connections between Peers.
- Add a simple routing strategy to support broadcasting arbitrary messages.
- Add a common directed graph (`Digraph`) class for representing network and trust topologies.
- Add a `PeerGroup` class for simpler specification of the trust and network topologies.
- Add a `LedgerOracle` class to ensure distinct ledger histories and simplify branch checking.
- Add a `Submitter` to send transactions in at fixed or random intervals to fixed or random peers.

Co-authored-by: Joseph McGee
2017-12-01 14:15:04 -05:00
Mike Ellery
b9fc9f6334 Add jenkins multiconfig pipeline support:
Fixes: RIPD-1514

Create a Jenkinsfile to build on rippled slaves. Modify
build_and_test.sh to support a few additional ENV settings.
2017-12-01 13:49:58 -05:00
Edward Hennis
1853c0d678 Add STAmount switchover support to tx queue:
* RIPD-1513
* New fix1513 amendment
2017-12-01 13:49:58 -05:00
Brad Chase
6dc79c23ed Set version to 0.80.1 2017-11-29 17:58:11 -05:00
JoelKatz
5a9c3c797c Support python3 in SCons 2017-11-29 17:57:58 -05:00
Brad Chase
a4a43a4de9 Improve WorkSSL:
Support Server Name Indication
Ensure windows uses available certificates
2017-11-29 17:39:57 -05:00
wilsonianb
25b002b37f Add make_Manifest test
Reject manifest with invalid public key type

RIPD-1560
2017-11-28 20:33:33 -05:00
wilsonianb
40c39c4afb Verify validator token manifest matches private key
RIPD-1552
2017-11-28 20:33:31 -05:00
Nikolaos D. Bougalis
f0e1024ad6 Explicitly use std::deque for the missing node handler stack:
We need to ensure that pointers and/or references to existing
elements will not be invalidated during the course of element
insertion and removal.

Containers like std::vector do not offer this guarantee, and
cannot be used as the underlying container for the stack.

By choosing to explicitly specify std::deque as the underlying
cotnainer, we avoid:

- the unlikely possibility of the C++ standards committee
  changing the default template parameter to a container;
- the more likely possibility of an accidental change by
  a programmer, without fully considering the consequences.
2017-11-28 20:31:39 -05:00
Edward Hennis
c11e186659 Grow TxQ expected size quickly, shrink slowly (RIPD-1534):
* Stores recent history of "good" ledgers. Uses the maximum as the
  expected ledger size. When a large value drops off, use a 90%
  backoff to go down to to the new maximum.
* If consensus is unhealthy, wipe the history in addition to the current
  clamping.
* Include .md doc files in xcode and VS projects
2017-11-28 20:27:54 -05:00
Brad Chase
0a48916d98 Properly use ledger hash to break ties 2017-11-28 20:26:01 -05:00
Mark Travis
dffb999efb Set quorum to tolerate 2 failures for validator list of 6 members. 2017-11-28 20:23:07 -05:00
Brad Chase
044dd53513 Add validator list RPC commands (RIPD-1541):
In support of dynamic validator list, this changeset:

1. Adds a new `validator_list_expires` field to `server_info` that
indicates when the current validator list will become stale.
2. Adds a new admin only `validator_lists` RPC that returns the
current list of known validators and the most recent published validator
lists.
3. Adds a new admin only `validator_sites` RPC that returns the list of
configured validator publisher sites and when they were most recently
queried.
2017-11-28 20:21:19 -05:00
wilsonianb
02059a27d6 Make quorum unreachable if validator list expires (RIPD-1539) 2017-11-27 09:15:43 -05:00
Mark Travis
8f347a5333 Log error conditions when fetching validator list. 2017-11-27 09:12:14 -05:00
wilsonianb
bce9bca2ce Include validator manifests in published list:
Manifests of validators newly added to a published validator list are
not reliably propagated to network nodes.
This solves the problem by allowing a published validator list to
include the manifest.

RIPD-1559
2017-11-22 16:25:40 -06:00
Nik Bougalis
cafe18c592 Set version to 0.80.0 2017-10-19 14:37:27 -07:00
Nik Bougalis
3e5490ef6d Set version to 0.80.0-rc3 2017-09-22 19:35:54 -07:00
Brad Chase
c76656cf7f Use rounded close time in Consensus (RIPD-1528):
Switches the default behavior of Consensus to use roundCloseTime instead of
effCloseTime. effCloseTime is still used when accepting the consensus ledger to
ensure the consensus close time comes after the parent ledger close time. This
change eliminates an edge case in which peers could reach agreement on the close
time, but end up generating ledgers with different close times.
2017-09-22 19:35:29 -07:00
Nik Bougalis
c7c1b3cc3b Track escrow in recipient's owner directory (RIPD-1523):
Introduce "fix1523" which corrects a minor technical flaw with
the original implementation of the escrow feature.

When creating an escrow, the entry would only be tracked in the
owner directory of the sender; as a result, an escrow recipient
would not be able to detect incoming escrows without monitoring
the ledger in real-time for transactions of interest or without
the sender communicating this information out of band.

With the fix in place, escrows where the recipient differs from
the sender will be listed in the recipient's owner directory as
well.
2017-09-22 17:19:18 -07:00
Nik Bougalis
39f9135104 Securely erase memory & reduce public API 2017-09-22 17:19:16 -07:00
Nik Bougalis
dd52bdd2c4 Merge master (0.70.2) into develop (0.80.0-rc2) 2017-09-22 17:16:17 -07:00
Christian Ramseier
7a0fa312ea Update link to whitepaper and remove dead link 2017-09-22 17:13:19 -07:00
Edward Hennis
cd2d52acdc Set version to 0.70.2 2017-09-21 15:04:23 -04:00
Edward Hennis
62127d725d Recover open ledger transactions to the queue (RIPD-1530):
* If the transaction can't be queued, recover to the open ledger once,
  and drop it on the next attempt.
* New result codes for transactions that can not queue.
* Add minimum queue size.
* Remove the obsolete and incorrect SF_RETRY flag.
* fix #2215
2017-09-21 15:02:23 -04:00
seelabs
fc640504ba Set version to 0.80.0-rc2 2017-09-01 14:17:24 -04:00
seelabs
3c0b35092c Remove unused lambda capture 2017-09-01 14:16:09 -04:00
Edward Hennis
89e8ea436a Fix typo in TxQ log message 2017-09-01 14:15:18 -04:00
seelabs
21dc05fc33 Better error codes for payment channel claim (RIPD-1512) 2017-09-01 14:14:05 -04:00
Nik Bougalis
589570daa3 Set version to 0.80.0-rc1 2017-08-17 12:42:12 -07:00
Brad Chase
a02a469b20 Fix unit test compiler warnings 2017-08-17 12:41:45 -07:00
Mark Travis
be1f734845 Make amendment blocked state thread-safe & simplify a constructor. 2017-08-17 12:41:45 -07:00
Nik Bougalis
98d7fa3fd9 Set version to 0.80.0-b5 2017-08-10 15:57:09 -07:00
Howard Hinnant
74823cb7d1 Enable -Wunused-variable in CMake on macOS 2017-08-10 15:56:50 -07:00
Edward Hennis
e47bfa223f Stop ignoring npm paths 2017-08-10 15:56:50 -07:00
seelabs
5c1ec051f0 Resolve compiler warning 2017-08-10 15:56:50 -07:00
Mike Ellery
65094d9c90 Change common libs for XCode/apple (RIPD-305):
Removes unnecessary common link libs related to openssl, which we link
to explicitly using finder vars now. THis allows the generated XCode
project to link without error. Tested with CMake 3.9.0 and XCode 8.3.3.
2017-08-10 15:56:45 -07:00
Mike Ellery
c00341a97e Add invariant check tests (RIPD-1493):
Add coverage for a few invariant checks. Handle exception in invariant
checking code so that an check that throws an exception will still
properly return tef/tecINVARIANT_FAILED.
2017-08-10 15:56:09 -07:00
Mike Ellery
36423a5f77 Add zlib to libs when finding openssl (RIPD-1496):
if openssl is configured with compression support AND you link
to the static lib, you are going to need to link to zlib.

Fundamentally the CMake finder should take care of this (as
described in https://gitlab.kitware.com/cmake/cmake/issues/16885),
but we work around it here by adding zlib explicitly. Update
beast error test for OpenSSL 1.1.0
2017-08-10 15:56:05 -07:00
Brad Chase
60dd194b72 Use ledger hash to break ties (RIPD-1468):
When two ledgers have the same number of validations, the code
will now use the ledger hash itself to break the tie rather than
the highest node ID supporting each validation.
2017-08-10 15:55:52 -07:00
Brad Chase
9ae717c433 Move tagged_integer to ripple/basics 2017-08-10 15:55:46 -07:00
Mark Travis
d90a0647d6 Change UNL and quorum rules:
* Use fixed size UNL if the total listed validators are below
    threshold.
  * Set quorum to provide Byzantine fault tolerance until a
    threshold of total validators is exceeded, at which time
    quorum is 80%.
  * Ensure that a quorum of 0 cannot be configured.
2017-08-10 15:55:26 -07:00
Niraj Pant
35d81e65c1 Enforce TransferRate Maximum (RIPD-1201):
Sets a maximum TransferRate value of 100%. Squashes any
requested TransferRate over the limit to the max value.

This change requires an amendment ("fix1201") in rippled.

Adds test functionality for enabling an amendment mid-test.
Additionally, adds tests utilizing varying transfer rates both
with and without the amendment enabled.
2017-08-10 15:53:48 -07:00
seelabs
cca574c9a9 Set version to 0.80.0-b4 2017-07-31 18:40:02 -04:00
Mark Travis
c96c423afb Allow a newly-started validator to participate in consensus. 2017-07-31 18:39:59 -04:00
Nik Bougalis
463b154e3d Improve directory insertion & deletion (RIPD-1353, RIPD-1488):
This commit introduces the "SortedDirectories" amendment, which
addresses two distinct issues:

First, it corrects a technical flaw that could, in some edge cases,
prevent an empty intermediate page from being deleted.

Second, it sorts directory entries within a page (other than order
book page entries, which remain strictly FIFO). This makes insert
operations deterministic, instead of pseudo-random and reliant on
temporal ordering.

Lastly, it removes the ability to perform a "soft delete" where
the page number of the item to delete need not be known if the
item is in the first 20 pages, and enforces a maximum limit to
the number of pages that a directory can span.
2017-07-31 18:39:59 -04:00
Nik Bougalis
3666948610 Make code self-documenting by using symbolic constants 2017-07-31 18:39:59 -04:00
seelabs
397410bac6 Resolve memory leaks from make_SSLContext:
* Move into ssl functions that release the unique ptr
* Use string ref in make_SSLContext
* Resolve memory leaks
2017-07-31 18:39:59 -04:00
Howard Hinnant
7aa838c091 Give each Throw statement in codec.h a unique message 2017-07-31 18:39:59 -04:00
seelabs
458ac470aa Return error on rippleCalc exception 2017-07-31 18:39:59 -04:00
seelabs
a79cb95c85 Fix use after free error in test code 2017-07-31 18:39:59 -04:00
seelabs
5ad49454f1 Remove unused lambda captures 2017-07-31 18:39:58 -04:00
Scott Schurr
1a56b9c5f2 Replace DeadlineTimer with asio::steadyTimer (RIPD-1356):
The two active users of DeadlineTimer, NetworkOPs and Application,
now use asio::steady_timers rather than DeadlineTimer.
DeadlineTimer is removed since it is no longer used.

To assure that all in-flight closures on timers are done before
Stoppables call stopped(), the JobCounter is made more generic.
It's now a ClosureCounter.  The ClosureCounter is currently used
to count closures in flight for the JobQueue, NetworkOPs, and the
Application.
2017-07-31 18:39:58 -04:00
Scott Schurr
efe3700f70 Don't use JobQueue during shutdown (RIPD-1356):
If the JobQueue is used during shutdown then those Jobs may access
Stoppables after they have already stopped.  This violates the
preconditions of Stoppables and may lead to undefined behavior.

The solution taken here is to reference count all Jobs in the
JobQueue.  At stop time all Jobs already in the JobQueue are
allowed to run to completion, but no further Jobs are allowed
into the JobQueue.

If a Job is rejected from the JobQueue (because we are stopping),
then JobQueue::addJob() returns false, so the caller can make any
necessary adjustments.
2017-07-31 18:39:58 -04:00
Mike Ellery
fc89d2e014 Fix limit setting and add ServerImp tests (RIPD-1463,1458):
Add more test coverage for ServerHandlerImp.cpp. Ensure limit parameter
is propagated from parsed object to in-memory config.

Release Notes
-------------

This fixes a bug whereby the limit parameter on a port configuration was
ignored.
2017-07-31 18:39:58 -04:00
Scott Schurr
3c37539cee Address Travis compiler out of memory and timeout errors 2017-07-31 18:39:58 -04:00
seelabs
3d977aeacb Updates for gcc compiler 2017-07-31 13:09:29 -04:00
seelabs
4308b124c2 Set version to 0.80.0-b3 2017-07-20 18:13:49 -04:00
seelabs
b23e9c207d Handle build when BOOST_ROOT env var is not set 2017-07-20 18:12:28 -04:00
seelabs
c3a6b1600f Set version to 0.80.0-b2 2017-07-20 14:28:00 -04:00
Niraj Pant
138e1ba9a8 Fix spelling in Main.cpp 2017-07-20 14:27:57 -04:00
Edward Hennis
2858661bce Support CMake in Test.py
* scons will remain the default build type.
* New set of command line switches to control CMake.
2017-07-20 14:27:57 -04:00
Scott Schurr
afc791835e RequireAuth account can create offers to buy their currency 2017-07-20 14:27:56 -04:00
Mike Ellery
d981bff8ea Check amendment block status and update w/ ledgers:
Check and modify amendment blocked status with each new ledger (provided
by @wilsonianb). Honor blocked status in certain RPC commands and when
deciding whether to propose/validate.

Fixes: RIPD-1479
Fixes: RIPD-1447

Release Notes
-------------

This resolves an issue whereby an amendment blocked server would still
serve some RPC requests that are unreliable in blocked state and would
continue to publish validations.
2017-07-20 14:27:56 -04:00
Scott Schurr
b24d47c093 Correct handling of unauthorized offers (RIPD-1481) 2017-07-20 14:14:03 -04:00
Brad Chase
df086301b6 Fix consensus quorum comparison 2017-07-20 14:14:03 -04:00
Brad Chase
01b4d5cdd4 Migrate thread safety to RCLConsensus (RIPD-1389):
Moves thread safety from generic Consensus to RCLConsensus and switch generic
Consensus to adaptor design.
2017-07-20 14:14:03 -04:00
wilsonianb
8c155dd875 Make minimum quorum Byzantine fault tolerant (RIPD-1461) 2017-07-20 11:45:39 -04:00
seelabs
3f9a38697d Cache gdb in CI 2017-07-20 11:45:39 -04:00
seelabs
87742a5e6c Set version to 0.80.0-b1 2017-07-11 22:02:39 -04:00
seelabs
5b9e8a77ca Install gdb 8 for CI:
A bug in gdb is crashing when demangling some symbols. Gdb needs to be updated
to at least gdb 8.

Only run the debug build under gdb (other builds can time out).
2017-07-11 22:02:35 -04:00
Brad Chase
2821624ede Add consensus tests to CMake (RIPD-1482) 2017-07-11 12:55:06 -04:00
Mike Ellery
ac1ab720c4 Add tests for subscribe/unsubscribe error cases:
Fixes: RIPD-1417

Fix incorrect error case messages. Fix crash in NetworkOps instance when
exiting with remaining RPC subscriptions. Add code to remove URL
subscription when requested.
2017-07-11 12:55:00 -04:00
Brad Chase
e0168b98d7 Add optional consensus hash field 2017-07-11 12:54:47 -04:00
Mike Ellery
18801b81de Add jemalloc support to cmake build (RIPD-1472):
Enable jemalloc under option. Create additional option that sets flags
appropriate for use with linux perf. Tested with jemalloc on linux and
macos.
2017-07-11 12:54:38 -04:00
Mike Ellery
1daefeb594 Add unit test for owner_info (RIPD-1416)
This RPC method is deprecated, but i've added a basic test to cover it
until we actually remove it.
2017-07-11 12:54:28 -04:00
Brad Chase
068048718e Use Boost ICL for RangeSet (RIPD-1473) 2017-07-11 12:54:11 -04:00
Mike Ellery
56946e8128 Change features default behavior in Env (RIPD-1460):
Enable all supported amendments in Env by default. Rename `features()`
to `with_features()` and add `all_features_except()` to support feature
subsets in Env. Refactor internal feature handling based on a bitset.
2017-07-11 12:53:59 -04:00
Brad Chase
3dfb4a13f1 Expose consensus parameters for simulation (RIPD-1355) 2017-07-11 12:53:53 -04:00
Brad Chase
7ae3c91015 Refactor Validations (RIPD-1412,RIPD-1356):
Introduces a generic Validations class for storing and querying current and
recent validations.  Aditionally migrates the validation related timing
constants from LedgerTiming to the new Validations code.

Introduces RCLValidations as the version of Validations adapted for use in the
RCL.  This adds support for flushing/writing validations to the sqlite log and
also manages concurrent access to the Validations data.

RCLValidations::flush() no longer uses the JobQueue for its database
write at shutdown.  It performs the write directly without
changing threads.
2017-07-11 12:53:34 -04:00
Miguel Portilla
95f107d487 Update to use latest beast 2017-07-11 12:18:53 -04:00
Vinnie Falco
61316c7f95 Changes for Beast version 59 2017-07-11 12:18:17 -04:00
Miguel Portilla
49bdf2e72d Update beast subtree to version 79
Merge commit 'f0f96bd1da42ccc0490d386a4e2f0ee8f2dadb43' into develop
2017-07-11 12:17:02 -04:00
Miguel Portilla
f0f96bd1da Squashed 'src/beast/' changes from 6d5547a..3bcd986
3bcd986 Set version to 79
4f42f8c Remove spurious fallthrough guidance
fa1ac16 Set version to 78
0cb9b63 Fix warning in root ca declaration
f78c73a Tidy up file_posix unused variable
72ce9ef Tidy up dstream for existing Boost versions
efe8e58 Add Boost.Locale utf8 benchmark comparison
f7c745e Remove string_view_body
7a79efa Tidy up FieldsReader doc
e51aefd Header file tidying
69898f4 Fix warning in zlib
53723c0 Add message keep_alive, chunked, content_length members
d7af73b Fix spurious uninitialized warning
ca42cc0 Tidy up invalid characters in test vector
48d3e60 Use make_unique_noinit
a1ff804 span, string, vector bodies are public
feab6a0 Documentation work
116c0b0 Add span_body
7fb901d Tidy up includes and javadocs
f0f58be Add span
7a8982b Add vector_body
524f73a Tidy up basic_string_body
a8ad67b Set version to 77
d555859 file_posix works without large file support
1bc30cb Set version to 76
9a1e7a8 Disable SSE4.2 optimizations
09af312 Fix parse illegal characters in obs-fold
7dd684c Add file_body_win32:
1bbc71c serializing file_body is not const
9a4b55e BodyReader, BodyWriter use two-phase init (API Change):
dc400ce Serializer members are not const
1a33c37 Rename to serializer::keep_alive (API Change):
55935c5 Add serializer::chunked
63ace37 Add serializer::get
2c4047b BodyReader may construct from a non-const message
0a0a225 Use Boost.Config
6f83d07 Always go through write_some:
0e23066 Set version to 75
28f3ece Doc tidying
3495331 Using SSE4.2 intrinsics in basic_parser if available
bc1f0ac file_body tests
4e03d7e Add serializer::limit
85e3ee8 Shrink serializer buffers using buffers_ref
78bcdb1 Tidy up BEAST_NO_BIG_VARIANTS
3ea6cf2 Construct buffer_prefix_view in-place
69f9f7a Use file_body for valid requests, string_body otherwise.
6f88f01 Set version to 74
93fed8e remove redundant flush() from example:
e0f56da Fix Beast include directories for cmake targets
5ff9e0a Add file_posix
1bb5705 Remove common/file_body.hpp
5c89d87 Add file_body
67a55c8 Add file_win32
647d3b0 Add file_stdio and File concept
89c416c Set version to 73
0efc32f Fixes for gcc-4.8
c8910ab Initialize local variable in basic_parser
8a28193 Adjust benchmarks
81e51d8 Verify certificates in SSL clients
a43f6d4 Jamroot tweak
8c85ee8 Put more... links on overload reference pages
ff1104e Documentation tidy
826ff0e serializer::next replaces serializer::get (API Change):
8d67775 Refactor header and message constructors:
6c79f19 Add basic_parser tests
25127d9 basic_parser optimizations:
9d082fd Set version to 72
c88e2b9 Various improvements to http_server_fast.cpp:
20b0fdb Documentation tidying
afd1fa7 Add websocket-server-async example
954b597 Add http-server-threaded example
df8f253 Refactor file_body for best practices
11c1037 Newly constructed responses have a 200 OK result
a648817 Refine Body::size specification
40aad37 Tidy up set payload in http-server-fast
52cefbc Set version to 71
8c51c77 Tidy up Jamroot /permissive-
5efecea Update README.md
5a47acd Tidy up http_sync_port error check
a2af2b5 Concept check in basic_dynamic_body
8b80a6f Fix buffer overflow handling for string_body and mutable_body
ec3b4fd Return std::size_t from Body::writer::put (API Change)
effbb37 Check trailers in test
f5368cf Call prepare_payload in HTTP example
a3e5e01 Fix spurious on_chunk invocation
96d94eb Add options for building examples and tests. Move zlib test sources to test/zlib
e0efdc7 Allow close, ping, and write to happen concurrently
9c1a419 Refactor websocket composed ops
d5659a4 Fine tune websocket asserts
b8e8943 std::pair "last" -> "first" in http_message.qbk
c691bf4 Fix can/cannot thinko in FAQ.
6dd006b Documentation revision
6d2e315 Fix extra ; warning
78a065b Set version to 70
00c7e9d Fix HEAD response in file_service
67d70d2 Fix BEAST_FALLTHROUGH in config
4f33655 Add parser::on_header to set a callback
9c16b21 Add basic_parser header and body limits:
b64d1f7 Rename to message::base (API Change):
436c66a Serialize in one step when possible
3e1061b Set version to 69
f709273 Add /permissive- to msvc toolchain
0dae464 Use BEAST_FALLTHROUGH to silence warnings
a70d386 basic_parser optimizations
4269f35 Set version to 68
544327f Link statically on cmake MSVC
e213ffe Add const_body, mutable_body to examples
0568763 Optimize field lookups
8fc3001 Use string_ref in older Boost versions
8af77da bad_target replaces bad_path (API Change):
325dd62 Adjust buffer size in fast server
be59785 Doc erratum
d9b44f3 Small speed up in fields comparisons
3e6ce38 Use Boost master on Appveyor
09f3d64 Split common tests to a new project
adfd22a Remove BodyReader::is_deferred (API Change):
582d28d Change BodyReader, BodyWriter requirements (API Change):
8982e14 Set version to 67
daa58a2 Group common example headers
afd8f1a Rename to http-server-fast
07cb9f7 control_callback replaces ping_callback (API Change):
91e83ed Use boost::string_view
067db87 Merge stream_base to stream and tidy
d61241a Add http-server-small example
eb08e92 Fix doc example link
7fb75d0 Set version to 66
df86723 Respect debug flag for marked output
c08565a Squelch spurious warning on gcc
188ef7c Documentation work
1c62d3a Add http-server example
3f54582 basic_fields optimizations
a8b05b8 Add header aliases
b94eac3 Tidy up message piecewise ctors
9c48b52 Handle bad_alloc in parser
1b57c54 Fix costly potential value-init in parser
1edc41e Make consuming_buffers smaller
72ac918 Add serializer request/response aliases
18f7606 string_param optimizations
c675252 Set version to 65
c398cdd Enable msvc warnings in Jamfile
380cceb Fix unused variable warnings
4172e7e Enable unused variable warning on msvc cmake
f04d227 Fix integer warnings
ca975b3 Fix narrowing in deflate_stream
2fab796 Fix narrowing in inflate_stream
cff87f6 Fix narrowing in ostream
1956886 Fix narrowing in static_ostream
69cdc4b Fix integer types in deflate_stream::bi_reverse
3e3dd21 Enable narrowing warning on msvc cmake
e11a294 Set version to 64
a00e070 Remove make_serializer (API Change):
8449697 Add link_directories to cmake
7b2b066 Doc tidying
158d3e8 async_write requires a non-const message:
d13328b Better User-Agent in examples
ebcb2c0 Exemplars are compiled code
b9054d3 Simplify websocket write_op
c2571fe Simplify ssl teardown composed op
d8ad3d1 Simplify buffered_read_stream composed op
f68dc34 Set version to 63
a99f7ef Control running with valgrind explicitly
4eb7af4 Tidy up Jamfiles
49b42a5 Tidy up CMakeLists.txt
dadb54f Only run the tests under ubasan
d1c7696 Move benchmarks to a separate project
a4aada8 Only build and run tests in variant=coverage
f835b9a Don't use cached Boost
a0edd82 Put num_jobs back up on Travis
a8d5823 Use std::to_string instead of lexical_cast
45d8b81 Set version to 62
09e07ce Put slow tests back for coverage builds
295b1d7 Doc tidy
f58425c Squelch harmless not_connected errors
9b537f7 Add http::is_fields trait
d43701b message::prepare_payload replaces message::prepare (API Change):
42ba289 Refine FieldsReader concept (API Change)
bde90a1 Narrow the use of Fields parameters:
5f47526 parser requires basic_fields (API Change):
60f58e4 Avoid explicit operator bool for error
352f8de Clear the error faster
34befd8 Tidy up namespaces in examples
9e0b4b5 Doc fixes and tidy
c003a2a Tidy up test build scripts and projects
b929130 Add server-framework tests
03d4301 Increase detail::static_ostream coverage
80d7cbc Remove libssl-dev from a Travis matrix item
4c15db4 Set version to 61
1dfbd0b Don't run slow tests on certain targets
6bb1109 Use one job less on CI
6f58342 Tidy up resolver calls
530b044 Add multi_port to server-framework
bfef5d1 Tidy up http-crawl example
e2f2f33 Reorganize SSL examples
adc301b Fix shadowing warnings
c60185e Add server-framework SSL HTTP and WebSocket ports
7912fb8 Refactor WebSocket, HTTP examples:
cd4b9e0 Flush the output stream in the example
d046b20 Tidy up names in error categories
9d4a422 status-codes is unsigned (API Change)
e3599b0 header::version is unsigned (API Change)
a26b043 Add message::header_part()
fc8d2e9 Tidy up some integer conversion warnings
c91732e Reorganize SSL examples
9907b31 Documentation work
4b2e78e Use generic_cateogry for errno
38c46cd Remove Spirit dependency
c111d6f Set version to 60
d78dc12 Documentation work
141a524 New server-framework, full featured server example:
3f7ffd9 Fix response message type in async websocket accept
13f3750 String comparisons are public interfaces
4e4bcf8 Set version to 59
5015cdb Remove obsolete doc/README.md
71c3f0a Fix base64 alphabet
aa2b843 Change Body::size signature (API Change):
80a599a Documentation work
9c19449 Integrated Beast interface.
3f8097b Set version to 58
4581777 Better message formal parameter names
5879cd8 Fix parsing chunk size with leading zeroes
56bd228 Remove redundant code
534ca63 Use static string in basic_fields::reader
a7b3810 basic_parser::put doc
1e4413f basic_fields::set optimization:
9b244c1 Fix basic_fields insert ordering
4f854d0 Avoid std::string in websocket
dc8f146 Renamed to basic_fields::set (API Change):
660c465 Specification for http read
981285b Documentation tidy
983d676 Reorganize examples:
a2a5c57 Qualify size_t in message template
d86769c Fix unaligned reads in utf8-checker
8ba182c Set version to 57
42e2791 Update doc/ for docca
1ee0afd Merge commit '101d7dbfb9725674cb9ce5a4196f19aa1d4bb801' as 'doc/docca'
101d7db Squashed 'doc/docca/' content from commit c50b3ba5
900c04e Documentation work
8eee932 Fix warning in basic_parser.cpp
437a616 Fix message.hpp javadocs
18c68ce Set version to 56
b058e90 Convert buffer in range loops
cbc9212 Add Beast INTERFACE CMake target
2914b59 More basic_parser tests
ed5c317 Reset error codes
ba14251 Test error code handling
e45e50b Tidy up README.md build instructions
16efb9b Try harder to find Boost (cmake)
e281d91 HTTP/1 is the default version
916fe4a Call on_chunk when the extension is empty
9855598 Add string_view_body
19d4520 Tidy up
6e59f9b Add provisional IANA header field names
84722f2 Revert "Add a Beast CMake interface target:"
fde6929 Set version to 55
01f6cc4 Documentation work
a7a388c read_size replaces read_size_helper:
ed8f0bb Tidy up type_traits
c2f6268 Avoid a parser allocation using non-flat buffer:
906db45 Add a Beast CMake interface target:
47f2541 Don't allocate memory to handle obs-fold
6a8912a Set version to 54
296ef3b Documentation work
e3e9b61 Fix incorrect use of [[fallthrough]]
3c44398 Retain ownership when reading using a message
a71bb2b basic_fields refactor (API Change):
d8d3562 Add string_param
83b2558 basic_fields members and coverage
c4f5fa5 consuming_buffers members and coverage
e10507c multi_buffer members and coverage
0e6bd3f flat_buffer coverage
7351d6e static_buffer coverage
18a8ef5 Set version to 53
452df59 Remove extraneous doc file
3ef0359 Fix read_size_helper usage:
b64e6d3 Fix basic_parser::maybe_flatten (#462)
76402f7 Set version to 52
b0ceb2a Add drain_buffer class
4c6735a flat_buffer is an AllocatorAwareContainer
9d5d4d5 Documentation work
d4ec693 finish(error_code&) is a BodyReader requirement (API Change)
7b24cad opcode is private (API Change):
068c2ac Documentation work
a1ff89b Disable std::future snippet for libstdc++ bug
b5ef664 read_frame returns `bool` fin (API Change):
7911847 Remove opcode from read, async_read (API Change):
c72d70f ping_callback is a member of stream (API Change):
720a309 write_buffer_size is a member of stream (API Change):
7ff0178 read_message_max is a member of stream (API Change):
cd40964 read_buffer_size is a member of stream (API Change):
a58e5e1 binary, text are members of stream (API Change):
ad35846 auto_fragment is a member of stream (API Change):
ccee139 Documentation work
13c64e3 Set version to 51
cafc8e2 Fix infinite loop in basic_parser
dc4b69d Add construct, destroy to handler_alloc
58c2739 multi_buffer implementation change (API Change):
dd7f5c0 DynamicBuffer benchmarks
1c4811b Use BOOST_STRINGIZE
31051ac Use BOOST_FALLTHROUGH
8f2430b Documentation work
eb35b92 Fix file_body::get() not setting the more flag correctly
566244a Tidy up file_body
53cbeea Tune up static_buffer (API Change):
20c59b7 Fix operator<< for header
a2c1117 Set version to 50
6045b74 http read_some, async_read_some don't return bytes (API Change):
4df6885 Fix chunk header parsing
36bf32b Fix test::pipe read_size
bf69ce1 Fix chunk delimiter parsing
0c6b6b1 Add missing handler_alloc nested types
a06b8f9 Tidy up message javadoc
3bd8260 Remove obsolete serializer allocator (API Change)
001c979 Remove message free functions (API Change)
745876b Remove message connection settings (API Change)
bcf2c33 Body documentation work
1e303b0 Fields concept work
9d0464a Tidy up basic_fields, header, and concepts
3ba81b5 Use field in basic_parser
b5f6cc1 Use field in basic_fields and call sites
cfd6d14 Documentation reference tidy
2adc80a Protect basic_fields special members (API Change)
d55b079 Fix basic_fields allocator awareness
d8febda Documentation work
485a6e5 Refactor prepare (API Change)
9fd3071 Derive header from Fields (API Change)
8ad26b8 Use allocator more in basic_fields
0071039 Add verb to on_request for parsers (API Change)
74f6cbb Add field enumeration
be0d74f Documentation fixes
054ac40 Remove header_parser (API Change)
a007eba parser is constructible from other body types
d89809f Documentation work (buffer_body)
ac5bc4f Set version to 49
af47128 Documentation work
a1848f0 Add HEAD request example
ddfbfbf Use <iosfwd> instead of <ostream>
e67c0ab Refactor header status, reason, and target (API Change):
60f044a Tidy up empty_body writer error
7d267f4 Canonicalize string_view parameter types
ac175cb Refactor method and verb (API Change):
e18efed Documentation work
d77e423 Set version to 48
d3a5a05 Documentation work
acf18fb Tidy up traits
6cb188e Remove detail::sync_ostream
d6092bc Documentation work
4707b21 Rename to parser (API Change):
3cb385d Consolidate parsers to parser.hpp
290bdf1 Documentation work
7cb442c Make buffer_prefix_view public
ef0b121 Rename to buffer_cat_view (API Change)
b9df187 Tidy up chunk decorator (API Change):
458fa7e Set version to 47
fc83a03 Documentation work
1ee3013 Fix leak in basic_flat_buffer
55fbf76 Fix undefined behavior in pausation
fe75a7c Refactor HTTP serialization and parsing (API Change):
50cba32 buffer_size overload for basic_multi_buffer::const_buffers_type
d977bf2 Disable operator<< for buffer_body
5db707a Refactor treatment of status code and obsolete reason (API Change):
9a585a8 Refactor treatment of request-method (API Change):
3ae76d0 Set version to 46
6004712 Documentation work
34ea0b3 Refactor serialization algorithms:
407b046 Rename to make_serializer
c29451a Refactor type_traits (API Change):
8578419 Refactor HTTP serialization (API Change):
f8612aa Remove HTTP header aliases (API Change):
b0054e3 Add test::pipe
dfba72b Set version to 45
6ba3697 Disable reverse_iterator buffer_view test
266ebac buffer_view skips empty buffer sequences
96b9892 Documentation work
c23f1e2 Fix header::reason
9796106 Better test::enable_yield_to
9a8bcb7 Fix message doc image
7a5e87e Workaround for boost::asio::basic_streambuf type check
663c275 Set version to 44
f205976 Make buffers_adapter meet requirements
8e39c60 Tidy up is_dynamic_buffer traits test
0088f7c Add buffers_adapter regression test
8a23de1 Fix README websocket example
949504a Fix async return values in docs
cd9f41b Use BOOST_STATIC_ASSERT
1b616fa Tidy up and make get_lowest_layer public
612e616 Require Boost 1.58 or later
1b1daa7 Tidy up read_size_helper and dynamic buffers
bf0145d Use BOOST_THROW_EXCEPTION
e762818 Add GitHub issue template
dab679c Set version to 43
386b817 Reformat README.md QR code
50e5123 Additional constructors for consuming_buffers
f7289b9 Add write limit to test::string_ostream
3aa87e0 Tidy up buffer_prefix overloads and test
bee583c Fix strict aliasing warnings in buffers_view
6b54d3a Require Boost 1.64.0
76f1084 Set version to 42
0bdb148 Make buffers_view a public interface
338fc81 Add formal review notes
784f965 Fix javadoc typo
823aee2 Set version to v41
88adbdd Remove handler helpers, tidy up hook invocations (API Change)
4974af2 Rename prepare_buffer(s) to buffer_prefix (API Change)
ebd459b Tidy up websocket::close_code enum and constructors
c3fd6f9 Tidy up formal parameter names
210cd70 Remove coveralls integration
d811962 Concept revision and documentation (API Change):
bdae92a Replace asynchronous helper macros with template aliases (API Change)
df95a09 Move prepare_buffers to prepare_buffer.hpp (API Change)
787de21 Remove placeholders (API Change)
c59b544 Trim Appveyor matrix rows
b7184f3 Return http::error::end_of_stream on HTTP read eof (API Change)
f2d8255 Set version to 40
40b9194 Tidy up .travis.yml:
9b240c7 Fix basic_streambuf movable trait
76a2617 Consolidate get_lowest_layer in type_traits.hpp
6d00321 Add to_static_string:
f888136 Set version to 39
47c82b5 Better travis deps
4ed7865 Squelch openssl spurious leak and memory errors
b6bc26f Fixed braced-init error with older gcc
59b2f8f ostream workaround for gcc 4.8.4
8363d86 Increase ostream test coverage
5631936 Tidy up HTTP reason_string (API Change):
2bf5150 Harmonize concepts and identifiers with net-ts (API Change):
728e9d8 Tidy up basic_parser javadocs
1c9067b Use beast::string_view alias
771c5ca Doc fixes and tidying
e2b5c31 Rename to buffered_read_stream (API Change):
a753f1c Rename to static_buffer, static_buffer_n (API Change):
24b6686 Rename to flat_buffer, basic_flat_buffer (API Change):
69259ef Rename to multi_buffer, basic_multi_buffer (API Change):
bef9ae1 New buffers() replaces to_string() (API Change):
a7ef4f5 New ostream() returns dynamic buffer output stream (API Change):
87fd60c Fix eof error on ssl::stream shutdown
606fc9d Add websocket async echo ssl server test:
ff5e659 Refactor http::header contents (API Change):
dd02097 Set version to 1.0.0-b38
5596e97 Prevent basic_fields operator[] assignment
c2b32dc Remove websocket::keep_alive option (API Change):
32dbfb2 Refactor WebSocket error codes (API Change):
dd6b500 WebSocket doc work
0b4d87c More flat_streambuf tests
aacefb4 Add test_allocator to extras/test
931a5fb Simplify get_lowest_layer test
ba4228a Use static_string for WebSocket handshakes:
6df3ff3 Refactor base64:
19b124d Refactor static_string:
30e8d16 Set version to 1.0.0-b37
b141020 Fix narrowing warning in table constants
d554b81 Add -funsigned-char to asan build target
bcc6ad8 Add ub sanitizer blacklist
e1f08e9 Fix flat_streambuf:
7d08f59 Fix typo in documentation example
21ef97d Rename to http::dynamic_body, consolidate header:
45a2d73 Rename project to http-bench
c86fee9 Move everything in basic_fields.hpp to fields.hpp (API Change)
a14a5d6 Rename to detail::is_invocable
540d037 Rename to websocket::detail::pausation
84e1739 Document websocket::stream thread safety
dc274af Add is_upgrade() free function:
2c17d04 Refactor websocket decorators (API Change):
235fe68 Provide websocket::stream accept() overloads (API Change):
a715825 CMake hide command lines in .vcxproj Output windows
32024d8 Set version to 1.0.0-b36
f48b95f Update README.md
d8db5f1 Set version to 1.0.0-b35
dd2a514 Tidy up doc declarations
2c50aba Fix README.md CMake instructions
4ffdce2 Update .gitignore for VS2017
403011f Remove http::empty_body (API Change)
f47b661 New HTTP interfaces (API Change):
f6835b8 Rename to BEAST_DOXYGEN
7e37723 Add flat_streambuf:
5b68faa Doc XSL support for list and table markdown
3de46de Make websocket::close_code a proper enum:
0128743 Tidy up MSVC CMake configuration
ccd188e Add appveyor build script

git-subtree-dir: src/beast
git-subtree-split: 3bcd9865f80f12ba5faad35c564918f85b02e271
2017-07-11 12:17:02 -04:00
Nik Bougalis
3bfd9de677 Set version to 0.70.1 2017-06-30 14:29:55 -07:00
JoelKatz
f9b5ab4728 Support OpenSSL 1.1.0:
Work around differences between OpenSSL 1.0 and 1.1 to
permit compiling on distributions that use newer versions.
2017-06-30 14:29:45 -07:00
seelabs
7abd70356d Ensure consistent transaction processing in debug mode:
When NDEBUG is undefined (which is typical in debug configurations)
existing code would perform additional checks which could result in
a `tel` error.
2017-06-30 13:35:36 -07:00
seelabs
d8313288ad Log invariant check messages at "fatal" level 2017-06-29 08:40:50 -07:00
Brad Chase
a89be5b269 Clean up consensus logic:
* Simplify bow-out handling
* Improve dispute updating
* Filter partial validations
2017-06-29 08:40:50 -07:00
Nik Bougalis
7b0d482810 Set version to 0.70.0 2017-06-15 08:05:33 -07:00
Warren Paul Anderson
e81f3eb1d2 Add integration support notice in README:
If you work at a digital asset exchange or wallet provider
and are trying to integrate with Ripple, please contact us
at support@ripple.com. We can help guide your integration.
2017-06-15 08:03:58 -07:00
Nik Bougalis
cd0a2d6ef3 Set version to 0.70.0-rc1 2017-06-14 12:13:44 -07:00
Nik Bougalis
d04d3d3c4a Implement additional invariant checks:
Invariant checks run after a transaction has been processed and
ensure that the end result of that transaction did not violate
any protocol rules.

New checks include:

* XRP balance checks for negative balances
* Offer balance checks for negative balances
* Zero balance checks when handling escrow
2017-06-14 12:11:26 -07:00
wilsonianb
da7da5527c Redact validator token from cleaned config (RIPD-1422) 2017-06-14 11:54:44 -07:00
Nik Bougalis
6f4bc30684 Set version to 0.70.0-b8 2017-06-08 21:38:22 -07:00
Nik Bougalis
fa72795d84 Display warning when generating brain wallets:
A brain wallet is a standard wallet that is generated not from a
random seed but by hashing a user-supplied passphrase. Typically,
human-selected passphrases can contain insufficient entropy.

When generating a wallet from a passphrase, we include a warning
to this effect. The warning would be incorrectly displayed even
if the wallet was being generated from a seed.
2017-06-08 21:38:21 -07:00
Mark Travis
68b8ffdb63 Improve automatic tuning of thread pool:
The job queue can automatically tune the number of threads that
it creates based on the number of processors or processor cores
that are available.

The existing tuning was very conservative, limiting the maximum
number of threads to only 6.

Adjust the new algorithm to allow a larger number of threads and
allow server administrators to override the value in the config
file.
2017-06-08 21:37:59 -07:00
mDuo13
cb91d56d07 Document optional SField type magic 2017-06-08 21:37:22 -07:00
mDuo13
6e889e6a18 Update source code directory map 2017-06-08 21:35:03 -07:00
Frank Cash
be9c3b218b Fix README formatting 2017-06-08 21:34:57 -07:00
seelabs
a92f5b8e5a Set version to 0.70.0-b7 2017-06-01 14:07:05 -04:00
seelabs
3eeb79ee12 Use sandbox for takerCross:
Using a PaymentSandbox for taker cross can cause transaction breaking changes. A
PaymentSandbox uses a deferred credits table, which can cause tiny differences
in computations.
2017-06-01 14:05:32 -04:00
Nik Bougalis
24e1b9911a Set version to 0.70.0-b6 2017-05-17 04:06:31 -07:00
Edward Hennis
3a973ab719 Improve TxQ locking 2017-05-17 04:06:22 -07:00
Edward Hennis
6f10fe8502 Tidy up MSVC CMake configuration 2017-05-17 04:06:22 -07:00
Edward Hennis
d471e533b7 Group CMake sources from arbitrary folder:
* Instead of default PROJECT_SOURCE_DIR
* Useful when called for submodules
2017-05-17 04:06:22 -07:00
Scott Schurr
1a238048d5 Reduce JobQueue interface 2017-05-17 04:06:21 -07:00
Brad Chase
aa2ff00485 Prevent duplicate txs in book subscription (RIPD-1465):
If an offer transaction touched multiple ledger entries associated with the same
book, that offer transaction would be published multiple times to anyone subscribed
to that book stream.

Fixes #2095.
2017-05-17 04:06:21 -07:00
Brad Chase
f2787dc35c Improve pseudo-transaction handling (RIPD-1454, RIPD-1455):
Adds additional checks to prevent relaying and retrying pseudo-transactions.
2017-05-17 04:06:21 -07:00
Mike Ellery
8002a13dd2 Add test for account_currencies (RIPD-1415) 2017-05-16 19:46:58 -07:00
seelabs
7dc2fe9ce7 Handle strand creation for erroneous self-payment 2017-05-16 19:46:58 -07:00
seelabs
5f37765292 Make Paystrand tests automatic:
Only PayStrandAllPairs is still manual
2017-05-16 19:46:58 -07:00
seelabs
a56d31910f Disallow account one in payments 2017-05-16 19:46:58 -07:00
seelabs
24505a358a Remove unneeded test and std::bind:
These changes are needed to support gcc 7
2017-05-16 15:16:30 -07:00
Nik Bougalis
c570695aa1 Merge master (0.60.3) into develop (0.70.0-b5) 2017-05-16 15:12:55 -07:00
Nik Bougalis
208028a142 Set version to 0.60.3 2017-05-10 10:21:04 -07:00
Brad Chase
dceef25e2c Update Travis dependency 2017-05-09 12:43:32 -07:00
JoelKatz
256e58204a Give statically-configured bootcache entries priority:
Make sure statically-configured bootcache entries have at least
a reasonable minimum priority. This provides additional protection
against Sybil attacks.

Show the bootcache in the ouput of the print command.
2017-05-09 12:42:35 -07:00
JoelKatz
c1d64e1b1a Overlay tuning and logging improvements:
Adjust overlay tuning to reflect measured behavior of the
network under increased load.

Improve logging of peer sendq size and disconnect reasons.
2017-05-09 12:42:21 -07:00
Scott Schurr
1dbc5a57e6 Set version to 0.70.0-b5 2017-04-24 16:13:46 -07:00
Edward Hennis
9cc542fe67 Fix include ordering 2017-04-24 15:37:23 -07:00
Edward Hennis
f7a7f13287 Run Travis CI unit tests using arguments from config 2017-04-24 15:36:06 -07:00
Edward Hennis
96ece1b9f0 Fix levelization
* Move `chooseLedgerEntryType` from protocol to RPC
2017-04-24 14:47:29 -07:00
Edward Hennis
46004158a2 Allow Json parser understand TER strings where appropriate 2017-04-24 13:44:45 -07:00
Edward Hennis
7e9ac16c22 Fix Json Int/UInt comparison limit check 2017-04-24 13:20:40 -07:00
Miguel Portilla
2e5ab4e0e3 Make Websocket send queue configurable 2017-04-24 13:19:10 -07:00
Brad Chase
00c60d408a Improve Consensus interface and documentation (RIPD-1340):
- Add Consensus::Result, which represents the result of the
establish state and includes the consensus transaction set, final
proposed position and disputes.
- Add Consensus::Mode to track how we are participating in
consensus and ensures the onAccept callback can distinguish when
we entered the round with consensus versus when we recovered from
a wrong ledger during a round.
- Rename Consensus::Phase to Consensus::State and eliminate the
processing phase.  Instead, accept is a terminal phase which
notifies RCLConsensus via onAccept callbacks.  Even if clients
dispatch accepting to another thread, all future calls except to
startRound will not change the state of consensus.
- Move validate_ status from Consensus to RCLConsensus, since
generic implementation does not directly reference whether a node
is validating or not.
- Eliminate gotTxSetInternal and handle externally received
TxSets distinct from locally generated positions.
- Change ConsensusProposal::changePosition to always update the
internal close time and position even if we have bowed out. This
enforces the invariant that our proposal's position always
matches our transaction set.
2017-04-24 13:13:23 -07:00
Scott Schurr
d5dc715d9c Unit test offer crossing with lsfRequireAuth (RIPD-1346) 2017-04-24 10:10:31 -07:00
Scott Schurr
369909df84 Use payment flow code for offer crossing (RIPD-1094):
Replace Taker.cpp with calls to the payment flow() code.

This change required a number of tweaks in the payment flow code.
These tweaks are conditionalized on whether or not offer crossing
is taking place.  The flag is explicitly passed as a parameter to
the flow code.

For testing, a class was added that identifies differences in the
contents of two PaymentSandboxes.  That code may be reusable in
the future.

None of the Taker offer crossing code is removed.  Both versions
of the code are co-resident to support an amendment cut-over.

The code that identifies differences between Taker and Flow offer
crossing is enabled by a feature.  That makes it easy to enable
or disable difference logging by changing the config file.  This
approach models what was done with the payment flow code.  The
differencing code should never be enabled on a production server.

Extensive offer crossing unit tests are added to examine and
verify the behavior of corner cases.  The tests are currently
configured to run against both Taker and Flow offer crossing.
This gives us confidence that most cases run identically and
some of the (few) differences in behavior are documented.
2017-04-24 09:24:46 -07:00
Vinnie Falco
2cd55ebf98 Fix json_body for Beast API changes 2017-04-20 13:45:28 -07:00
Vinnie Falco
4e43e22a3a Update to Beast 1.0.0-b34:
Merge commit 'd8dea963fa5dc26b4be699ce6d4bf699a429ca92' into develop
2017-04-20 13:42:52 -07:00
Vinnie Falco
d8dea963fa Squashed 'src/beast/' changes from 1b9a714..6d5547a
6d5547a Set version to 1.0.0-b34
6fab138 Fix and tidy up CMake build scripts:
ccefa54 Set version to 1.0.0-b33
32afe41 Set internal state correctly when writing frames:
fe3e20b Add write_frames unit test
578dcd0 Add decorator unit test
aaa3733 Use fwrite return value in file_body
df66165 Require Visual Studio 2015 Update 3 or later
b8e5a21 Set version to 1.0.0-b32
ffb1758 Update CMake scripts for finding packages:
b893749 Remove http Writer suspend and resume feature (API Change):
27864fb Add io_service completion invariants tests
eba05a7 Set version to 1.0.0-b31
484bcef Fix badge markdown in README.md
5663bea Add missing dynabuf_readstream member
0d7a551 Tidy up build settings
0fd4030 Move the handler, don't copy it

git-subtree-dir: src/beast
git-subtree-split: 6d5547a32c50ec95832c4779311502555ab0ee1f
2017-04-20 13:40:52 -07:00
Nik Bougalis
84816d1c21 Set version to 0.70.0-b4 2017-04-19 12:25:04 -07:00
MarkusTeufelberger
8430f9deff Fix video link in README 2017-04-19 12:24:56 -07:00
Edward Hennis
fcceb0aac1 Update developer documentation 2017-04-19 12:24:56 -07:00
seelabs
2680b78b5b Rename featureToStrandV2 to fix1373 2017-04-19 12:24:56 -07:00
seelabs
068889e5b1 Cleanup fix1449 2017-04-19 12:24:56 -07:00
seelabs
3bd9772c04 Rename timebase switches from 'amendment' to 'fix' 2017-04-19 12:24:56 -07:00
Mike Ellery
af66c62814 Add Unit Test for peers RPC Request (RIPD-1419) 2017-04-19 12:24:56 -07:00
Mike Ellery
5bc8f2e3e8 Add test for noripple_check (RIPD-1400):
Add tests. Fix an error type returned in the handler.
2017-04-19 12:24:56 -07:00
Mike Ellery
22c97ba801 Use beast::temp_dir in tests (RIPD-1414):
Use non-throwing fs function in temp_dir destructor. Eliminate only use
of BOOST_SCOPE_EXIT.
2017-04-19 12:24:56 -07:00
Mike Ellery
026a249173 Implement transaction invariant checks (RIPD-1425):
Add new functionality to enforce one or more sanity checks (invariants)
on transactions. Add tests for each new invariant check. Allow
for easily adding additional invariant checks in the future.

Also Resolves
-------------

  - RIPD-1426
  - RIPD-1427
  - RIPD-1428
  - RIPD-1429
  - RIPD-1430
  - RIPD-1431
  - RIPD-1432

Release Notes
-------------

Creates a new ammendment named "EnforceInvariants" which must be
enabled in order for these new checks to run on each transaction.
2017-04-19 12:24:49 -07:00
JoelKatz
e52614ac81 HTTPClient should support large replies (RIPD-1366):
A full ledger on the production Ripple network could
exceed the default maximum reply size for the
HTTPClient code. Remove the reply size maximum for
responses that include a Content-Length header.
2017-04-19 12:24:45 -07:00
JoelKatz
10a7f5b933 ledger_request should confirm ledger is present (RIPD-1365):
The ledger_request RPC call, under some conditions, did not
actually check that the entire ledger was present in the
database, making it unsuitable for use in cases where the
database was believed to be incorrect or incomplete.
With this change, the full ledger will be checked for
integrity unless it has already recently been checked
(according to the InboundLedgers cache).
2017-04-19 12:24:37 -07:00
Nik Bougalis
c6b6d82a75 Set version to 0.70.0-b3 2017-03-31 14:53:57 -07:00
Mike Ellery
9a0249e793 Add unit test for tx_history RPC (RIPD-1402) 2017-03-31 14:53:44 -07:00
Mike Ellery
e92760eec8 Add unit test for crypto_prng class 2017-03-31 14:53:44 -07:00
Mike Ellery
7b82051bdb Add test for feature RPC (RIPD-1391):
Create unit test for feature RPC method. Add client_error field to env
RPC requests to provide information about parsing errors.
2017-03-31 13:17:26 -07:00
Mike Ellery
aea54b7230 Add RPC filters for Escrow an PayChan (RIPD-1414) 2017-03-31 12:10:48 -07:00
Howard Hinnant
1a7a6f22e2 Add 'type' param to ledger_data and ledger rpc commands (RIPD-1446):
The 'type' field allows the rpc client to specify what type of ledger
entries to retrieve. The available types are:

    "account"
    "amendments"
    "directory"
    "fee"
    "hashes"
    "offer"
    "signer_list"
    "state"
    "ticket"
2017-03-31 12:10:11 -07:00
Edward Hennis
fab3ec0b56 CMake: build consistently with default (unspecified) target 2017-03-31 12:05:47 -07:00
Brad Chase
2449f9c18d Fix handleLCL consensus bug:
Consensus::checkLCL can change state_ but it was being called inside
timerEntry after a switch on the current state_.  In rare cases, this might
end up calling stateEstablish even when the state_ was open.
2017-03-31 11:54:51 -07:00
Nik Bougalis
fee30262ac Merge master (0.60.2) into develop (0.70.0-b2) 2017-03-31 11:53:49 -07:00
seelabs
7cd4d78897 Set version to 0.60.2 2017-03-30 14:43:14 -04:00
seelabs
4ff40d4954 Enforce rippling constraints between offers and direct steps 2017-03-30 14:42:01 -04:00
seelabs
0d4fe469c6 Set version to 0.60.1 2017-03-29 15:53:21 -04:00
Miguel Portilla
8b43d67a73 Cancel websocket timer on failure:
This fixes a problem where the timeout timer
would not be canceled with some errors.
fix #2026
2017-03-29 15:53:16 -04:00
Vinnie Falco
128f7cefb1 Send a websocket ping before timing out in server:
This fixes a problem where idle websocket client
connections could be disconnected due to inactivity.
2017-03-29 15:53:16 -04:00
seelabs
09f9720ebb Correctly calculate Escrow and PayChan identifiers:
This change fixes a technical flaw that resulted from using
32-bit space identifiers instead of the protocol-defined
16-bit values.

Details: https://ripple.com/build/ledger-format/#tree-format
2017-03-29 15:52:42 -04:00
Brad Chase
dbe74dffcb Set version to 0.70.0-b2 2017-03-21 19:14:26 -04:00
Brad Chase
b958fa413e Fix may be used uninitialized warnings 2017-03-21 19:14:21 -04:00
Scott Schurr
c453df927f NetworkOPs isn't stopped() until Jobs done (RIPD-1356):
A new JobCounter class is introduced.  The JobCounter keeps
a reference count of Jobs in flight to the JobQueue.  When
NetworkOPs needs to stop, in addition to other work, it calls
JobCounter::join(), which waits until all Jobs in flight
have been destroyed before returning.  This ensures that all
NetworkOPs Jobs are completed before NetworkOPs declares
itself stopped().

Also, once a JobCounter is join()ed, it refuses to produce
more counted Jobs for the JobQueue.  So, once all old Jobs
in flight are done, then NetworkOPs will add no additional
Jobs to the JobQueue.

Other classes besides NetworkOPs should also be able to use
JobCounter.  NetworkOPs is a first test case.

Also unneeded #includes were removed from files touched for
other reasons.
2017-03-21 18:55:05 -04:00
seelabs
1bb92d40aa Fix tx re-ordering bug in test:
`env.fund` requires two transactions: `pay` and `set account`. If there is a
`trust` transaction in the same set of txs, the txs may be reordered so
`pay` -> `trust` -> `set account` so the wrong `no ripple` flag would be used
on the trust line.

Adding a `close` between `env.fund` and `env.trust` resolves this problem.
2017-03-21 18:55:05 -04:00
Vinnie Falco
15f969a469 Send a websocket ping before timing out in server:
This fixes a problem where idle websocket client
connections could be disconnected due to inactivity.
2017-03-21 18:55:05 -04:00
Brad Chase
bc5a74057d Refactor consensus for simulation (RIPD-1011):
This is a substantial refactor of the consensus code and also introduces
a basic consensus simulation and testing framework.  The new generic/templated
version is in src/ripple/consensus and documents the current type requirements.
The version adapted for the RCL is in src/ripple/app/consensus.  The testing
framework is in src/test/csf.

Minor behavioral changes/fixes include:
* Adjust close time offset even when not validating.
* Remove spurious proposing_ = false call at end of handleLCL.
* Remove unused functionality provided by checkLastValidation.
* Separate open and converge time
* Don't send a bow out if we're not proposing
* Prevent consensus stopping if NetworkOPs switches to disconnect mode while
  consensus accepts a ledger
* Prevent a corner case in which Consensus::gotTxSet or Consensus::peerProposal
  has the potential to update internal state while an dispatched accept job is
  running.
* Distinguish external and internal calls to startNewRound.  Only external
  calls can reset the proposing_ state of consensus
2017-03-21 18:54:57 -04:00
Scott Schurr
fc0d64f5ee Set version to 0.70.0-b1 2017-03-20 19:10:26 -07:00
wilsonianb
885aaab8c8 Remove ledger and manifest Python tools 2017-03-20 18:58:50 -07:00
Scott Schurr
9d4500cf69 Prevent low-likelihood crash on shutdown (RIPD-1392):
The DatabaseImp has threads that asynchronously call JobQueue to
perform database reads.  Formerly these threads had the same
lifespan as Database, which was until the end-of-life of
ApplicationImp.  During shutdown these threads could call JobQueue
after JobQueue had already stopped.  Or, even worse, occasionally
call JobQueue after JobQueue's destructor had run.

To avoid these shutdown conditions, Database is made a Stoppable,
with JobQueue as its parent.  When Database stops, it shuts down
its asynchronous read threads.  This prevents Database from
accessing JobQueue after JobQueue has stopped, but allows
Database to perform stores for the remainder of shutdown.

During development it was noted that the Database::close()
method was never called.  So that method is removed from Database
and all derived classes.

Stoppable is also adjusted so it can be constructed using either
a char const* or a std::string.

For those files touched for other reasons, unneeded #includes
are removed.
2017-03-20 18:08:49 -07:00
Scott Schurr
9ff9fa0aea Prevent low-likelihood hang on shutdown (RIPD-1392):
Calling OverlayImpl::list_[].second->stop() may cause list_ to be
modified (OverlayImpl::remove() may be called on this same thread).
So iterating directly over OverlayImpl::list_ to call
OverlayImpl::list_[].second->stop() could give undefined behavior.
On MacOS that undefined behavior exhibited as a hang.

Therefore we copy all of the weak/shared ptrs out of
OverlayImpl::list_ before we start calling stop() on them.  That
guarantees OverlayImpl::remove() won't be called until
OverlayImpl::stop() completes.
2017-03-20 18:08:24 -07:00
Scott Schurr
1d482eeecb Prevent DatabaseRotateImp crash on shutdown (RIPD-1392):
The DatabaseImp holds threads that access DatabaseRotateImp.  But
the DatabaseRotateImp's destructor runs before the DatabaseImp
destructor.  The DatabaseRotateImp now assures that the
DatabaseImp threads are stopped before the DatabaseRotateImp
destructor completes.
2017-03-20 18:08:02 -07:00
Scott Schurr
b4e765362b Remove timing window from RootStoppable (RIPD-1392):
RootStoppable was using two separate flags to identify that it
was stopping.  LoadManager was being notified when one flag was
set, but checking the other flag (not yet set) to see if we were
stopping.  There is no strong motivation for two flags.  The
timing window is closed by removing one flag and moving around
a chunk of code.
2017-03-20 17:49:16 -07:00
Brad Chase
c981eb81d9 Improve log warnings:
Log non-account transaction in warning (RIPD-1440)
Log warning on PeerImp::fail (RIPD-1444)
2017-03-20 17:08:57 -07:00
Mike Ellery
95aebfc38c Add timer start param to Application (RIPD 1405):
Modify doStart Application method to specify whether or not to start the
DeadlineTimers. Specify inactive timers for jtx::Env Applications and
active timers for standard Applications.
2017-03-20 16:22:26 -07:00
Edward Hennis
7265729446 TxQ full queue RPC info (RIPD-1404):
* RPC `ledger` command returns all queue entries in "queue_data"
  when requesting open ledger, and including boolean "queue: true".
  * Includes queue state. e.g.: fee_level, retries, last_result, tx.
  * Respects "expand" and "binary" parameters for the txs.
* Remove some unused code.
2017-03-20 16:18:48 -07:00
seelabs
846723d771 New rules for payment paths:
* Sanity check on newly created strands
* Better loop detection
* Better tests (test every combination of path element pairs)
* Disallow any root issuer (even for xrp)
* Disallow compount element typs in path
* Issue was not reset when currency was XRP
* Add amendment
2017-03-20 14:56:40 -07:00
Mike Ellery
80d9b0464a Add helper to modify Env configs (RIPD-1247)
Add envconfig test helper for manipulating Env config via
callables. Create new common modifiers for non-admin config,
validator config and one for using different server port values.
2017-03-20 14:38:15 -07:00
David Schwartz
09a1d1a593 Improve getMissingNodes:
* Clean up and refactor
* Resume parents of nodes read asynchronously
* Resume at tip of new stack if exhausted prior stack
* No need to restart at root
2017-03-20 14:25:38 -07:00
JoelKatz
aebcc2115d Don't send a bow out if we're not proposing 2017-03-20 14:19:49 -07:00
David Schwartz
6fac038320 Make ledger fetch tuning saner 2017-03-20 14:12:06 -07:00
3920 changed files with 366190 additions and 940793 deletions

87
.clang-format Normal file
View File

@@ -0,0 +1,87 @@
---
Language: Cpp
AccessModifierOffset: -4
AlignAfterOpenBracket: AlwaysBreak
AlignConsecutiveAssignments: false
AlignConsecutiveDeclarations: false
AlignEscapedNewlinesLeft: true
AlignOperands: false
AlignTrailingComments: true
AllowAllParametersOfDeclarationOnNextLine: false
AllowShortBlocksOnASingleLine: false
AllowShortCaseLabelsOnASingleLine: false
AllowShortFunctionsOnASingleLine: false
AllowShortIfStatementsOnASingleLine: false
AllowShortLoopsOnASingleLine: false
AlwaysBreakAfterReturnType: All
AlwaysBreakBeforeMultilineStrings: true
AlwaysBreakTemplateDeclarations: true
BinPackArguments: false
BinPackParameters: false
BraceWrapping:
AfterClass: true
AfterControlStatement: true
AfterEnum: false
AfterFunction: true
AfterNamespace: false
AfterObjCDeclaration: true
AfterStruct: true
AfterUnion: true
BeforeCatch: true
BeforeElse: true
IndentBraces: false
BreakBeforeBinaryOperators: false
BreakBeforeBraces: Custom
BreakBeforeTernaryOperators: true
BreakConstructorInitializersBeforeComma: true
ColumnLimit: 80
CommentPragmas: '^ IWYU pragma:'
ConstructorInitializerAllOnOneLineOrOnePerLine: true
ConstructorInitializerIndentWidth: 4
ContinuationIndentWidth: 4
Cpp11BracedListStyle: true
DerivePointerAlignment: false
DisableFormat: false
ExperimentalAutoDetectBinPacking: false
ForEachMacros: [ Q_FOREACH, BOOST_FOREACH ]
IncludeCategories:
- Regex: '^<(BeastConfig)'
Priority: 0
- Regex: '^<(ripple)/'
Priority: 2
- Regex: '^<(boost)/'
Priority: 3
- Regex: '.*'
Priority: 4
IncludeIsMainRegex: '$'
IndentCaseLabels: true
IndentFunctionDeclarationAfterType: false
IndentWidth: 4
IndentWrappedFunctionNames: false
KeepEmptyLinesAtTheStartOfBlocks: false
MaxEmptyLinesToKeep: 1
NamespaceIndentation: None
ObjCSpaceAfterProperty: false
ObjCSpaceBeforeProtocolList: false
PenaltyBreakBeforeFirstCallParameter: 1
PenaltyBreakComment: 300
PenaltyBreakFirstLessLess: 120
PenaltyBreakString: 1000
PenaltyExcessCharacter: 1000000
PenaltyReturnTypeOnItsOwnLine: 200
PointerAlignment: Left
ReflowComments: true
SortIncludes: true
SpaceAfterCStyleCast: false
SpaceBeforeAssignmentOperators: true
SpaceBeforeParens: ControlStatements
SpaceInEmptyParentheses: false
SpacesBeforeTrailingComments: 2
SpacesInAngles: false
SpacesInContainerLiterals: true
SpacesInCStyleCastParentheses: false
SpacesInParentheses: false
SpacesInSquareBrackets: false
Standard: Cpp11
TabWidth: 8
UseTab: Never

6
.codecov.yml Normal file
View File

@@ -0,0 +1,6 @@
coverage:
status:
project:
default:
target: 60%
threshold: 2%

4
.git-blame-ignore-revs Normal file
View File

@@ -0,0 +1,4 @@
# This feature requires Git >= 2.24
# To use it by default in git blame:
# git config blame.ignoreRevsFile .git-blame-ignore-revs
50760c693510894ca368e90369b0cc2dabfd07f3

31
.github/ISSUE_TEMPLATE/bug_report.md vendored Normal file
View File

@@ -0,0 +1,31 @@
---
name: Bug Report
about: Create a report to help us improve rippled
title: "[Title with short description] (Version: [rippled version])"
labels: ''
assignees: ''
---
<!-- Please search existing issues to avoid creating duplicates.-->
## Issue Description
<!--Provide a summary for your issue/bug.-->
## Steps to Reproduce
<!--List in detail the exact steps to reproduce the unexpected behavior of the software.-->
## Expected Result
<!--Explain in detail what behavior you expected to happen.-->
## Actual Result
<!--Explain in detail what behavior actually happened.-->
## Environment
<!--Please describe your environment setup (such as Ubuntu 18.04 with Boost 1.70).-->
<!-- If you are using a formal release, please use the version returned by './rippled --version' as the version number-->
<!-- If you are working off of develop, please add the git hash via 'git rev-parse HEAD'-->
## Supporting Files
<!--If you have supporting files such as a log, feel free to post a link here using Github Gist.-->
<!--Consider adding configuration files with private information removed via Github Gist. -->

8
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@@ -0,0 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: XRP Ledger Documentation
url: https://xrpl.org/
about: All things about XRPL
- name: Security bug bounty program
url: https://ripple.com/bug-bounty/
about: Please report security-relevant bugs in our software here.

View File

@@ -0,0 +1,21 @@
---
name: Feature Request
about: Suggest a new feature for the rippled project
title: "[Title with short description] (Version: [rippled version])"
labels: Feature Request
assignees: ''
---
<!-- Please search existing issues to avoid creating duplicates.-->
## Summary
<!-- Provide a summary to the feature request-->
## Motivation
<!-- Why do we need this feature?-->
## Solution
<!-- What is the solution?-->
## Paths Not Taken
<!-- What other alternatives have been considered?-->

32
.github/actions/build/action.yml vendored Normal file
View File

@@ -0,0 +1,32 @@
name: build
inputs:
generator:
default: null
configuration:
required: true
cmake-args:
default: null
cmake-target:
default: all
# An implicit input is the environment variable `build_dir`.
runs:
using: composite
steps:
- name: configure
shell: bash
run: |
cd ${build_dir}
cmake \
${{ inputs.generator && format('-G "{0}"', inputs.generator) || '' }} \
-DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \
-DCMAKE_BUILD_TYPE=${{ inputs.configuration }} \
${{ inputs.cmake-args }} \
..
- name: build
shell: bash
run: |
cmake \
--build ${build_dir} \
--config ${{ inputs.configuration }} \
--parallel ${NUM_PROCESSORS:-$(nproc)} \
--target ${{ inputs.cmake-target }}

26
.github/actions/dependencies/action.yml vendored Normal file
View File

@@ -0,0 +1,26 @@
name: dependencies
inputs:
configuration:
required: true
# An implicit input is the environment variable `build_dir`.
runs:
using: composite
steps:
- name: unlock Conan
shell: bash
run: conan remove --locks
- name: export custom recipes
shell: bash
run: |
conan export external/snappy snappy/1.1.10@
conan export external/soci soci/4.0.3@
- name: install dependencies
shell: bash
run: |
mkdir ${build_dir}
cd ${build_dir}
conan install \
--output-folder . \
--build missing \
--settings build_type=${{ inputs.configuration }} \
..

79
.github/pull_request_template.md vendored Normal file
View File

@@ -0,0 +1,79 @@
<!--
This PR template helps you to write a good pull request description.
Please feel free to include additional useful information even beyond what is requested below.
-->
## High Level Overview of Change
<!--
Please include a summary of the changes.
This may be a direct input to the release notes.
If too broad, please consider splitting into multiple PRs.
If a relevant task or issue, please link it here.
-->
### Context of Change
<!--
Please include the context of a change.
If a bug fix, when was the bug introduced? What was the behavior?
If a new feature, why was this architecture chosen? What were the alternatives?
If a refactor, how is this better than the previous implementation?
If there is a spec or design document for this feature, please link it here.
-->
### Type of Change
<!--
Please check [x] relevant options, delete irrelevant ones.
-->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Refactor (non-breaking change that only restructures code)
- [ ] Performance (increase or change in throughput and/or latency)
- [ ] Tests (you added tests for code that already exists, or your new feature included in this PR)
- [ ] Documentation update
- [ ] Chore (no impact to binary, e.g. `.gitignore`, formatting, dropping support for older tooling)
- [ ] Release
### API Impact
<!--
Please check [x] relevant options, delete irrelevant ones.
* If there is any impact to the public API methods (HTTP / WebSocket), please update https://github.com/xrplf/rippled/blob/develop/API-CHANGELOG.md
* Update API-CHANGELOG.md and add the change directly in this PR by pushing to your PR branch.
* libxrpl: See https://github.com/XRPLF/rippled/blob/develop/docs/build/depend.md
* Peer Protocol: See https://xrpl.org/peer-protocol.html
-->
- [ ] Public API: New feature (new methods and/or new fields)
- [ ] Public API: Breaking change (in general, breaking changes should only impact the next api_version)
- [ ] `libxrpl` change (any change that may affect `libxrpl` or dependents of `libxrpl`)
- [ ] Peer protocol change (must be backward compatible or bump the peer protocol version)
<!--
## Before / After
If relevant, use this section for an English description of the change at a technical level.
If this change affects an API, examples should be included here.
For performance-impacting changes, please provide these details:
1. Is this a new feature, bug fix, or improvement to existing functionality?
2. What behavior/functionality does the change impact?
3. In what processing can the impact be measured? Be as specific as possible - e.g. RPC client call, payment transaction that involves LOB, AMM, caching, DB operations, etc.
4. Does this change affect concurrent processing - e.g. does it involve acquiring locks, multi-threaded processing, or async processing?
-->
<!--
## Test Plan
If helpful, please describe the tests that you ran to verify your changes and provide instructions so that others can reproduce.
This section may not be needed if your change includes thoroughly commented unit tests.
-->
<!--
## Future Tasks
For future tasks related to PR.
-->

61
.github/workflows/clang-format.yml vendored Normal file
View File

@@ -0,0 +1,61 @@
name: clang-format
on: [push, pull_request]
jobs:
check:
runs-on: ubuntu-20.04
env:
CLANG_VERSION: 10
steps:
- uses: actions/checkout@v3
- name: Install clang-format
run: |
codename=$( lsb_release --codename --short )
sudo tee /etc/apt/sources.list.d/llvm.list >/dev/null <<EOF
deb http://apt.llvm.org/${codename}/ llvm-toolchain-${codename}-${CLANG_VERSION} main
deb-src http://apt.llvm.org/${codename}/ llvm-toolchain-${codename}-${CLANG_VERSION} main
EOF
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add
sudo apt-get update
sudo apt-get install clang-format-${CLANG_VERSION}
- name: Format src/ripple
run: find src/ripple -type f \( -name '*.cpp' -o -name '*.h' -o -name '*.ipp' \) -print0 | xargs -0 clang-format-${CLANG_VERSION} -i
- name: Format src/test
run: find src/test -type f \( -name '*.cpp' -o -name '*.h' -o -name '*.ipp' \) -print0 | xargs -0 clang-format-${CLANG_VERSION} -i
- name: Check for differences
id: assert
run: |
set -o pipefail
git diff --exit-code | tee "clang-format.patch"
- name: Upload patch
if: failure() && steps.assert.outcome == 'failure'
uses: actions/upload-artifact@v2
continue-on-error: true
with:
name: clang-format.patch
if-no-files-found: ignore
path: clang-format.patch
- name: What happened?
if: failure() && steps.assert.outcome == 'failure'
env:
PREAMBLE: |
If you are reading this, you are looking at a failed Github Actions
job. That means you pushed one or more files that did not conform
to the formatting specified in .clang-format. That may be because
you neglected to run 'git clang-format' or 'clang-format' before
committing, or that your version of clang-format has an
incompatibility with the one on this
machine, which is:
SUGGESTION: |
To fix it, you can do one of two things:
1. Download and apply the patch generated as an artifact of this
job to your repo, commit, and push.
2. Run 'git-clang-format --extensions c,cpp,h,cxx,ipp develop'
in your repo, commit, and push.
run: |
echo "${PREAMBLE}"
clang-format-${CLANG_VERSION} --version
echo "${SUGGESTION}"
exit 1

37
.github/workflows/doxygen.yml vendored Normal file
View File

@@ -0,0 +1,37 @@
name: Build and publish Doxygen documentation
# To test this workflow, push your changes to your fork's `develop` branch.
on:
push:
branches:
- develop
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
job:
runs-on: ubuntu-latest
permissions:
contents: write
container:
image: docker://rippleci/rippled-ci-builder:2944b78d22db
steps:
- name: checkout
uses: actions/checkout@v3
- name: check environment
run: |
echo ${PATH} | tr ':' '\n'
cmake --version
doxygen --version
env | sort
- name: build
run: |
mkdir build
cd build
cmake -Donly_docs=TRUE ..
cmake --build . --target docs --parallel $(nproc)
- name: publish
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: build/docs/html

49
.github/workflows/levelization.yml vendored Normal file
View File

@@ -0,0 +1,49 @@
name: levelization
on: [push, pull_request]
jobs:
check:
runs-on: ubuntu-latest
env:
CLANG_VERSION: 10
steps:
- uses: actions/checkout@v3
- name: Check levelization
run: Builds/levelization/levelization.sh
- name: Check for differences
id: assert
run: |
set -o pipefail
git diff --exit-code | tee "levelization.patch"
- name: Upload patch
if: failure() && steps.assert.outcome == 'failure'
uses: actions/upload-artifact@v2
continue-on-error: true
with:
name: levelization.patch
if-no-files-found: ignore
path: levelization.patch
- name: What happened?
if: failure() && steps.assert.outcome == 'failure'
env:
MESSAGE: |
If you are reading this, you are looking at a failed Github
Actions job. That means you changed the dependency relationships
between the modules in rippled. That may be an improvement or a
regression. This check doesn't judge.
A rule of thumb, though, is that if your changes caused
something to be removed from loops.txt, that's probably an
improvement. If something was added, it's probably a regression.
To fix it, you can do one of two things:
1. Download and apply the patch generated as an artifact of this
job to your repo, commit, and push.
2. Run './Builds/levelization/levelization.sh' in your repo,
commit, and push.
See Builds/levelization/README.md for more info.
run: |
echo "${MESSAGE}"
exit 1

51
.github/workflows/macos.yml vendored Normal file
View File

@@ -0,0 +1,51 @@
name: macos
on: [push, pull_request]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
strategy:
matrix:
platform:
- macos
generator:
- Ninja
configuration:
- Release
runs-on: [self-hosted, macOS]
env:
# The `build` action requires these variables.
build_dir: .build
NUM_PROCESSORS: 12
steps:
- name: checkout
uses: actions/checkout@v3
- name: install Ninja
if: matrix.generator == 'Ninja'
run: brew install ninja
- name: check environment
run: |
echo ${PATH} | tr ':' '\n'
python --version
conan --version
cmake --version
env | sort
- name: configure Conan
run : |
conan profile get env.CXXFLAGS default || true
conan profile update 'conf.tools.build:cxxflags+=["-DBOOST_ASIO_DISABLE_CONCEPTS"]' default
- name: dependencies
uses: ./.github/actions/dependencies
with:
configuration: ${{ matrix.configuration }}
- name: build
uses: ./.github/actions/build
with:
generator: ${{ matrix.generator }}
configuration: ${{ matrix.configuration }}
- name: test
run: |
${build_dir}/rippled --unittest

234
.github/workflows/nix.yml vendored Normal file
View File

@@ -0,0 +1,234 @@
name: nix
on: [push, pull_request]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
# This workflow has two job matrixes.
# They can be considered phases because the second matrix ("test")
# depends on the first ("dependencies").
#
# The first phase has a job in the matrix for each combination of
# variables that affects dependency ABI:
# platform, compiler, and configuration.
# It creates a GitHub artifact holding the Conan profile,
# and builds and caches binaries for all the dependencies.
# If an Artifactory remote is configured, they are cached there.
# If not, they are added to the GitHub artifact.
# GitHub's "cache" action has a size limit (10 GB) that is too small
# to hold the binaries if they are built locally.
# We must use the "{upload,download}-artifact" actions instead.
#
# The second phase has a job in the matrix for each test configuration.
# It installs dependency binaries from the cache, whichever was used,
# and builds and tests rippled.
jobs:
dependencies:
strategy:
fail-fast: false
matrix:
platform:
- linux
compiler:
- gcc
- clang
configuration:
- Debug
- Release
include:
- compiler: gcc
profile:
version: 11
cc: /usr/bin/gcc
cxx: /usr/bin/g++
- compiler: clang
profile:
version: 14
cc: /usr/bin/clang-14
cxx: /usr/bin/clang++-14
runs-on: [self-hosted, heavy]
container: rippleci/rippled-build-ubuntu:aaf5e3e
env:
build_dir: .build
steps:
- name: checkout
uses: actions/checkout@v3
- name: check environment
run: |
echo ${PATH} | tr ':' '\n'
conan --version
cmake --version
env | sort
- name: configure Conan
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
run: |
conan profile new default --detect
conan profile update settings.compiler.cppstd=20 default
conan profile update settings.compiler=${{ matrix.compiler }} default
conan profile update settings.compiler.version=${{ matrix.profile.version }} default
conan profile update settings.compiler.libcxx=libstdc++11 default
conan profile update env.CC=${{ matrix.profile.cc }} default
conan profile update env.CXX=${{ matrix.profile.cxx }} default
conan profile update conf.tools.build:compiler_executables='{"c": "${{ matrix.profile.cc }}", "cpp": "${{ matrix.profile.cxx }}"}' default
# Do not quote the URL. An empty string will be accepted (with
# a non-fatal warning), but a missing argument will not.
conan remote add ripple ${{ env.CONAN_URL }} --insert 0
- name: try to authenticate to ripple Conan remote
id: remote
run: |
echo outcome=$(conan user --remote ripple ${{ secrets.CONAN_USERNAME }} --password ${{ secrets.CONAN_TOKEN }} >&2 && echo success || echo failure) | tee ${GITHUB_OUTPUT}
- name: archive profile
# Create this archive before dependencies are added to the local cache.
run: tar -czf conan.tar -C ~/.conan .
- name: list missing binaries
id: binaries
# Print the list of dependencies that would need to be built locally.
# A non-empty list means we have "failed" to cache binaries remotely.
run: |
echo missing=$(conan info . --build missing --settings build_type=${{ matrix.configuration }} --json 2>/dev/null | grep '^\[') | tee ${GITHUB_OUTPUT}
- name: build dependencies
if: (steps.binaries.outputs.missing != '[]')
uses: ./.github/actions/dependencies
with:
configuration: ${{ matrix.configuration }}
- name: upload dependencies to remote
if: (steps.binaries.outputs.missing != '[]') && (steps.remote.outputs.outcome == 'success')
run: conan upload --remote ripple '*' --all --parallel --confirm
- name: recreate archive with dependencies
if: (steps.binaries.outputs.missing != '[]') && (steps.remote.outputs.outcome == 'failure')
run: tar -czf conan.tar -C ~/.conan .
- name: upload archive
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.platform }}-${{ matrix.compiler }}-${{ matrix.configuration }}
path: conan.tar
if-no-files-found: error
test:
strategy:
fail-fast: false
matrix:
platform:
- linux
compiler:
- gcc
- clang
configuration:
- Debug
- Release
cmake-args:
-
- "-Dunity=ON"
needs: dependencies
runs-on: [self-hosted, heavy]
container: rippleci/rippled-build-ubuntu:aaf5e3e
env:
build_dir: .build
steps:
- name: download cache
uses: actions/download-artifact@v3
with:
name: ${{ matrix.platform }}-${{ matrix.compiler }}-${{ matrix.configuration }}
- name: extract cache
run: |
mkdir -p ~/.conan
tar -xzf conan.tar -C ~/.conan
- name: check environment
run: |
echo ${PATH} | tr ':' '\n'
conan --version
cmake --version
env | sort
ls ~/.conan
- name: checkout
uses: actions/checkout@v3
- name: dependencies
uses: ./.github/actions/dependencies
with:
configuration: ${{ matrix.configuration }}
- name: build
uses: ./.github/actions/build
with:
generator: Ninja
configuration: ${{ matrix.configuration }}
cmake-args: ${{ matrix.cmake-args }}
- name: test
run: |
${build_dir}/rippled --unittest --unittest-jobs $(nproc)
coverage:
strategy:
fail-fast: false
matrix:
platform:
- linux
compiler:
- gcc
configuration:
- Debug
needs: dependencies
runs-on: [self-hosted, heavy]
container: rippleci/rippled-build-ubuntu:aaf5e3e
env:
build_dir: .build
steps:
- name: download cache
uses: actions/download-artifact@v3
with:
name: ${{ matrix.platform }}-${{ matrix.compiler }}-${{ matrix.configuration }}
- name: extract cache
run: |
mkdir -p ~/.conan
tar -xzf conan.tar -C ~/.conan
- name: check environment
run: |
echo ${PATH} | tr ':' '\n'
conan --version
cmake --version
gcovr --version
env | sort
ls ~/.conan
- name: checkout
uses: actions/checkout@v3
- name: dependencies
uses: ./.github/actions/dependencies
with:
configuration: ${{ matrix.configuration }}
- name: build
uses: ./.github/actions/build
with:
generator: Ninja
configuration: ${{ matrix.configuration }}
cmake-args: >-
-Dcoverage=ON
-Dcoverage_format=xml
-DCODE_COVERAGE_VERBOSE=ON
-DCMAKE_CXX_FLAGS="-O0"
-DCMAKE_C_FLAGS="-O0"
cmake-target: coverage
- name: build
shell: bash
run: |
mv "${build_dir}/coverage.xml" ./
- name: archive coverage report
uses: actions/upload-artifact@v3
with:
name: coverage.xml
path: coverage.xml
retention-days: 30
- name: upload coverage report
uses: wandalen/wretry.action@v1.3.0
with:
action: codecov/codecov-action@v3
with: |
files: coverage.xml
fail_ci_if_error: true
verbose: true
token: ${{ secrets.CODECOV_TOKEN }}
attempt_limit: 5
attempt_delay: 35000 # in milliseconds

101
.github/workflows/windows.yml vendored Normal file
View File

@@ -0,0 +1,101 @@
name: windows
on: [push, pull_request]
# https://docs.github.com/en/actions/using-jobs/using-concurrency
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
strategy:
fail-fast: false
matrix:
generator:
- Visual Studio 16 2019
configuration:
- Release
# Github hosted runners tend to hang when running Debug unit tests.
# Instead of trying to work around it, disable the Debug job until
# something beefier (i.e. a heavy self-hosted runner) becomes
# available.
# - Debug
runs-on: windows-2019
env:
build_dir: .build
steps:
- name: checkout
uses: actions/checkout@v3
- name: choose Python
uses: actions/setup-python@v3
with:
python-version: 3.9
- name: learn Python cache directory
id: pip-cache
shell: bash
run: |
python -m pip install --upgrade pip
echo "dir=$(pip cache dir)" | tee ${GITHUB_OUTPUT}
- name: restore Python cache directory
uses: actions/cache@v3
with:
path: ${{ steps.pip-cache.outputs.dir }}
key: ${{ runner.os }}-${{ hashFiles('.github/workflows/windows.yml') }}
- name: install Conan
run: pip install wheel 'conan<2'
- name: check environment
run: |
$env:PATH -split ';'
python --version
conan --version
cmake --version
dir env:
- name: configure Conan
shell: bash
env:
CONAN_URL: http://18.143.149.228:8081/artifactory/api/conan/conan-non-prod
run: |
conan profile new default --detect
conan profile update settings.compiler.runtime=MT${{ matrix.configuration == 'Debug' && 'd' || '' }} default
# Do not quote the URL. An empty string will be accepted (with
# a non-fatal warning), but a missing argument will not.
conan remote add ripple ${{ env.CONAN_URL }} --insert 0
- name: try to authenticate to ripple Conan remote
shell: bash
id: remote
run: |
echo outcome=$(conan user --remote ripple ${{ secrets.CONAN_USERNAME }} \
--password ${{ secrets.CONAN_TOKEN }} >&2 && echo success || \
echo failure) | tee ${GITHUB_OUTPUT}
- name: list missing binaries
id: binaries
shell: bash
# Print the list of dependencies that would need to be built locally.
# A non-empty list means we have "failed" to cache binaries remotely.
run: |
echo missing=$(conan info . --build missing --settings build_type=${{ matrix.configuration }} --json 2>/dev/null | grep '^\[') | tee ${GITHUB_OUTPUT}
- name: build dependencies
uses: ./.github/actions/dependencies
with:
configuration: ${{ matrix.configuration }}
- name: upload dependencies to remote
if: (steps.binaries.outputs.missing != '[]') && (steps.remote.outputs.outcome == 'success')
run: conan upload --remote ripple '*' --all --parallel --confirm
- name: build
uses: ./.github/actions/build
with:
generator: '${{ matrix.generator }}'
configuration: ${{ matrix.configuration }}
# Hard code for now. Move to the matrix if varied options are needed
cmake-args: '-Dassert=ON -Dreporting=OFF -Dunity=ON'
cmake-target: install
- name: test (permitted to silently fail)
shell: bash
# Github runners are resource limited, which causes unit tests to fail
# (e.g. OOM). To allow forward progress until self-hosted runners are
# up and running reliably, allow the job to succeed even if tests fail.
continue-on-error: true
run: |
${build_dir}/${{ matrix.configuration }}/rippled --unittest --unittest-jobs $(nproc)

28
.gitignore vendored
View File

@@ -21,9 +21,12 @@ bin/project-cache.jam
# Ignore object files.
*.o
build
.nih_c
tags
TAGS
GTAGS
GRTAGS
GPATH
bin/rippled
Debug/*.*
Release/*.*
@@ -33,8 +36,11 @@ Release/*.*
*.gcda
*.gcov
# Ignore locally installed node_modules
/node_modules
# Levelization checking
Builds/levelization/results/rawincludes.txt
Builds/levelization/results/paths.txt
Builds/levelization/results/includes/
Builds/levelization/results/includedby/
# Ignore tmp directory.
tmp
@@ -50,15 +56,15 @@ debug_log.txt
# Ignore customized configs
rippled.cfg
validators.txt
test/config.js
# Doxygen generated documentation output
HtmlDocumentation
docs/html_doc
# Xcode user-specific project settings
# Xcode
.DS_Store
*/build/*
/build/
*.pbxuser
!default.pbxuser
*.mode1v3
@@ -94,3 +100,15 @@ Builds/VisualStudio2015/*.sdf
# MSVC
*.pdb
.vs/
CMakeSettings.json
compile_commands.json
.clangd
packages
pkg_out
pkg
CMakeUserPresets.json
bld.rippled/
.vscode
# Suggested in-tree build directory
/.build/

12
.gitmodules vendored
View File

@@ -1,12 +0,0 @@
[submodule "docs/docca"]
path = docs/docca
url = https://github.com/vinniefalco/docca.git
[submodule "src/nudb/extras/beast"]
path = src/nudb/extras/beast
url = https://github.com/vinniefalco/Beast.git
[submodule "src/nudb/extras/rocksdb"]
path = src/nudb/extras/rocksdb
url = https://github.com/facebook/rocksdb.git
[submodule "src/nudb/doc/docca"]
path = src/nudb/doc/docca
url = https://github.com/vinniefalco/docca.git

6
.pre-commit-config.yaml Normal file
View File

@@ -0,0 +1,6 @@
# .pre-commit-config.yaml
repos:
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v10.0.1
hooks:
- id: clang-format

View File

@@ -1,71 +0,0 @@
sudo: false
language: cpp
env:
global:
- LLVM_VERSION=3.8.0
# Maintenance note: to move to a new version
# of boost, update both BOOST_ROOT and BOOST_URL.
# Note that for simplicity, BOOST_ROOT's final
# namepart must match the folder name internal
# to boost's .tar.gz.
- LCOV_ROOT=$HOME/lcov
- BOOST_ROOT=$HOME/boost_1_60_0
- BOOST_URL='http://sourceforge.net/projects/boost/files/boost/1.60.0/boost_1_60_0.tar.gz'
addons:
apt:
sources: ['ubuntu-toolchain-r-test']
packages:
- gcc-5
- g++-5
- python-software-properties
- protobuf-compiler
- libprotobuf-dev
- libssl-dev
- libstdc++6
- binutils-gold
# Provides a backtrace if the unittests crash
- gdb
matrix:
include:
# Default BUILD is "scons".
- compiler: gcc
env: GCC_VER=5 BUILD=cmake TARGET=debug.nounity PATH=$PWD/cmake/bin:$PATH
- compiler: gcc
env: GCC_VER=5 TARGET=coverage
- compiler: clang
env: GCC_VER=5 TARGET=debug CLANG_VER=3.8 PATH=$PWD/llvm-$LLVM_VERSION/bin:$PATH
- compiler: clang
env: GCC_VER=5 TARGET=debug.nounity CLANG_VER=3.8 PATH=$PWD/llvm-$LLVM_VERSION/bin:$PATH
# The clang cmake builds do not link.
# - compiler: clang
# env: GCC_VER=5 BUILD=cmake TARGET=debug CLANG_VER=3.8 PATH=$PWD/llvm-$LLVM_VERSION/bin:$PWD/cmake/bin:$PATH
# - compiler: clang
# env: GCC_VER=5 BUILD=cmake TARGET=debug.nounity CLANG_VER=3.8 PATH=$PWD/llvm-$LLVM_VERSION/bin:$PWD/cmake/bin:$PATH
cache:
directories:
- $BOOST_ROOT
- llvm-$LLVM_VERSION
- cmake
before_install:
- bin/ci/ubuntu/install-dependencies.sh
script:
- travis_retry bin/ci/ubuntu/build-and-test.sh
notifications:
email:
false
irc:
channels:
- "chat.freenode.net#ripple-dev"

194
API-CHANGELOG.md Normal file
View File

@@ -0,0 +1,194 @@
# API Changelog
This changelog is intended to list all updates to the [public API methods](https://xrpl.org/public-api-methods.html).
For info about how [API versioning](https://xrpl.org/request-formatting.html#api-versioning) works, including examples, please view the [XLS-22d spec](https://github.com/XRPLF/XRPL-Standards/discussions/54). For details about the implementation of API versioning, view the [implementation PR](https://github.com/XRPLF/rippled/pull/3155). API versioning ensures existing integrations and users continue to receive existing behavior, while those that request a higher API version will experience new behavior.
The API version controls the API behavior you see. This includes what properties you see in responses, what parameters you're permitted to send in requests, and so on. You specify the API version in each of your requests. When a breaking change is introduced to the `rippled` API, a new version is released. To avoid breaking your code, you should set (or increase) your version when you're ready to upgrade.
For a log of breaking changes, see the **API Version [number]** headings. In general, breaking changes are associated with a particular API Version number. For non-breaking changes, scroll to the **XRP Ledger version [x.y.z]** headings. Non-breaking changes are associated with a particular XRP Ledger (`rippled`) release.
## API Version 1
This version is supported by all `rippled` versions. At time of writing, it is the default API version, used when no `api_version` is specified. When a new API version is introduced, the command line interface will default to the latest API version. The command line is intended for ad-hoc usage by humans, not programs or automated scripts. The command line is not meant for use in production code.
### Idiosyncrasies
#### V1 account_info response
In [the response to the `account_info` command](https://xrpl.org/account_info.html#response-format), there is `account_data` - which is supposed to be an `AccountRoot` object - and `signer_lists` is returned in this object. However, the docs say that `signer_lists` should be at the root level of the reponse.
It makes sense for `signer_lists` to be at the root level because signer lists are not part of the AccountRoot object. (First reported in [xrpl-dev-portal#938](https://github.com/XRPLF/xrpl-dev-portal/issues/938).)
In `api_version: 2`, the `signer_lists` field [will be moved](#modifications-to-account_info-response-in-v2) to the root level of the account_info response. (https://github.com/XRPLF/rippled/pull/3770)
#### server_info - network_id
The `network_id` field was added in the `server_info` response in version 1.5.0 (2019), but it is not returned in [reporting mode](https://xrpl.org/rippled-server-modes.html#reporting-mode).
## XRP Ledger server version 2.0.0
### Additions in 2.0
Additions are intended to be non-breaking (because they are purely additive).
- `server_definitions`: A new RPC that generates a `definitions.json`-like output that can be used in XRPL libraries.
- In `Payment` transactions, `DeliverMax` has been added. This is a replacement for the `Amount` field, which should not be used. Typically, the `delivered_amount` (in transaction metadata) should be used. To ease the transition, `DeliverMax` is present regardless of API version, since adding a field is non-breaking.
- API version 2 has been moved from beta to supported, meaning that it is generally available (regardless of the `beta_rpc_api` setting).
## XRP Ledger server version 1.12.0
[Version 1.12.0](https://github.com/XRPLF/rippled/releases/tag/1.12.0) was released on Sep 6, 2023.
### Additions in 1.12
Additions are intended to be non-breaking (because they are purely additive).
- `server_info`: Added `ports`, an array which advertises the RPC and WebSocket ports. This information is also included in the `/crawl` endpoint (which calls `server_info` internally). `grpc` and `peer` ports are also included. (https://github.com/XRPLF/rippled/pull/4427)
- `ports` contains objects, each containing a `port` for the listening port (a number string), and a `protocol` array listing the supported protocols on that port.
- This allows crawlers to build a more detailed topology without needing to port-scan nodes.
- (For peers and other non-admin clients, the info about admin ports is excluded.)
- Clawback: The following additions are gated by the Clawback amendment (`featureClawback`). (https://github.com/XRPLF/rippled/pull/4553)
- Adds an [AccountRoot flag](https://xrpl.org/accountroot.html#accountroot-flags) called `lsfAllowTrustLineClawback` (https://github.com/XRPLF/rippled/pull/4617)
- Adds the corresponding `asfAllowTrustLineClawback` [AccountSet Flag](https://xrpl.org/accountset.html#accountset-flags) as well.
- Clawback is disabled by default, so if an issuer desires the ability to claw back funds, they must use an `AccountSet` transaction to set the AllowTrustLineClawback flag. They must do this before creating any trust lines, offers, escrows, payment channels, or checks.
- Adds the [Clawback transaction type](https://github.com/XRPLF/XRPL-Standards/blob/master/XLS-39d-clawback/README.md#331-clawback-transaction), containing these fields:
- `Account`: The issuer of the asset being clawed back. Must also be the sender of the transaction.
- `Amount`: The amount being clawed back, with the `Amount.issuer` being the token holder's address.
- Adds [AMM](https://github.com/XRPLF/XRPL-Standards/discussions/78) ([#4294](https://github.com/XRPLF/rippled/pull/4294), [#4626](https://github.com/XRPLF/rippled/pull/4626)) feature:
- Adds `amm_info` API to retrieve AMM information for a given tokens pair.
- Adds `AMMCreate` transaction type to create `AMM` instance.
- Adds `AMMDeposit` transaction type to deposit funds into `AMM` instance.
- Adds `AMMWithdraw` transaction type to withdraw funds from `AMM` instance.
- Adds `AMMVote` transaction type to vote for the trading fee of `AMM` instance.
- Adds `AMMBid` transaction type to bid for the Auction Slot of `AMM` instance.
- Adds `AMMDelete` transaction type to delete `AMM` instance.
- Adds `sfAMMID` to `AccountRoot` to indicate that the account is `AMM`'s account. `AMMID` is used to fetch `ltAMM`.
- Adds `lsfAMMNode` `TrustLine` flag to indicate that one side of the `TrustLine` is `AMM` account.
- Adds `tfLPToken`, `tfSingleAsset`, `tfTwoAsset`, `tfOneAssetLPToken`, `tfLimitLPToken`, `tfTwoAssetIfEmpty`,
`tfWithdrawAll`, `tfOneAssetWithdrawAll` which allow a trader to specify different fields combination
for `AMMDeposit` and `AMMWithdraw` transactions.
- Adds new transaction result codes:
- tecUNFUNDED_AMM: insufficient balance to fund AMM. The account does not have funds for liquidity provision.
- tecAMM_BALANCE: AMM has invalid balance. Calculated balances greater than the current pool balances.
- tecAMM_FAILED: AMM transaction failed. Fails due to a processing failure.
- tecAMM_INVALID_TOKENS: AMM invalid LP tokens. Invalid input values, format, or calculated values.
- tecAMM_EMPTY: AMM is in empty state. Transaction requires AMM in non-empty state (LP tokens > 0).
- tecAMM_NOT_EMPTY: AMM is not in empty state. Transaction requires AMM in empty state (LP tokens == 0).
- tecAMM_ACCOUNT: AMM account. Clawback of AMM account.
- tecINCOMPLETE: Some work was completed, but more submissions required to finish. AMMDelete partially deletes the trustlines.
## XRP Ledger server version 1.11.0
[Version 1.11.0](https://github.com/XRPLF/rippled/releases/tag/1.11.0) was released on Jun 20, 2023.
### Breaking changes in 1.11
- Added the ability to mark amendments as obsolete. For the `feature` admin API, there is a new possible value for the `vetoed` field. (https://github.com/XRPLF/rippled/pull/4291)
- The value of `vetoed` can now be `true`, `false`, or `"Obsolete"`.
- Removed the acceptance of seeds or public keys in place of account addresses. (https://github.com/XRPLF/rippled/pull/4404)
- This simplifies the API and encourages better security practices (i.e. seeds should never be sent over the network).
- For the `ledger_data` method, when all entries are filtered out, the `state` field of the response is now an empty list (in other words, an empty array, `[]`). (Previously, it would return `null`.) While this is technically a breaking change, the new behavior is consistent with the documentation, so this is considered only a bug fix. (https://github.com/XRPLF/rippled/pull/4398)
- If and when the `fixNFTokenRemint` amendment activates, there will be a new AccountRoot field, `FirstNFTSequence`. This field is set to the current account sequence when the account issues their first NFT. If an account has not issued any NFTs, then the field is not set. ([#4406](https://github.com/XRPLF/rippled/pull/4406))
- There is a new account deletion restriction: an account can only be deleted if `FirstNFTSequence` + `MintedNFTokens` + `256` is less than the current ledger sequence.
- This is potentially a breaking change if clients have logic for determining whether an account can be deleted.
- NetworkID
- For sidechains and networks with a network ID greater than 1024, there is a new [transaction common field](https://xrpl.org/transaction-common-fields.html), `NetworkID`. (https://github.com/XRPLF/rippled/pull/4370)
- This field helps to prevent replay attacks and is now required for chains whose network ID is 1025 or higher.
- The field must be omitted for Mainnet, so there is no change for Mainnet users.
- There are three new local error codes:
- `telNETWORK_ID_MAKES_TX_NON_CANONICAL`: a `NetworkID` is present but the chain's network ID is less than 1025. Remove the field from the transaction, and try again.
- `telREQUIRES_NETWORK_ID`: a `NetworkID` is required, but is not present. Add the field to the transaction, and try again.
- `telWRONG_NETWORK`: a `NetworkID` is specified, but it is for a different network. Submit the transaction to a different server which is connected to the correct network.
### Additions and bug fixes in 1.11
- Added `nftoken_id`, `nftoken_ids` and `offer_id` meta fields into NFT `tx` and `account_tx` responses. (https://github.com/XRPLF/rippled/pull/4447)
- Added an `account_flags` object to the `account_info` method response. (https://github.com/XRPLF/rippled/pull/4459)
- Added `NFTokenPages` to the `account_objects` RPC. (https://github.com/XRPLF/rippled/pull/4352)
- Fixed: `marker` returned from the `account_lines` command would not work on subsequent commands. (https://github.com/XRPLF/rippled/pull/4361)
## XRP Ledger server version 1.10.0
[Version 1.10.0](https://github.com/XRPLF/rippled/releases/tag/1.10.0)
was released on Mar 14, 2023.
### Breaking changes in 1.10
- If the `XRPFees` feature is enabled, the `fee_ref` field will be
removed from the [ledger subscription stream](https://xrpl.org/subscribe.html#ledger-stream), because it will no longer
have any meaning.
## API Version 2
API version 2 is introduced in `rippled` version 2.0. Users can request it explicitly by specifying `"api_version" : 2`.
#### Removed methods
In API version 2, the following deprecated methods are no longer available: (https://github.com/XRPLF/rippled/pull/4759)
- `tx_history` - Instead, use other methods such as `account_tx` or `ledger` with the `transactions` field set to `true`.
- `ledger_header` - Instead, use the `ledger` method.
#### Modifications to JSON transaction element in V2
In API version 2, JSON elements for transaction output have been changed and made consistent for all methods which output transactions. (https://github.com/XRPLF/rippled/pull/4775)
This helps to unify the JSON serialization format of transactions. (https://github.com/XRPLF/clio/issues/722, https://github.com/XRPLF/rippled/issues/4727)
- JSON transaction element is named `tx_json`
- Binary transaction element is named `tx_blob`
- JSON transaction metadata element is named `meta`
- Binary transaction metadata element is named `meta_blob`
Additionally, these elements are now consistently available next to `tx_json` (i.e. sibling elements), where possible:
- `hash` - Transaction ID. This data was stored inside transaction output in API version 1, but in API version 2 is a sibling element.
- `ledger_index` - Ledger index (only set on validated ledgers)
- `ledger_hash` - Ledger hash (only set on closed or validated ledgers)
- `close_time_iso` - Ledger close time expressed in ISO 8601 time format (only set on validated ledgers)
- `validated` - Bool element set to `true` if the transaction is in a validated ledger, otherwise `false`
This change affects the following methods:
- `tx` - Transaction data moved into element `tx_json` (was inline inside `result`) or, if binary output was requested, moved from `tx` to `tx_blob`. Renamed binary transaction metadata element (if it was requested) from `meta` to `meta_blob`. Changed location of `hash` and added new elements
- `account_tx` - Renamed transaction element from `tx` to `tx_json`. Renamed binary transaction metadata element (if it was requested) from `meta` to `meta_blob`. Changed location of `hash` and added new elements
- `transaction_entry` - Renamed transaction metadata element from `metadata` to `meta`. Changed location of `hash` and added new elements
- `subscribe` - Renamed transaction element from `transaction` to `tx_json`. Changed location of `hash` and added new elements
- `sign`, `sign_for`, `submit` and `submit_multisigned` - Changed location of `hash` element.
#### Modification to `Payment` transaction JSON schema
When reading Payments, the `Amount` field should generally **not** be used. Instead, use [delivered_amount](https://xrpl.org/partial-payments.html#the-delivered_amount-field) to see the amount that the Payment delivered. To clarify its meaning, the `Amount` field is being renamed to `DeliverMax`. (https://github.com/XRPLF/rippled/pull/4733)
- In `Payment` transaction type, JSON RPC field `Amount` is renamed to `DeliverMax`. To enable smooth client transition, `Amount` is still handled, as described below: (https://github.com/XRPLF/rippled/pull/4733)
- On JSON RPC input (e.g. `submit_multisigned` etc. methods), `Amount` is recognized as an alias to `DeliverMax` for both API version 1 and version 2 clients.
- On JSON RPC input, submitting both `Amount` and `DeliverMax` fields is allowed _only_ if they are identical; otherwise such input is rejected with `rpcINVALID_PARAMS` error.
- On JSON RPC output (e.g. `subscribe`, `account_tx` etc. methods), `DeliverMax` is present in both API version 1 and version 2.
- On JSON RPC output, `Amount` is only present in API version 1 and _not_ in version 2.
#### Modifications to account_info response
- `signer_lists` is returned in the root of the response. In API version 1, it was nested under `account_data`. (https://github.com/XRPLF/rippled/pull/3770)
- When using an invalid `signer_lists` value, the API now returns an "invalidParams" error. (https://github.com/XRPLF/rippled/pull/4585)
- (`signer_lists` must be a boolean. In API version 1, strings were accepted and may return a normal response - i.e. as if `signer_lists` were `true`.)
#### Modifications to [account_tx](https://xrpl.org/account_tx.html#account_tx) response
- Using `ledger_index_min`, `ledger_index_max`, and `ledger_index` returns `invalidParams` because if you use `ledger_index_min` or `ledger_index_max`, then it does not make sense to also specify `ledger_index`. In API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4571)
- The same applies for `ledger_index_min`, `ledger_index_max`, and `ledger_hash`. (https://github.com/XRPLF/rippled/issues/4545#issuecomment-1565065579)
- Using a `ledger_index_min` or `ledger_index_max` beyond the range of ledgers that the server has:
- returns `lgrIdxMalformed` in API version 2. Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/issues/4288)
- Attempting to use a non-boolean value (such as a string) for the `binary` or `forward` parameters returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4620)
#### Modifications to [noripple_check](https://xrpl.org/noripple_check.html#noripple_check) response
- Attempting to use a non-boolean value (such as a string) for the `transactions` parameter returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4620)
# Unit tests for API changes
The following information is useful to developers contributing to this project:
The purpose of unit tests is to catch bugs and prevent regressions. In general, it often makes sense to create a test function when there is a breaking change to the API. For APIs that have changed in a new API version, the tests should be modified so that both the prior version and the new version are properly tested.
To take one example: for `account_info` version 1, WebSocket and JSON-RPC behavior should be tested. The latest API version, i.e. API version 2, should be tested over WebSocket, JSON-RPC, and command line.

428
BUILD.md Normal file
View File

@@ -0,0 +1,428 @@
| :warning: **WARNING** :warning:
|---|
| These instructions assume you have a C++ development environment ready with Git, Python, Conan, CMake, and a C++ compiler. For help setting one up on Linux, macOS, or Windows, [see this guide](./docs/build/environment.md). |
> These instructions also assume a basic familiarity with Conan and CMake.
> If you are unfamiliar with Conan,
> you can read our [crash course](./docs/build/conan.md)
> or the official [Getting Started][3] walkthrough.
## Branches
For a stable release, choose the `master` branch or one of the [tagged
releases](https://github.com/ripple/rippled/releases).
```
git checkout master
```
For the latest release candidate, choose the `release` branch.
```
git checkout release
```
For the latest set of untested features, or to contribute, choose the `develop`
branch.
```
git checkout develop
```
## Minimum Requirements
See [System Requirements](https://xrpl.org/system-requirements.html).
Building rippled generally requires git, Python, Conan, CMake, and a C++ compiler. Some guidance on setting up such a [C++ development environment can be found here](./docs/build/environment.md).
- [Python 3.7](https://www.python.org/downloads/)
- [Conan 1.55](https://conan.io/downloads.html)
- [CMake 3.16](https://cmake.org/download/)
`rippled` is written in the C++20 dialect and includes the `<concepts>` header.
The [minimum compiler versions][2] required are:
| Compiler | Version |
|-------------|---------|
| GCC | 11 |
| Clang | 13 |
| Apple Clang | 13.1.6 |
| MSVC | 19.23 |
### Linux
The Ubuntu operating system has received the highest level of
quality assurance, testing, and support.
Here are [sample instructions for setting up a C++ development environment on Linux](./docs/build/environment.md#linux).
### Mac
Many rippled engineers use macOS for development.
Here are [sample instructions for setting up a C++ development environment on macOS](./docs/build/environment.md#macos).
### Windows
Windows is not recommended for production use at this time.
- Additionally, 32-bit Windows development is not supported.
- Visual Studio 2022 is not yet supported.
- rippled generally requires [Boost][] 1.77, which Conan cannot build with VS 2022.
- Until rippled is updated for compatibility with later versions of Boost, Windows developers may need to use Visual Studio 2019.
[Boost]: https://www.boost.org/
## Steps
### Set Up Conan
After you have a [C++ development environment](./docs/build/environment.md) ready with Git, Python, Conan, CMake, and a C++ compiler, you may need to set up your Conan profile.
These instructions assume a basic familiarity with Conan and CMake.
If you are unfamiliar with Conan, then please read [this crash course](./docs/build/conan.md) or the official [Getting Started][3] walkthrough.
You'll need at least one Conan profile:
```
conan profile new default --detect
```
Update the compiler settings:
```
conan profile update settings.compiler.cppstd=20 default
```
**Linux** developers will commonly have a default Conan [profile][] that compiles
with GCC and links with libstdc++.
If you are linking with libstdc++ (see profile setting `compiler.libcxx`),
then you will need to choose the `libstdc++11` ABI:
```
conan profile update settings.compiler.libcxx=libstdc++11 default
```
**Windows** developers may need to use the x64 native build tools.
An easy way to do that is to run the shortcut "x64 Native Tools Command
Prompt" for the version of Visual Studio that you have installed.
Windows developers must also build `rippled` and its dependencies for the x64
architecture:
```
conan profile update settings.arch=x86_64 default
```
### Multiple compilers
When `/usr/bin/g++` exists on a platform, it is the default cpp compiler. This
default works for some users.
However, if this compiler cannot build rippled or its dependencies, then you can
install another compiler and set Conan and CMake to use it.
Update the `conf.tools.build:compiler_executables` setting in order to set the correct variables (`CMAKE_<LANG>_COMPILER`) in the
generated CMake toolchain file.
For example, on Ubuntu 20, you may have gcc at `/usr/bin/gcc` and g++ at `/usr/bin/g++`; if that is the case, you can select those compilers with:
```
conan profile update 'conf.tools.build:compiler_executables={"c": "/usr/bin/gcc", "cpp": "/usr/bin/g++"}' default
```
Replace `/usr/bin/gcc` and `/usr/bin/g++` with paths to the desired compilers.
It should choose the compiler for dependencies as well,
but not all of them have a Conan recipe that respects this setting (yet).
For the rest, you can set these environment variables.
Replace `<path>` with paths to the desired compilers:
- `conan profile update env.CC=<path> default`
- `conan profile update env.CXX=<path> default`
Export our [Conan recipe for Snappy](./external/snappy).
It does not explicitly link the C++ standard library,
which allows you to statically link it with GCC, if you want.
```
conan export external/snappy snappy/1.1.10@
```
Export our [Conan recipe for SOCI](./external/soci).
It patches their CMake to correctly import its dependencies.
```
conan export external/soci soci/4.0.3@
```
### Build and Test
1. Create a build directory and move into it.
```
mkdir .build
cd .build
```
You can use any directory name. Conan treats your working directory as an
install folder and generates files with implementation details.
You don't need to worry about these files, but make sure to change
your working directory to your build directory before calling Conan.
**Note:** You can specify a directory for the installation files by adding
the `install-folder` or `-if` option to every `conan install` command
in the next step.
2. Generate CMake files for every configuration you want to build.
```
conan install .. --output-folder . --build missing --settings build_type=Release
conan install .. --output-folder . --build missing --settings build_type=Debug
```
For a single-configuration generator, e.g. `Unix Makefiles` or `Ninja`,
you only need to run this command once.
For a multi-configuration generator, e.g. `Visual Studio`, you may want to
run it more than once.
Each of these commands should also have a different `build_type` setting.
A second command with the same `build_type` setting will overwrite the files
generated by the first. You can pass the build type on the command line with
`--settings build_type=$BUILD_TYPE` or in the profile itself,
under the section `[settings]` with the key `build_type`.
If you are using a Microsoft Visual C++ compiler,
then you will need to ensure consistency between the `build_type` setting
and the `compiler.runtime` setting.
When `build_type` is `Release`, `compiler.runtime` should be `MT`.
When `build_type` is `Debug`, `compiler.runtime` should be `MTd`.
```
conan install .. --output-folder . --build missing --settings build_type=Release --settings compiler.runtime=MT
conan install .. --output-folder . --build missing --settings build_type=Debug --settings compiler.runtime=MTd
```
3. Configure CMake and pass the toolchain file generated by Conan, located at
`$OUTPUT_FOLDER/build/generators/conan_toolchain.cmake`.
Single-config generators:
```
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release ..
```
Pass the CMake variable [`CMAKE_BUILD_TYPE`][build_type]
and make sure it matches the `build_type` setting you chose in the previous
step.
Multi-config generators:
```
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake ..
```
**Note:** You can pass build options for `rippled` in this step.
4. Build `rippled`.
For a single-configuration generator, it will build whatever configuration
you passed for `CMAKE_BUILD_TYPE`. For a multi-configuration generator,
you must pass the option `--config` to select the build configuration.
Single-config generators:
```
cmake --build .
```
Multi-config generators:
```
cmake --build . --config Release
cmake --build . --config Debug
```
5. Test rippled.
Single-config generators:
```
./rippled --unittest
```
Multi-config generators:
```
./Release/rippled --unittest
./Debug/rippled --unittest
```
The location of `rippled` in your build directory depends on your CMake
generator. Pass `--help` to see the rest of the command line options.
## Coverage report
The coverage report is intended for developers using compilers GCC
or Clang (including Apple Clang). It is generated by the build target `coverage`,
which is only enabled when the `coverage` option is set, e.g. with
`--options coverage=True` in `conan` or `-Dcoverage=ON` variable in `cmake`
Prerequisites for the coverage report:
- [gcovr tool][gcovr] (can be installed e.g. with [pip][python-pip])
- `gcov` for GCC (installed with the compiler by default) or
- `llvm-cov` for Clang (installed with the compiler by default)
- `Debug` build type
A coverage report is created when the following steps are completed, in order:
1. `rippled` binary built with instrumentation data, enabled by the `coverage`
option mentioned above
2. completed run of unit tests, which populates coverage capture data
3. completed run of the `gcovr` tool (which internally invokes either `gcov` or `llvm-cov`)
to assemble both instrumentation data and the coverage capture data into a coverage report
The above steps are automated into a single target `coverage`. The instrumented
`rippled` binary can also be used for regular development or testing work, at
the cost of extra disk space utilization and a small performance hit
(to store coverage capture). In case of a spurious failure of unit tests, it is
possible to re-run the `coverage` target without rebuilding the `rippled` binary
(since it is simply a dependency of the coverage report target). It is also possible
to select only specific tests for the purpose of the coverage report, by setting
the `coverage_test` variable in `cmake`
The default coverage report format is `html-details`, but the user
can override it to any of the formats listed in `Builds/CMake/CodeCoverage.cmake`
by setting the `coverage_format` variable in `cmake`. It is also possible
to generate more than one format at a time by setting the `coverage_extra_args`
variable in `cmake`. The specific command line used to run the `gcovr` tool will be
displayed if the `CODE_COVERAGE_VERBOSE` variable is set.
By default, the code coverage tool runs parallel unit tests with `--unittest-jobs`
set to the number of available CPU cores. This may cause spurious test
errors on Apple. Developers can override the number of unit test jobs with
the `coverage_test_parallelism` variable in `cmake`.
Example use with some cmake variables set:
```
cd .build
conan install .. --output-folder . --build missing --settings build_type=Debug
cmake -DCMAKE_BUILD_TYPE=Debug -Dcoverage=ON -Dcoverage_test_parallelism=2 -Dcoverage_format=html-details -Dcoverage_extra_args="--json coverage.json" -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake ..
cmake --build . --target coverage
```
After the `coverage` target is completed, the generated coverage report will be
stored inside the build directory, as either of:
- file named `coverage.`_extension_ , with a suitable extension for the report format, or
- directory named `coverage`, with the `index.html` and other files inside, for the `html-details` or `html-nested` report formats.
## Options
| Option | Default Value | Description |
| --- | ---| ---|
| `assert` | OFF | Enable assertions.
| `reporting` | OFF | Build the reporting mode feature. |
| `coverage` | OFF | Prepare the coverage report. |
| `tests` | ON | Build tests. |
| `unity` | ON | Configure a unity build. |
| `san` | N/A | Enable a sanitizer with Clang. Choices are `thread` and `address`. |
[Unity builds][5] may be faster for the first build
(at the cost of much more memory) since they concatenate sources into fewer
translation units. Non-unity builds may be faster for incremental builds,
and can be helpful for detecting `#include` omissions.
## Troubleshooting
### Conan
After any updates or changes to dependencies, you may need to do the following:
1. Remove your build directory.
2. Remove the Conan cache:
```
rm -rf ~/.conan/data
```
4. Re-run [conan install](#build-and-test).
### no std::result_of
If your compiler version is recent enough to have removed `std::result_of` as
part of C++20, e.g. Apple Clang 15.0, then you might need to add a preprocessor
definition to your build.
```
conan profile update 'options.boost:extra_b2_flags="define=BOOST_ASIO_HAS_STD_INVOKE_RESULT"' default
conan profile update 'env.CFLAGS="-DBOOST_ASIO_HAS_STD_INVOKE_RESULT"' default
conan profile update 'env.CXXFLAGS="-DBOOST_ASIO_HAS_STD_INVOKE_RESULT"' default
conan profile update 'conf.tools.build:cflags+=["-DBOOST_ASIO_HAS_STD_INVOKE_RESULT"]' default
conan profile update 'conf.tools.build:cxxflags+=["-DBOOST_ASIO_HAS_STD_INVOKE_RESULT"]' default
```
### call to 'async_teardown' is ambiguous
If you are compiling with an early version of Clang 16, then you might hit
a [regression][6] when compiling C++20 that manifests as an [error in a Boost
header][7]. You can workaround it by adding this preprocessor definition:
```
conan profile update 'env.CXXFLAGS="-DBOOST_ASIO_DISABLE_CONCEPTS"' default
conan profile update 'conf.tools.build:cxxflags+=["-DBOOST_ASIO_DISABLE_CONCEPTS"]' default
```
### recompile with -fPIC
If you get a linker error suggesting that you recompile Boost with
position-independent code, such as:
```
/usr/bin/ld.gold: error: /home/username/.conan/data/boost/1.77.0/_/_/package/.../lib/libboost_container.a(alloc_lib.o):
requires unsupported dynamic reloc 11; recompile with -fPIC
```
Conan most likely downloaded a bad binary distribution of the dependency.
This seems to be a [bug][1] in Conan just for Boost 1.77.0 compiled with GCC
for Linux. The solution is to build the dependency locally by passing
`--build boost` when calling `conan install`.
```
conan install --build boost ...
```
## Add a Dependency
If you want to experiment with a new package, follow these steps:
1. Search for the package on [Conan Center](https://conan.io/center/).
2. Modify [`conanfile.py`](./conanfile.py):
- Add a version of the package to the `requires` property.
- Change any default options for the package by adding them to the
`default_options` property (with syntax `'$package:$option': $value`).
3. Modify [`CMakeLists.txt`](./CMakeLists.txt):
- Add a call to `find_package($package REQUIRED)`.
- Link a library from the package to the target `ripple_libs`
(search for the existing call to `target_link_libraries(ripple_libs INTERFACE ...)`).
4. Start coding! Don't forget to include whatever headers you need from the package.
[1]: https://github.com/conan-io/conan-center-index/issues/13168
[2]: https://en.cppreference.com/w/cpp/compiler_support/20
[3]: https://docs.conan.io/en/latest/getting_started.html
[5]: https://en.wikipedia.org/wiki/Unity_build
[6]: https://github.com/boostorg/beast/issues/2648
[7]: https://github.com/boostorg/beast/issues/2661
[gcovr]: https://gcovr.com/en/stable/getting-started.html
[python-pip]: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/
[build_type]: https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
[profile]: https://docs.conan.io/en/latest/reference/profiles.html

View File

@@ -1,38 +0,0 @@
# Maintainer: Roberto Catini <roberto.catini@gmail.com>
pkgname=rippled
pkgrel=1
pkgver=0
pkgdesc="Ripple peer-to-peer network daemon"
arch=('i686' 'x86_64')
url="https://github.com/ripple/rippled"
license=('custom:ISC')
depends=('protobuf' 'openssl' 'boost-libs')
makedepends=('git' 'scons' 'boost')
backup=("etc/$pkgname/rippled.cfg")
source=("git://github.com/ripple/rippled.git#branch=master")
sha512sums=('SKIP')
pkgver() {
cd "$srcdir/$pkgname"
git describe --long --tags | sed -r 's/([^-]*-g)/r\1/;s/-/./g'
}
build() {
cd "$srcdir/$pkgname"
scons
}
check() {
cd "$srcdir/$pkgname"
build/rippled --unittest
}
package() {
cd "$srcdir/$pkgname"
install -D -m644 LICENSE "$pkgdir/usr/share/licenses/$pkgname/LICENSE"
install -D build/rippled "$pkgdir/usr/bin/rippled"
install -D -m644 doc/rippled-example.cfg "$pkgdir/etc/$pkgname/rippled.cfg"
mkdir -p "$pkgdir/var/lib/$pkgname/db"
mkdir -p "$pkgdir/var/log/$pkgname"
}

View File

@@ -1,706 +1,207 @@
# This is a set of common functions and settings for rippled
# and derived products.
############################################################
cmake_minimum_required(VERSION 3.1.0)
if("${CMAKE_SOURCE_DIR}" STREQUAL "${CMAKE_BINARY_DIR}")
message(WARNING "Builds are strongly discouraged in "
"${CMAKE_SOURCE_DIR}.")
endif()
macro(parse_target)
if (NOT target AND NOT CMAKE_BUILD_TYPE)
if (APPLE)
set(target clang.debug)
elseif(WIN32)
set(target msvc)
macro(group_sources_in source_dir curdir)
file(GLOB children RELATIVE ${source_dir}/${curdir}
${source_dir}/${curdir}/*)
foreach (child ${children})
if (IS_DIRECTORY ${source_dir}/${curdir}/${child})
group_sources_in(${source_dir} ${curdir}/${child})
else()
set(target gcc.debug)
string(REPLACE "/" "\\" groupname ${curdir})
source_group(${groupname} FILES
${source_dir}/${curdir}/${child})
endif()
endif()
if (target)
# Parse the target
set(remaining ${target})
while (remaining)
# get the component up to the next dot or end
string(REGEX REPLACE "^\\.?([^\\.]+).*$" "\\1" cur_component ${remaining})
string(REGEX REPLACE "^\\.?[^\\.]+(.*$)" "\\1" remaining ${remaining})
if (${cur_component} STREQUAL gcc)
if (DEFINED ENV{GNU_CC})
set(CMAKE_C_COMPILER $ENV{GNU_CC})
elseif ($ENV{CC} MATCHES .*gcc.*)
set(CMAKE_C_COMPILER $ENV{CC})
else()
find_program(CMAKE_C_COMPILER gcc)
endif()
if (DEFINED ENV{GNU_CXX})
set(CMAKE_CXX_COMPILER $ENV{GNU_CXX})
elseif ($ENV{CXX} MATCHES .*g\\+\\+.*)
set(CMAKE_CXX_COMPILER $ENV{CXX})
else()
find_program(CMAKE_CXX_COMPILER g++)
endif()
endif()
if (${cur_component} STREQUAL clang)
if (DEFINED ENV{CLANG_CC})
set(CMAKE_C_COMPILER $ENV{CLANG_CC})
elseif ($ENV{CC} MATCHES .*clang.*)
set(CMAKE_C_COMPILER $ENV{CC})
else()
find_program(CMAKE_C_COMPILER clang)
endif()
if (DEFINED ENV{CLANG_CXX})
set(CMAKE_CXX_COMPILER $ENV{CLANG_CXX})
elseif ($ENV{CXX} MATCHES .*clang.*)
set(CMAKE_CXX_COMPILER $ENV{CXX})
else()
find_program(CMAKE_CXX_COMPILER clang++)
endif()
endif()
if (${cur_component} STREQUAL msvc)
# TBD
endif()
if (${cur_component} STREQUAL unity)
set(unity true)
set(nonunity false)
endif()
if (${cur_component} STREQUAL nounity)
set(unity false)
set(nonunity true)
endif()
if (${cur_component} STREQUAL debug)
set(release false)
endif()
if (${cur_component} STREQUAL release)
set(release true)
endif()
if (${cur_component} STREQUAL coverage)
set(coverage true)
set(debug true)
endif()
if (${cur_component} STREQUAL profile)
set(profile true)
endif()
if (${cur_component} STREQUAL ci)
# Workarounds that make various CI builds work, but that
# we don't want in the general case.
set(ci true)
set(openssl_min 1.0.1)
endif()
endwhile()
# Promote these values to the CACHE, then unset the locals
# to prevent shadowing.
set(CMAKE_C_COMPILER ${CMAKE_C_COMPILER} CACHE FILEPATH
"Path to a program" FORCE)
unset(CMAKE_C_COMPILER)
set(CMAKE_CXX_COMPILER ${CMAKE_CXX_COMPILER} CACHE FILEPATH
"Path to a program" FORCE)
unset(CMAKE_CXX_COMPILER)
if (release)
set(CMAKE_BUILD_TYPE Release)
else()
set(CMAKE_BUILD_TYPE Debug)
endif()
# ensure that the unity flags are set and exclusive
if (NOT DEFINED unity OR unity)
# Default to unity builds
set(unity true)
set(nonunity false)
else()
set(unity false)
set(nonunity true)
endif()
if (NOT unity)
set(CMAKE_BUILD_TYPE ${CMAKE_BUILD_TYPE}Classic)
endif()
# Promote this value to the CACHE, then unset the local
# to prevent shadowing.
set(CMAKE_BUILD_TYPE ${CMAKE_BUILD_TYPE} CACHE INTERNAL
"Choose the type of build, options are in CMAKE_CONFIGURATION_TYPES"
FORCE)
unset(CMAKE_BUILD_TYPE)
endif()
endmacro()
############################################################
macro(setup_build_cache)
set(san "" CACHE STRING "On gcc & clang, add sanitizer
instrumentation")
set_property(CACHE san PROPERTY STRINGS ";address;thread")
set(assert false CACHE BOOL "Enables asserts, even in release builds")
set(static false CACHE BOOL
"On linux, link protobuf, openssl, libc++, and boost statically")
if (static AND (WIN32 OR APPLE))
message(FATAL_ERROR "Static linking is only supported on linux.")
endif()
if (${CMAKE_GENERATOR} STREQUAL "Unix Makefiles" AND NOT CMAKE_BUILD_TYPE)
set(CMAKE_BUILD_TYPE Debug)
endif()
# Can't exclude files from configurations, so can't support both
# unity and nonunity configurations at the same time
if (NOT DEFINED unity OR unity)
set(CMAKE_CONFIGURATION_TYPES
Debug
Release)
else()
set(CMAKE_CONFIGURATION_TYPES
DebugClassic
ReleaseClassic)
endif()
# Promote this value to the CACHE, then unset the local
# to prevent shadowing.
set(CMAKE_CONFIGURATION_TYPES
${CMAKE_CONFIGURATION_TYPES} CACHE STRING "" FORCE)
unset(CMAKE_CONFIGURATION_TYPES)
endmacro()
############################################################
function(prepend var prefix)
set(listVar "")
foreach(f ${ARGN})
list(APPEND listVar "${prefix}${f}")
endforeach(f)
set(${var} "${listVar}" PARENT_SCOPE)
endfunction()
macro(append_flags name)
foreach (arg ${ARGN})
set(${name} "${${name}} ${arg}")
endforeach()
endmacro()
macro(group_sources curdir)
file(GLOB children RELATIVE ${PROJECT_SOURCE_DIR}/${curdir}
${PROJECT_SOURCE_DIR}/${curdir}/*)
foreach (child ${children})
if (IS_DIRECTORY ${PROJECT_SOURCE_DIR}/${curdir}/${child})
group_sources(${curdir}/${child})
else()
string(REPLACE "/" "\\" groupname ${curdir})
source_group(${groupname} FILES
${PROJECT_SOURCE_DIR}/${curdir}/${child})
group_sources_in(${PROJECT_SOURCE_DIR} ${curdir})
endmacro()
macro (exclude_from_default target_)
set_target_properties (${target_} PROPERTIES EXCLUDE_FROM_ALL ON)
set_target_properties (${target_} PROPERTIES EXCLUDE_FROM_DEFAULT_BUILD ON)
endmacro ()
macro (exclude_if_included target_)
get_directory_property(has_parent PARENT_DIRECTORY)
if (has_parent)
exclude_from_default (${target_})
endif ()
endmacro ()
function (print_ep_logs _target)
ExternalProject_Get_Property (${_target} STAMP_DIR)
add_custom_command(TARGET ${_target} POST_BUILD
COMMENT "${_target} BUILD OUTPUT"
COMMAND ${CMAKE_COMMAND}
-DIN_FILE=${STAMP_DIR}/${_target}-build-out.log
-P ${CMAKE_CURRENT_SOURCE_DIR}/Builds/CMake/echo_file.cmake
COMMAND ${CMAKE_COMMAND}
-DIN_FILE=${STAMP_DIR}/${_target}-build-err.log
-P ${CMAKE_CURRENT_SOURCE_DIR}/Builds/CMake/echo_file.cmake)
endfunction ()
#[=========================================================[
This is a function override for one function in the
standard ExternalProject module. We want to change
the generated build script slightly to include printing
the build logs in the case of failure. Those modifications
have been made here. This function override could break
in the future if the ExternalProject module changes internal
function names or changes the way it generates the build
scripts.
See:
https://gitlab.kitware.com/cmake/cmake/blob/df1ddeec128d68cc636f2dde6c2acd87af5658b6/Modules/ExternalProject.cmake#L1855-1952
#]=========================================================]
function(_ep_write_log_script name step cmd_var)
ExternalProject_Get_Property(${name} stamp_dir)
set(command "${${cmd_var}}")
set(make "")
set(code_cygpath_make "")
if(command MATCHES "^\\$\\(MAKE\\)")
# GNU make recognizes the string "$(MAKE)" as recursive make, so
# ensure that it appears directly in the makefile.
string(REGEX REPLACE "^\\$\\(MAKE\\)" "\${make}" command "${command}")
set(make "-Dmake=$(MAKE)")
if(WIN32 AND NOT CYGWIN)
set(code_cygpath_make "
if(\${make} MATCHES \"^/\")
execute_process(
COMMAND cygpath -w \${make}
OUTPUT_VARIABLE cygpath_make
ERROR_VARIABLE cygpath_make
RESULT_VARIABLE cygpath_error
OUTPUT_STRIP_TRAILING_WHITESPACE
)
if(NOT cygpath_error)
set(make \${cygpath_make})
endif()
endif()
")
endif()
endif()
set(config "")
if("${CMAKE_CFG_INTDIR}" MATCHES "^\\$")
string(REPLACE "${CMAKE_CFG_INTDIR}" "\${config}" command "${command}")
set(config "-Dconfig=${CMAKE_CFG_INTDIR}")
endif()
# Wrap multiple 'COMMAND' lines up into a second-level wrapper
# script so all output can be sent to one log file.
if(command MATCHES "(^|;)COMMAND;")
set(code_execute_process "
${code_cygpath_make}
execute_process(COMMAND \${command} RESULT_VARIABLE result)
if(result)
set(msg \"Command failed (\${result}):\\n\")
foreach(arg IN LISTS command)
set(msg \"\${msg} '\${arg}'\")
endforeach()
endmacro()
message(FATAL_ERROR \"\${msg}\")
endif()
")
set(code "")
set(cmd "")
set(sep "")
foreach(arg IN LISTS command)
if("x${arg}" STREQUAL "xCOMMAND")
if(NOT "x${cmd}" STREQUAL "x")
string(APPEND code "set(command \"${cmd}\")${code_execute_process}")
endif()
set(cmd "")
set(sep "")
else()
string(APPEND cmd "${sep}${arg}")
set(sep ";")
endif()
endforeach()
string(APPEND code "set(command \"${cmd}\")${code_execute_process}")
file(GENERATE OUTPUT "${stamp_dir}/${name}-${step}-$<CONFIG>-impl.cmake" CONTENT "${code}")
set(command ${CMAKE_COMMAND} "-Dmake=\${make}" "-Dconfig=\${config}" -P ${stamp_dir}/${name}-${step}-$<CONFIG>-impl.cmake)
endif()
macro(add_with_props src_var files)
list(APPEND ${src_var} ${files})
foreach (arg ${ARGN})
set(props "${props} ${arg}")
# Wrap the command in a script to log output to files.
set(script ${stamp_dir}/${name}-${step}-$<CONFIG>.cmake)
set(logbase ${stamp_dir}/${name}-${step})
set(code "
${code_cygpath_make}
function (_echo_file _fil)
file (READ \${_fil} _cont)
execute_process (COMMAND \${CMAKE_COMMAND} -E echo \"\${_cont}\")
endfunction ()
set(command \"${command}\")
execute_process(
COMMAND \${command}
RESULT_VARIABLE result
OUTPUT_FILE \"${logbase}-out.log\"
ERROR_FILE \"${logbase}-err.log\"
)
if(result)
set(msg \"Command failed: \${result}\\n\")
foreach(arg IN LISTS command)
set(msg \"\${msg} '\${arg}'\")
endforeach()
set_source_files_properties(
${files}
PROPERTIES COMPILE_FLAGS
${props})
endmacro()
execute_process (COMMAND \${CMAKE_COMMAND} -E echo \"Build output for ${logbase} : \")
_echo_file (\"${logbase}-out.log\")
_echo_file (\"${logbase}-err.log\")
set(msg \"\${msg}\\nSee above\\n\")
message(FATAL_ERROR \"\${msg}\")
else()
set(msg \"${name} ${step} command succeeded. See also ${logbase}-*.log\")
message(STATUS \"\${msg}\")
endif()
")
file(GENERATE OUTPUT "${script}" CONTENT "${code}")
set(command ${CMAKE_COMMAND} ${make} ${config} -P ${script})
set(${cmd_var} "${command}" PARENT_SCOPE)
endfunction()
############################################################
find_package(Git)
macro(determine_build_type)
if ("${CMAKE_CXX_COMPILER_ID}" MATCHES ".*Clang") # both Clang and AppleClang
set(is_clang true)
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
set(is_gcc true)
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC")
set(is_msvc true)
endif()
if (${CMAKE_GENERATOR} STREQUAL "Xcode")
set(is_xcode true)
else()
set(is_xcode false)
endif()
if (NOT is_gcc AND NOT is_clang AND NOT is_msvc)
message("Current compiler is ${CMAKE_CXX_COMPILER_ID}")
message(FATAL_ERROR "Missing compiler. Must be GNU, Clang, or MSVC")
endif()
endmacro()
############################################################
macro(check_gcc4_abi)
# Check if should use gcc4's ABI
set(gcc4_abi false)
if ($ENV{RIPPLED_OLD_GCC_ABI})
set(gcc4_abi true)
endif()
if (is_gcc AND NOT gcc4_abi)
if (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER 5)
execute_process(COMMAND lsb_release -si OUTPUT_VARIABLE lsb)
string(STRIP "${lsb}" lsb)
if ("${lsb}" STREQUAL "Ubuntu")
execute_process(COMMAND lsb_release -sr OUTPUT_VARIABLE lsb)
string(STRIP ${lsb} lsb)
if (${lsb} VERSION_LESS 15.1)
set(gcc4_abi true)
endif()
endif()
endif()
endif()
if (gcc4_abi)
add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)
endif()
endmacro()
############################################################
macro(special_build_flags)
if (coverage)
add_compile_options(-fprofile-arcs -ftest-coverage)
append_flags(CMAKE_EXE_LINKER_FLAGS -fprofile-arcs -ftest-coverage)
endif()
if (profile)
add_compile_options(-p -pg)
append_flags(CMAKE_EXE_LINKER_FLAGS -p -pg)
endif()
endmacro()
############################################################
# Params: Boost components to search for.
macro(use_boost)
if ((NOT DEFINED BOOST_ROOT) AND (DEFINED ENV{BOOST_ROOT}))
set(BOOST_ROOT $ENV{BOOST_ROOT})
endif()
if(WIN32 OR CYGWIN)
# Workaround for MSVC having two boost versions - x86 and x64 on same PC in stage folders
if(DEFINED BOOST_ROOT)
if(CMAKE_SIZEOF_VOID_P EQUAL 8 AND IS_DIRECTORY ${BOOST_ROOT}/stage64/lib)
set(Boost_LIBRARY_DIR ${BOOST_ROOT}/stage64/lib)
else()
set(Boost_LIBRARY_DIR ${BOOST_ROOT}/stage/lib)
endif()
endif()
endif()
if (is_clang AND DEFINED ENV{CLANG_BOOST_ROOT})
set(BOOST_ROOT $ENV{CLANG_BOOST_ROOT})
endif()
set(Boost_USE_STATIC_LIBS on)
set(Boost_USE_MULTITHREADED on)
set(Boost_USE_STATIC_RUNTIME off)
find_package(Boost COMPONENTS
${ARGN})
if (Boost_FOUND OR
((CYGWIN OR WIN32) AND Boost_INCLUDE_DIRS AND Boost_LIBRARY_DIRS))
if(NOT Boost_FOUND)
message(WARNING "Boost directory found, but not all components. May not be able to build.")
endif()
include_directories(SYSTEM ${Boost_INCLUDE_DIRS})
link_directories(${Boost_LIBRARY_DIRS})
else()
message(FATAL_ERROR "Boost not found")
endif()
endmacro()
macro(use_pthread)
if (NOT WIN32)
set(THREADS_PREFER_PTHREAD_FLAG ON)
find_package(Threads)
add_compile_options(${CMAKE_THREAD_LIBS_INIT})
endif()
endmacro()
macro(use_openssl openssl_min)
if (APPLE AND NOT DEFINED ENV{OPENSSL_ROOT_DIR})
find_program(HOMEBREW brew)
if (NOT HOMEBREW STREQUAL "HOMEBREW-NOTFOUND")
execute_process(COMMAND brew --prefix openssl
OUTPUT_VARIABLE OPENSSL_ROOT_DIR
OUTPUT_STRIP_TRAILING_WHITESPACE)
endif()
endif()
if (WIN32)
if (DEFINED ENV{OPENSSL_ROOT})
include_directories($ENV{OPENSSL_ROOT}/include)
link_directories($ENV{OPENSSL_ROOT}/lib)
endif()
else()
if (static)
set(tmp CMAKE_FIND_LIBRARY_SUFFIXES)
set(CMAKE_FIND_LIBRARY_SUFFIXES .a)
endif()
find_package(OpenSSL)
if (static)
set(CMAKE_FIND_LIBRARY_SUFFIXES tmp)
endif()
if (OPENSSL_FOUND)
include_directories(${OPENSSL_INCLUDE_DIR})
else()
message(FATAL_ERROR "OpenSSL not found")
endif()
if (UNIX AND NOT APPLE AND ${OPENSSL_VERSION} VERSION_LESS ${openssl_min})
message(FATAL_ERROR
"Your openssl is Version: ${OPENSSL_VERSION}, ${openssl_min} or better is required.")
endif()
endif()
endmacro()
macro(use_protobuf)
if (WIN32)
if (DEFINED ENV{PROTOBUF_ROOT})
include_directories($ENV{PROTOBUF_ROOT}/src)
link_directories($ENV{PROTOBUF_ROOT}/src/.libs)
endif()
# Modified from FindProtobuf.cmake
FUNCTION(PROTOBUF_GENERATE_CPP SRCS HDRS PROTOFILES)
# argument parsing
IF(NOT PROTOFILES)
MESSAGE(SEND_ERROR "Error: PROTOBUF_GENERATE_CPP() called without any proto files")
RETURN()
ENDIF()
SET(OUTPATH ${CMAKE_CURRENT_BINARY_DIR})
SET(PROTOROOT ${CMAKE_CURRENT_SOURCE_DIR})
# the real logic
SET(${SRCS})
SET(${HDRS})
FOREACH(PROTOFILE ${PROTOFILES})
# ensure that the file ends with .proto
STRING(REGEX MATCH "\\.proto$$" PROTOEND ${PROTOFILE})
IF(NOT PROTOEND)
MESSAGE(SEND_ERROR "Proto file '${PROTOFILE}' does not end with .proto")
ENDIF()
GET_FILENAME_COMPONENT(PROTO_PATH ${PROTOFILE} PATH)
GET_FILENAME_COMPONENT(ABS_FILE ${PROTOFILE} ABSOLUTE)
GET_FILENAME_COMPONENT(FILE_WE ${PROTOFILE} NAME_WE)
STRING(REGEX MATCH "^${PROTOROOT}" IN_ROOT_PATH ${PROTOFILE})
STRING(REGEX MATCH "^${PROTOROOT}" IN_ROOT_ABS_FILE ${ABS_FILE})
IF(IN_ROOT_PATH)
SET(MATCH_PATH ${PROTOFILE})
ELSEIF(IN_ROOT_ABS_FILE)
SET(MATCH_PATH ${ABS_FILE})
ELSE()
MESSAGE(SEND_ERROR "Proto file '${PROTOFILE}' is not in protoroot '${PROTOROOT}'")
ENDIF()
# build the result file name
STRING(REGEX REPLACE "^${PROTOROOT}(/?)" "" ROOT_CLEANED_FILE ${MATCH_PATH})
STRING(REGEX REPLACE "\\.proto$$" "" EXT_CLEANED_FILE ${ROOT_CLEANED_FILE})
SET(CPP_FILE "${OUTPATH}/${EXT_CLEANED_FILE}.pb.cc")
SET(H_FILE "${OUTPATH}/${EXT_CLEANED_FILE}.pb.h")
LIST(APPEND ${SRCS} "${CPP_FILE}")
LIST(APPEND ${HDRS} "${H_FILE}")
ADD_CUSTOM_COMMAND(
OUTPUT "${CPP_FILE}" "${H_FILE}"
COMMAND ${CMAKE_COMMAND} -E make_directory ${OUTPATH}
COMMAND ${PROTOBUF_PROTOC_EXECUTABLE}
ARGS "--cpp_out=${OUTPATH}" --proto_path "${PROTOROOT}" "${MATCH_PATH}"
DEPENDS ${ABS_FILE}
COMMENT "Running C++ protocol buffer compiler on ${MATCH_PATH} with root ${PROTOROOT}, generating: ${CPP_FILE}"
VERBATIM)
ENDFOREACH()
SET_SOURCE_FILES_PROPERTIES(${${SRCS}} ${${HDRS}} PROPERTIES GENERATED TRUE)
SET(${SRCS} ${${SRCS}} PARENT_SCOPE)
SET(${HDRS} ${${HDRS}} PARENT_SCOPE)
ENDFUNCTION()
set(PROTOBUF_PROTOC_EXECUTABLE Protoc) # must be on path
else()
if (static)
set(tmp CMAKE_FIND_LIBRARY_SUFFIXES)
set(CMAKE_FIND_LIBRARY_SUFFIXES .a)
endif()
find_package(Protobuf REQUIRED)
if (static)
set(CMAKE_FIND_LIBRARY_SUFFIXES tmp)
endif()
if (is_clang AND DEFINED ENV{CLANG_PROTOBUF_ROOT})
link_directories($ENV{CLANG_PROTOBUF_ROOT}/src/.libs)
include_directories($ENV{CLANG_PROTOBUF_ROOT}/src)
else()
include_directories(${PROTOBUF_INCLUDE_DIRS})
endif()
endif()
include_directories(${CMAKE_CURRENT_BINARY_DIR})
file(GLOB ripple_proto src/ripple/proto/*.proto)
PROTOBUF_GENERATE_CPP(PROTO_SRCS PROTO_HDRS ${ripple_proto})
if (WIN32)
include_directories(src/protobuf/src
src/protobuf/vsprojects
${CMAKE_CURRENT_BINARY_DIR}/src/ripple/proto)
endif()
endmacro()
############################################################
macro(setup_build_boilerplate)
if (NOT WIN32 AND san)
add_compile_options(-fsanitize=${san} -fno-omit-frame-pointer)
append_flags(CMAKE_EXE_LINKER_FLAGS
-fsanitize=${san})
string(TOLOWER ${san} ci_san)
if (${ci_san} STREQUAL address)
set(SANITIZER_LIBRARIES asan)
add_definitions(-DSANITIZER=ASAN)
endif()
if (${ci_san} STREQUAL thread)
set(SANITIZER_LIBRARIES tsan)
add_definitions(-DSANITIZER=TSAN)
endif()
endif()
############################################################
add_definitions(
-DOPENSSL_NO_SSL2
-DDEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER
-DHAVE_USLEEP=1
-DSOCI_CXX_C11=1
-D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS
-DBOOST_NO_AUTO_PTR
)
if (is_gcc)
add_compile_options(-Wno-unused-but-set-variable -Wno-deprecated)
# use gold linker if available
execute_process(
COMMAND ${CMAKE_CXX_COMPILER} -fuse-ld=gold -Wl,--version
ERROR_QUIET OUTPUT_VARIABLE LD_VERSION)
if ("${LD_VERSION}" MATCHES "GNU gold")
append_flags(CMAKE_EXE_LINKER_FLAGS -fuse-ld=gold)
# function that calls git log to get current hash
function (git_hash hash_val)
# note: optional second extra string argument not in signature
if (NOT GIT_FOUND)
return ()
endif ()
set (_hash "")
set (_format "%H")
if (ARGC GREATER_EQUAL 2)
string (TOLOWER ${ARGV1} _short)
if (_short STREQUAL "short")
set (_format "%h")
endif ()
unset(LD_VERSION)
endif()
endif ()
execute_process (COMMAND ${GIT_EXECUTABLE} "log" "--pretty=${_format}" "-n1"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
RESULT_VARIABLE _git_exit_code
OUTPUT_VARIABLE _temp_hash
OUTPUT_STRIP_TRAILING_WHITESPACE
ERROR_QUIET)
if (_git_exit_code EQUAL 0)
set (_hash ${_temp_hash})
endif ()
set (${hash_val} "${_hash}" PARENT_SCOPE)
endfunction ()
# Generator expressions are not supported in add_definitions, use set_property instead
set_property(
DIRECTORY
APPEND
PROPERTY COMPILE_DEFINITIONS
$<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:DEBUG _DEBUG>)
function (git_branch branch_val)
if (NOT GIT_FOUND)
return ()
endif ()
set (_branch "")
execute_process (COMMAND ${GIT_EXECUTABLE} "rev-parse" "--abbrev-ref" "HEAD"
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
RESULT_VARIABLE _git_exit_code
OUTPUT_VARIABLE _temp_branch
OUTPUT_STRIP_TRAILING_WHITESPACE
ERROR_QUIET)
if (_git_exit_code EQUAL 0)
set (_branch ${_temp_branch})
endif ()
set (${branch_val} "${_branch}" PARENT_SCOPE)
endfunction ()
if (NOT assert)
set_property(
DIRECTORY
APPEND
PROPERTY COMPILE_DEFINITIONS
$<$<OR:$<BOOL:${profile}>,$<CONFIG:Release>,$<CONFIG:ReleaseClassic>>:NDEBUG>)
else()
# CMAKE_CXX_FLAGS_RELEASE is created by CMake for most / all generators
# with defaults including /DNDEBUG or -DNDEBUG, and that value is stored
# in the cache. Override that locally so that the cache value will be
# avaiable if "assert" is ever changed.
STRING(REGEX REPLACE "[-/]DNDEBUG" "" CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE}")
STRING(REGEX REPLACE "[-/]DNDEBUG" "" CMAKE_CXX_FLAGS_RELEASECLASSIC "${CMAKE_CXX_FLAGS_RELEASECLASSIC}")
STRING(REGEX REPLACE "[-/]DNDEBUG" "" CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE}")
STRING(REGEX REPLACE "[-/]DNDEBUG" "" CMAKE_C_FLAGS_RELEASECLASSIC "${CMAKE_C_FLAGS_RELEASECLASSIC}")
endif()
if (NOT WIN32)
add_definitions(-D_FILE_OFFSET_BITS=64)
append_flags(CMAKE_CXX_FLAGS -frtti -std=c++14 -Wno-invalid-offsetof
-DBOOST_COROUTINE_NO_DEPRECATION_WARNING -DBOOST_COROUTINES_NO_DEPRECATION_WARNING)
add_compile_options(-Wall -Wno-sign-compare -Wno-char-subscripts -Wno-format
-Wno-unused-local-typedefs -g)
# There seems to be an issue using generator experssions with multiple values,
# split the expression
add_compile_options($<$<OR:$<CONFIG:Release>,$<CONFIG:ReleaseClassic>>:-O3>)
add_compile_options($<$<OR:$<CONFIG:Release>,$<CONFIG:ReleaseClassic>>:-fno-strict-aliasing>)
append_flags(CMAKE_EXE_LINKER_FLAGS -rdynamic -g)
if (is_clang)
add_compile_options(
-Wno-redeclared-class-member -Wno-mismatched-tags -Wno-deprecated-register)
add_definitions(-DBOOST_ASIO_HAS_STD_ARRAY)
# use ldd linker if available
execute_process(
COMMAND ${CMAKE_CXX_COMPILER} -fuse-ld=lld -Wl,--version
ERROR_QUIET OUTPUT_VARIABLE LD_VERSION)
if ("${LD_VERSION}" MATCHES "LLD")
append_flags(CMAKE_EXE_LINKER_FLAGS -fuse-ld=lld)
endif ()
unset(LD_VERSION)
endif()
if (APPLE)
add_definitions(-DBEAST_COMPILE_OBJECTIVE_CPP=1)
add_compile_options(
-Wno-deprecated -Wno-deprecated-declarations -Wno-unused-variable -Wno-unused-function)
endif()
if (is_gcc)
add_compile_options(-Wno-unused-but-set-variable -Wno-unused-local-typedefs)
add_compile_options($<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:-O0>)
endif (is_gcc)
else(NOT WIN32)
add_compile_options(
/bigobj # Increase object file max size
/EHa # ExceptionHandling all
/fp:precise # Floating point behavior
/Gd # __cdecl calling convention
/Gm- # Minimal rebuild: disabled
/GR # Enable RTTI
/Gy- # Function level linking: disabled
/FS
/MP # Multiprocessor compilation
/openmp- # pragma omp: disabled
/Zc:forScope # Language extension: for scope
/Zi # Generate complete debug info
/errorReport:none # No error reporting to Internet
/nologo # Suppress login banner
/W3 # Warning level 3
/WX- # Disable warnings as errors
/wd"4018"
/wd"4244"
/wd"4267"
/wd"4800" # Disable C4800(int to bool performance)
/wd"4503" # Decorated name length exceeded, name was truncated
)
add_definitions(
-D_WIN32_WINNT=0x6000
-D_SCL_SECURE_NO_WARNINGS
-D_CRT_SECURE_NO_WARNINGS
-DWIN32_CONSOLE
-DNOMINMAX
-DBOOST_COROUTINE_NO_DEPRECATION_WARNING
-DBOOST_COROUTINES_NO_DEPRECATION_WARNING)
append_flags(CMAKE_EXE_LINKER_FLAGS
/DEBUG
/DYNAMICBASE
/ERRORREPORT:NONE
/MACHINE:X64
/MANIFEST
/nologo
/NXCOMPAT
/SUBSYSTEM:CONSOLE
/TLBID:1)
# There seems to be an issue using generator experssions with multiple values,
# split the expression
# /GS Buffers security check: enable
add_compile_options($<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:/GS>)
# /MTd Language: Multi-threaded Debug CRT
add_compile_options($<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:/MTd>)
# /Od Optimization: Disabled
add_compile_options($<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:/Od>)
# /RTC1 Run-time error checks:
add_compile_options($<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:/RTC1>)
# Generator expressions are not supported in add_definitions, use set_property instead
set_property(
DIRECTORY
APPEND
PROPERTY COMPILE_DEFINITIONS
$<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:_CRTDBG_MAP_ALLOC>)
# /MT Language: Multi-threaded CRT
add_compile_options($<$<OR:$<CONFIG:Release>,$<CONFIG:ReleaseClassic>>:/MT>)
add_compile_options($<$<OR:$<CONFIG:Release>,$<CONFIG:ReleaseClassic>>:/Ox>)
# /Ox Optimization: Full
endif (NOT WIN32)
if (static)
append_flags(CMAKE_EXE_LINKER_FLAGS -static-libstdc++)
# set_target_properties(ripple-libpp PROPERTIES LINK_SEARCH_START_STATIC 1)
# set_target_properties(ripple-libpp PROPERTIES LINK_SEARCH_END_STATIC 1)
endif()
endmacro()
############################################################
macro(create_build_folder cur_project)
if (NOT WIN32)
ADD_CUSTOM_TARGET(build_folder ALL
COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_CURRENT_BINARY_DIR}
COMMENT "Creating build output folder")
add_dependencies(${cur_project} build_folder)
endif()
endmacro()
macro(set_startup_project cur_project)
if (WIN32 AND NOT ci)
if (CMAKE_VERSION VERSION_LESS 3.6)
message(WARNING
"Setting the VS startup project requires cmake 3.6 or later. Please upgrade.")
endif()
set_property(DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY
VS_STARTUP_PROJECT ${cur_project})
endif()
endmacro()
macro(link_common_libraries cur_project)
if (NOT MSVC)
target_link_libraries(${cur_project} ${Boost_LIBRARIES})
target_link_libraries(${cur_project} dl)
target_link_libraries(${cur_project} Threads::Threads)
if (APPLE)
find_library(app_kit AppKit)
find_library(foundation Foundation)
target_link_libraries(${cur_project}
crypto ssl ${app_kit} ${foundation})
else()
target_link_libraries(${cur_project} rt)
endif()
else(NOT MSVC)
target_link_libraries(${cur_project}
$<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:VC/static/ssleay32MTd>
$<$<OR:$<CONFIG:Debug>,$<CONFIG:DebugClassic>>:VC/static/libeay32MTd>)
target_link_libraries(${cur_project}
$<$<OR:$<CONFIG:Release>,$<CONFIG:ReleaseClassic>>:VC/static/ssleay32MT>
$<$<OR:$<CONFIG:Release>,$<CONFIG:ReleaseClassic>>:VC/static/libeay32MT>)
target_link_libraries(${cur_project}
legacy_stdio_definitions.lib Shlwapi kernel32 user32 gdi32 winspool comdlg32
advapi32 shell32 ole32 oleaut32 uuid odbc32 odbccp32)
endif (NOT MSVC)
endmacro()

View File

@@ -0,0 +1,60 @@
#[=========================================================[
SQLITE doesn't provide build files in the
standard source-only distribution. So we wrote
a simple cmake file and we copy it to the
external project folder so that we can use
this file to build the lib with ExternalProject
#]=========================================================]
add_library (sqlite3 STATIC sqlite3.c)
#[=========================================================[
When compiled with SQLITE_THREADSAFE=1, SQLite operates
in serialized mode. In this mode, SQLite can be safely
used by multiple threads with no restriction.
NOTE: This implies a global mutex!
When compiled with SQLITE_THREADSAFE=2, SQLite can be
used in a multithreaded program so long as no two
threads attempt to use the same database connection at
the same time.
NOTE: This is the preferred threading model, but not
currently enabled because we need to investigate our
use-model and concurrency requirements.
TODO: consider whether any other options should be
used: https://www.sqlite.org/compile.html
#]=========================================================]
target_compile_definitions (sqlite3
PRIVATE
SQLITE_THREADSAFE=1
HAVE_USLEEP=1)
target_compile_options (sqlite3
PRIVATE
$<$<BOOL:${MSVC}>:
-wd4100
-wd4127
-wd4232
-wd4244
-wd4701
-wd4706
-wd4996
>
$<$<NOT:$<BOOL:${MSVC}>>:-Wno-array-bounds>)
install (
TARGETS
sqlite3
LIBRARY DESTINATION lib
ARCHIVE DESTINATION lib
RUNTIME DESTINATION bin
INCLUDES DESTINATION include)
install (
FILES
sqlite3.h
sqlite3ext.h
DESTINATION include)

View File

@@ -0,0 +1,440 @@
# Copyright (c) 2012 - 2017, Lars Bilke
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without modification,
# are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this
# list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
# this list of conditions and the following disclaimer in the documentation
# and/or other materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors
# may be used to endorse or promote products derived from this software without
# specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# CHANGES:
#
# 2012-01-31, Lars Bilke
# - Enable Code Coverage
#
# 2013-09-17, Joakim Söderberg
# - Added support for Clang.
# - Some additional usage instructions.
#
# 2016-02-03, Lars Bilke
# - Refactored functions to use named parameters
#
# 2017-06-02, Lars Bilke
# - Merged with modified version from github.com/ufz/ogs
#
# 2019-05-06, Anatolii Kurotych
# - Remove unnecessary --coverage flag
#
# 2019-12-13, FeRD (Frank Dana)
# - Deprecate COVERAGE_LCOVR_EXCLUDES and COVERAGE_GCOVR_EXCLUDES lists in favor
# of tool-agnostic COVERAGE_EXCLUDES variable, or EXCLUDE setup arguments.
# - CMake 3.4+: All excludes can be specified relative to BASE_DIRECTORY
# - All setup functions: accept BASE_DIRECTORY, EXCLUDE list
# - Set lcov basedir with -b argument
# - Add automatic --demangle-cpp in lcovr, if 'c++filt' is available (can be
# overridden with NO_DEMANGLE option in setup_target_for_coverage_lcovr().)
# - Delete output dir, .info file on 'make clean'
# - Remove Python detection, since version mismatches will break gcovr
# - Minor cleanup (lowercase function names, update examples...)
#
# 2019-12-19, FeRD (Frank Dana)
# - Rename Lcov outputs, make filtered file canonical, fix cleanup for targets
#
# 2020-01-19, Bob Apthorpe
# - Added gfortran support
#
# 2020-02-17, FeRD (Frank Dana)
# - Make all add_custom_target()s VERBATIM to auto-escape wildcard characters
# in EXCLUDEs, and remove manual escaping from gcovr targets
#
# 2021-01-19, Robin Mueller
# - Add CODE_COVERAGE_VERBOSE option which will allow to print out commands which are run
# - Added the option for users to set the GCOVR_ADDITIONAL_ARGS variable to supply additional
# flags to the gcovr command
#
# 2020-05-04, Mihchael Davis
# - Add -fprofile-abs-path to make gcno files contain absolute paths
# - Fix BASE_DIRECTORY not working when defined
# - Change BYPRODUCT from folder to index.html to stop ninja from complaining about double defines
#
# 2021-05-10, Martin Stump
# - Check if the generator is multi-config before warning about non-Debug builds
#
# 2022-02-22, Marko Wehle
# - Change gcovr output from -o <filename> for --xml <filename> and --html <filename> output respectively.
# This will allow for Multiple Output Formats at the same time by making use of GCOVR_ADDITIONAL_ARGS, e.g. GCOVR_ADDITIONAL_ARGS "--txt".
#
# 2022-09-28, Sebastian Mueller
# - fix append_coverage_compiler_flags_to_target to correctly add flags
# - replace "-fprofile-arcs -ftest-coverage" with "--coverage" (equivalent)
#
# 2024-01-04, Bronek Kozicki
# - remove setup_target_for_coverage_lcov (slow) and setup_target_for_coverage_fastcov (no support for Clang)
# - fix Clang support by adding find_program( ... llvm-cov )
# - add Apple Clang support by adding execute_process( COMMAND xcrun -f llvm-cov ... )
# - add CODE_COVERAGE_GCOV_TOOL to explicitly select gcov tool and disable find_program
# - replace both functions setup_target_for_coverage_gcovr_* with a single setup_target_for_coverage_gcovr
# - add support for all gcovr output formats
#
# USAGE:
#
# 1. Copy this file into your cmake modules path.
#
# 2. Add the following line to your CMakeLists.txt (best inside an if-condition
# using a CMake option() to enable it just optionally):
# include(CodeCoverage)
#
# 3. Append necessary compiler flags for all supported source files:
# append_coverage_compiler_flags()
# Or for specific target:
# append_coverage_compiler_flags_to_target(YOUR_TARGET_NAME)
#
# 3.a (OPTIONAL) Set appropriate optimization flags, e.g. -O0, -O1 or -Og
#
# 4. If you need to exclude additional directories from the report, specify them
# using full paths in the COVERAGE_EXCLUDES variable before calling
# setup_target_for_coverage_*().
# Example:
# set(COVERAGE_EXCLUDES
# '${PROJECT_SOURCE_DIR}/src/dir1/*'
# '/path/to/my/src/dir2/*')
# Or, use the EXCLUDE argument to setup_target_for_coverage_*().
# Example:
# setup_target_for_coverage_gcovr(
# NAME coverage
# EXECUTABLE testrunner
# EXCLUDE "${PROJECT_SOURCE_DIR}/src/dir1/*" "/path/to/my/src/dir2/*")
#
# 4.a NOTE: With CMake 3.4+, COVERAGE_EXCLUDES or EXCLUDE can also be set
# relative to the BASE_DIRECTORY (default: PROJECT_SOURCE_DIR)
# Example:
# set(COVERAGE_EXCLUDES "dir1/*")
# setup_target_for_coverage_gcovr(
# NAME coverage
# EXECUTABLE testrunner
# FORMAT html-details
# BASE_DIRECTORY "${PROJECT_SOURCE_DIR}/src"
# EXCLUDE "dir2/*")
#
# 4.b If you need to pass specific options to gcovr, specify them in
# GCOVR_ADDITIONAL_ARGS variable.
# Example:
# set (GCOVR_ADDITIONAL_ARGS --exclude-throw-branches --exclude-noncode-lines -s)
# setup_target_for_coverage_gcovr(
# NAME coverage
# EXECUTABLE testrunner
# EXCLUDE "src/dir1" "src/dir2")
#
# 5. Use the functions described below to create a custom make target which
# runs your test executable and produces a code coverage report.
#
# 6. Build a Debug build:
# cmake -DCMAKE_BUILD_TYPE=Debug ..
# make
# make my_coverage_target
include(CMakeParseArguments)
option(CODE_COVERAGE_VERBOSE "Verbose information" FALSE)
# Check prereqs
find_program( GCOVR_PATH gcovr PATHS ${CMAKE_SOURCE_DIR}/scripts/test)
if(DEFINED CODE_COVERAGE_GCOV_TOOL)
set(GCOV_TOOL "${CODE_COVERAGE_GCOV_TOOL}")
elseif(DEFINED ENV{CODE_COVERAGE_GCOV_TOOL})
set(GCOV_TOOL "$ENV{CODE_COVERAGE_GCOV_TOOL}")
elseif("${CMAKE_CXX_COMPILER_ID}" MATCHES "(Apple)?[Cc]lang")
if(APPLE)
execute_process( COMMAND xcrun -f llvm-cov
OUTPUT_VARIABLE LLVMCOV_PATH
OUTPUT_STRIP_TRAILING_WHITESPACE
)
else()
find_program( LLVMCOV_PATH llvm-cov )
endif()
if(LLVMCOV_PATH)
set(GCOV_TOOL "${LLVMCOV_PATH} gcov")
endif()
elseif("${CMAKE_CXX_COMPILER_ID}" MATCHES "GNU")
find_program( GCOV_PATH gcov )
set(GCOV_TOOL "${GCOV_PATH}")
endif()
# Check supported compiler (Clang, GNU and Flang)
get_property(LANGUAGES GLOBAL PROPERTY ENABLED_LANGUAGES)
foreach(LANG ${LANGUAGES})
if("${CMAKE_${LANG}_COMPILER_ID}" MATCHES "(Apple)?[Cc]lang")
if("${CMAKE_${LANG}_COMPILER_VERSION}" VERSION_LESS 3)
message(FATAL_ERROR "Clang version must be 3.0.0 or greater! Aborting...")
endif()
elseif(NOT "${CMAKE_${LANG}_COMPILER_ID}" MATCHES "GNU"
AND NOT "${CMAKE_${LANG}_COMPILER_ID}" MATCHES "(LLVM)?[Ff]lang")
message(FATAL_ERROR "Compiler is not GNU or Flang! Aborting...")
endif()
endforeach()
set(COVERAGE_COMPILER_FLAGS "-g --coverage"
CACHE INTERNAL "")
if(CMAKE_CXX_COMPILER_ID MATCHES "(GNU|Clang)")
include(CheckCXXCompilerFlag)
check_cxx_compiler_flag(-fprofile-abs-path HAVE_cxx_fprofile_abs_path)
if(HAVE_cxx_fprofile_abs_path)
set(COVERAGE_CXX_COMPILER_FLAGS "${COVERAGE_COMPILER_FLAGS} -fprofile-abs-path")
endif()
include(CheckCCompilerFlag)
check_c_compiler_flag(-fprofile-abs-path HAVE_c_fprofile_abs_path)
if(HAVE_c_fprofile_abs_path)
set(COVERAGE_C_COMPILER_FLAGS "${COVERAGE_COMPILER_FLAGS} -fprofile-abs-path")
endif()
endif()
set(CMAKE_Fortran_FLAGS_COVERAGE
${COVERAGE_COMPILER_FLAGS}
CACHE STRING "Flags used by the Fortran compiler during coverage builds."
FORCE )
set(CMAKE_CXX_FLAGS_COVERAGE
${COVERAGE_COMPILER_FLAGS}
CACHE STRING "Flags used by the C++ compiler during coverage builds."
FORCE )
set(CMAKE_C_FLAGS_COVERAGE
${COVERAGE_COMPILER_FLAGS}
CACHE STRING "Flags used by the C compiler during coverage builds."
FORCE )
set(CMAKE_EXE_LINKER_FLAGS_COVERAGE
""
CACHE STRING "Flags used for linking binaries during coverage builds."
FORCE )
set(CMAKE_SHARED_LINKER_FLAGS_COVERAGE
""
CACHE STRING "Flags used by the shared libraries linker during coverage builds."
FORCE )
mark_as_advanced(
CMAKE_Fortran_FLAGS_COVERAGE
CMAKE_CXX_FLAGS_COVERAGE
CMAKE_C_FLAGS_COVERAGE
CMAKE_EXE_LINKER_FLAGS_COVERAGE
CMAKE_SHARED_LINKER_FLAGS_COVERAGE )
get_property(GENERATOR_IS_MULTI_CONFIG GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)
if(NOT (CMAKE_BUILD_TYPE STREQUAL "Debug" OR GENERATOR_IS_MULTI_CONFIG))
message(WARNING "Code coverage results with an optimised (non-Debug) build may be misleading")
endif() # NOT (CMAKE_BUILD_TYPE STREQUAL "Debug" OR GENERATOR_IS_MULTI_CONFIG)
if(CMAKE_C_COMPILER_ID STREQUAL "GNU" OR CMAKE_Fortran_COMPILER_ID STREQUAL "GNU")
link_libraries(gcov)
endif()
# Defines a target for running and collection code coverage information
# Builds dependencies, runs the given executable and outputs reports.
# NOTE! The executable should always have a ZERO as exit code otherwise
# the coverage generation will not complete.
#
# setup_target_for_coverage_gcovr(
# NAME ctest_coverage # New target name
# EXECUTABLE ctest -j ${PROCESSOR_COUNT} # Executable in PROJECT_BINARY_DIR
# DEPENDENCIES executable_target # Dependencies to build first
# BASE_DIRECTORY "../" # Base directory for report
# # (defaults to PROJECT_SOURCE_DIR)
# FORMAT "cobertura" # Output format, one of:
# # xml cobertura sonarqube json-summary
# # json-details coveralls csv txt
# # html-single html-nested html-details
# # (xml is an alias to cobertura;
# # if no format is set, defaults to xml)
# EXCLUDE "src/dir1/*" "src/dir2/*" # Patterns to exclude (can be relative
# # to BASE_DIRECTORY, with CMake 3.4+)
# )
# The user can set the variable GCOVR_ADDITIONAL_ARGS to supply additional flags to the
# GCVOR command.
function(setup_target_for_coverage_gcovr)
set(options NONE)
set(oneValueArgs BASE_DIRECTORY NAME FORMAT)
set(multiValueArgs EXCLUDE EXECUTABLE EXECUTABLE_ARGS DEPENDENCIES)
cmake_parse_arguments(Coverage "${options}" "${oneValueArgs}" "${multiValueArgs}" ${ARGN})
if(NOT GCOV_TOOL)
message(FATAL_ERROR "Could not find gcov or llvm-cov tool! Aborting...")
endif()
if(NOT GCOVR_PATH)
message(FATAL_ERROR "Could not find gcovr tool! Aborting...")
endif()
# Set base directory (as absolute path), or default to PROJECT_SOURCE_DIR
if(DEFINED Coverage_BASE_DIRECTORY)
get_filename_component(BASEDIR ${Coverage_BASE_DIRECTORY} ABSOLUTE)
else()
set(BASEDIR ${PROJECT_SOURCE_DIR})
endif()
if(NOT DEFINED Coverage_FORMAT)
set(Coverage_FORMAT xml)
endif()
if("--output" IN_LIST GCOVR_ADDITIONAL_ARGS)
message(FATAL_ERROR "Unsupported --output option detected in GCOVR_ADDITIONAL_ARGS! Aborting...")
else()
if((Coverage_FORMAT STREQUAL "html-details")
OR (Coverage_FORMAT STREQUAL "html-nested"))
set(GCOVR_OUTPUT_FILE ${PROJECT_BINARY_DIR}/${Coverage_NAME}/index.html)
set(GCOVR_CREATE_FOLDER ${PROJECT_BINARY_DIR}/${Coverage_NAME})
elseif(Coverage_FORMAT STREQUAL "html-single")
set(GCOVR_OUTPUT_FILE ${Coverage_NAME}.html)
elseif((Coverage_FORMAT STREQUAL "json-summary")
OR (Coverage_FORMAT STREQUAL "json-details")
OR (Coverage_FORMAT STREQUAL "coveralls"))
set(GCOVR_OUTPUT_FILE ${Coverage_NAME}.json)
elseif(Coverage_FORMAT STREQUAL "txt")
set(GCOVR_OUTPUT_FILE ${Coverage_NAME}.txt)
elseif(Coverage_FORMAT STREQUAL "csv")
set(GCOVR_OUTPUT_FILE ${Coverage_NAME}.csv)
else()
set(GCOVR_OUTPUT_FILE ${Coverage_NAME}.xml)
endif()
endif()
if((Coverage_FORMAT STREQUAL "cobertura")
OR (Coverage_FORMAT STREQUAL "xml"))
list(APPEND GCOVR_ADDITIONAL_ARGS --cobertura "${GCOVR_OUTPUT_FILE}" )
list(APPEND GCOVR_ADDITIONAL_ARGS --cobertura-pretty )
set(Coverage_FORMAT cobertura) # overwrite xml
elseif(Coverage_FORMAT STREQUAL "sonarqube")
list(APPEND GCOVR_ADDITIONAL_ARGS --sonarqube "${GCOVR_OUTPUT_FILE}" )
elseif(Coverage_FORMAT STREQUAL "json-summary")
list(APPEND GCOVR_ADDITIONAL_ARGS --json-summary "${GCOVR_OUTPUT_FILE}" )
list(APPEND GCOVR_ADDITIONAL_ARGS --json-summary-pretty)
elseif(Coverage_FORMAT STREQUAL "json-details")
list(APPEND GCOVR_ADDITIONAL_ARGS --json "${GCOVR_OUTPUT_FILE}" )
list(APPEND GCOVR_ADDITIONAL_ARGS --json-pretty)
elseif(Coverage_FORMAT STREQUAL "coveralls")
list(APPEND GCOVR_ADDITIONAL_ARGS --coveralls "${GCOVR_OUTPUT_FILE}" )
list(APPEND GCOVR_ADDITIONAL_ARGS --coveralls-pretty)
elseif(Coverage_FORMAT STREQUAL "csv")
list(APPEND GCOVR_ADDITIONAL_ARGS --csv "${GCOVR_OUTPUT_FILE}" )
elseif(Coverage_FORMAT STREQUAL "txt")
list(APPEND GCOVR_ADDITIONAL_ARGS --txt "${GCOVR_OUTPUT_FILE}" )
elseif(Coverage_FORMAT STREQUAL "html-single")
list(APPEND GCOVR_ADDITIONAL_ARGS --html "${GCOVR_OUTPUT_FILE}" )
list(APPEND GCOVR_ADDITIONAL_ARGS --html-self-contained)
elseif(Coverage_FORMAT STREQUAL "html-nested")
list(APPEND GCOVR_ADDITIONAL_ARGS --html-nested "${GCOVR_OUTPUT_FILE}" )
elseif(Coverage_FORMAT STREQUAL "html-details")
list(APPEND GCOVR_ADDITIONAL_ARGS --html-details "${GCOVR_OUTPUT_FILE}" )
else()
message(FATAL_ERROR "Unsupported output style ${Coverage_FORMAT}! Aborting...")
endif()
# Collect excludes (CMake 3.4+: Also compute absolute paths)
set(GCOVR_EXCLUDES "")
foreach(EXCLUDE ${Coverage_EXCLUDE} ${COVERAGE_EXCLUDES} ${COVERAGE_GCOVR_EXCLUDES})
if(CMAKE_VERSION VERSION_GREATER 3.4)
get_filename_component(EXCLUDE ${EXCLUDE} ABSOLUTE BASE_DIR ${BASEDIR})
endif()
list(APPEND GCOVR_EXCLUDES "${EXCLUDE}")
endforeach()
list(REMOVE_DUPLICATES GCOVR_EXCLUDES)
# Combine excludes to several -e arguments
set(GCOVR_EXCLUDE_ARGS "")
foreach(EXCLUDE ${GCOVR_EXCLUDES})
list(APPEND GCOVR_EXCLUDE_ARGS "-e")
list(APPEND GCOVR_EXCLUDE_ARGS "${EXCLUDE}")
endforeach()
# Set up commands which will be run to generate coverage data
# Run tests
set(GCOVR_EXEC_TESTS_CMD
${Coverage_EXECUTABLE} ${Coverage_EXECUTABLE_ARGS}
)
# Create folder
if(DEFINED GCOVR_CREATE_FOLDER)
set(GCOVR_FOLDER_CMD
${CMAKE_COMMAND} -E make_directory ${GCOVR_CREATE_FOLDER})
else()
set(GCOVR_FOLDER_CMD echo) # dummy
endif()
# Running gcovr
set(GCOVR_CMD
${GCOVR_PATH}
--gcov-executable ${GCOV_TOOL}
--gcov-ignore-parse-errors=negative_hits.warn_once_per_file
-r ${BASEDIR}
${GCOVR_ADDITIONAL_ARGS}
${GCOVR_EXCLUDE_ARGS}
--object-directory=${PROJECT_BINARY_DIR}
)
if(CODE_COVERAGE_VERBOSE)
message(STATUS "Executed command report")
message(STATUS "Command to run tests: ")
string(REPLACE ";" " " GCOVR_EXEC_TESTS_CMD_SPACED "${GCOVR_EXEC_TESTS_CMD}")
message(STATUS "${GCOVR_EXEC_TESTS_CMD_SPACED}")
if(NOT GCOVR_FOLDER_CMD STREQUAL "echo")
message(STATUS "Command to create a folder: ")
string(REPLACE ";" " " GCOVR_FOLDER_CMD_SPACED "${GCOVR_FOLDER_CMD}")
message(STATUS "${GCOVR_FOLDER_CMD_SPACED}")
endif()
message(STATUS "Command to generate gcovr coverage data: ")
string(REPLACE ";" " " GCOVR_CMD_SPACED "${GCOVR_CMD}")
message(STATUS "${GCOVR_CMD_SPACED}")
endif()
add_custom_target(${Coverage_NAME}
COMMAND ${GCOVR_EXEC_TESTS_CMD}
COMMAND ${GCOVR_FOLDER_CMD}
COMMAND ${GCOVR_CMD}
BYPRODUCTS ${GCOVR_OUTPUT_FILE}
WORKING_DIRECTORY ${PROJECT_BINARY_DIR}
DEPENDS ${Coverage_DEPENDENCIES}
VERBATIM # Protect arguments to commands
COMMENT "Running gcovr to produce code coverage report."
)
# Show info where to find the report
add_custom_command(TARGET ${Coverage_NAME} POST_BUILD
COMMAND ;
COMMENT "Code coverage report saved in ${GCOVR_OUTPUT_FILE} formatted as ${Coverage_FORMAT}"
)
endfunction() # setup_target_for_coverage_gcovr
function(append_coverage_compiler_flags)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${COVERAGE_COMPILER_FLAGS}" PARENT_SCOPE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${COVERAGE_COMPILER_FLAGS}" PARENT_SCOPE)
set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} ${COVERAGE_COMPILER_FLAGS}" PARENT_SCOPE)
message(STATUS "Appending code coverage compiler flags: ${COVERAGE_COMPILER_FLAGS}")
endfunction() # append_coverage_compiler_flags
# Setup coverage for specific library
function(append_coverage_compiler_flags_to_target name)
separate_arguments(_flag_list NATIVE_COMMAND "${COVERAGE_COMPILER_FLAGS}")
target_compile_options(${name} PRIVATE ${_flag_list})
if(CMAKE_C_COMPILER_ID STREQUAL "GNU" OR CMAKE_Fortran_COMPILER_ID STREQUAL "GNU")
target_link_libraries(${name} PRIVATE gcov)
endif()
endfunction()

View File

@@ -0,0 +1,56 @@
include (CMakeFindDependencyMacro)
# need to represent system dependencies of the lib here
#[=========================================================[
Boost
#]=========================================================]
if (static OR APPLE OR MSVC)
set (Boost_USE_STATIC_LIBS ON)
endif ()
set (Boost_USE_MULTITHREADED ON)
if (static OR MSVC)
set (Boost_USE_STATIC_RUNTIME ON)
else ()
set (Boost_USE_STATIC_RUNTIME OFF)
endif ()
find_dependency (Boost 1.70
COMPONENTS
chrono
container
context
coroutine
date_time
filesystem
program_options
regex
system
thread)
#[=========================================================[
OpenSSL
#]=========================================================]
if (NOT DEFINED OPENSSL_ROOT_DIR)
if (DEFINED ENV{OPENSSL_ROOT})
set (OPENSSL_ROOT_DIR $ENV{OPENSSL_ROOT})
elseif (APPLE)
find_program (homebrew brew)
if (homebrew)
execute_process (COMMAND ${homebrew} --prefix openssl
OUTPUT_VARIABLE OPENSSL_ROOT_DIR
OUTPUT_STRIP_TRAILING_WHITESPACE)
endif ()
endif ()
file (TO_CMAKE_PATH "${OPENSSL_ROOT_DIR}" OPENSSL_ROOT_DIR)
endif ()
if (static OR APPLE OR MSVC)
set (OPENSSL_USE_STATIC_LIBS ON)
endif ()
set (OPENSSL_MSVC_STATIC_RT ON)
find_dependency (OpenSSL 1.1.1 REQUIRED)
find_dependency (ZLIB)
find_dependency (date)
if (TARGET ZLIB::ZLIB)
set_target_properties(OpenSSL::Crypto PROPERTIES
INTERFACE_LINK_LIBRARIES ZLIB::ZLIB)
endif ()
include ("${CMAKE_CURRENT_LIST_DIR}/RippleTargets.cmake")

View File

@@ -0,0 +1,191 @@
#[===================================================================[
setup project-wide compiler settings
#]===================================================================]
#[=========================================================[
TODO some/most of these common settings belong in a
toolchain file, especially the ABI-impacting ones
#]=========================================================]
add_library (common INTERFACE)
add_library (Ripple::common ALIAS common)
# add a single global dependency on this interface lib
link_libraries (Ripple::common)
set_target_properties (common
PROPERTIES INTERFACE_POSITION_INDEPENDENT_CODE ON)
set(CMAKE_CXX_EXTENSIONS OFF)
target_compile_definitions (common
INTERFACE
$<$<CONFIG:Debug>:DEBUG _DEBUG>
$<$<AND:$<BOOL:${profile}>,$<NOT:$<BOOL:${assert}>>>:NDEBUG>)
# ^^^^ NOTE: CMAKE release builds already have NDEBUG
# defined, so no need to add it explicitly except for
# this special case of (profile ON) and (assert OFF)
# -- presumably this is because we don't want profile
# builds asserting unless asserts were specifically
# requested
if (MSVC)
# remove existing exception flag since we set it to -EHa
string (REGEX REPLACE "[-/]EH[a-z]+" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
foreach (var_
CMAKE_C_FLAGS_DEBUG CMAKE_C_FLAGS_RELEASE
CMAKE_CXX_FLAGS_DEBUG CMAKE_CXX_FLAGS_RELEASE)
# also remove dynamic runtime
string (REGEX REPLACE "[-/]MD[d]*" " " ${var_} "${${var_}}")
# /ZI (Edit & Continue debugging information) is incompatible with Gy-
string (REPLACE "/ZI" "/Zi" ${var_} "${${var_}}")
# omit debug info completely under CI (not needed)
if (is_ci)
string (REPLACE "/Zi" " " ${var_} "${${var_}}")
endif ()
endforeach ()
target_compile_options (common
INTERFACE
-bigobj # Increase object file max size
-fp:precise # Floating point behavior
-Gd # __cdecl calling convention
-Gm- # Minimal rebuild: disabled
-Gy- # Function level linking: disabled
-MP # Multiprocessor compilation
-openmp- # pragma omp: disabled
-errorReport:none # No error reporting to Internet
-nologo # Suppress login banner
-wd4018 # Disable signed/unsigned comparison warnings
-wd4244 # Disable float to int possible loss of data warnings
-wd4267 # Disable size_t to T possible loss of data warnings
-wd4800 # Disable C4800(int to bool performance)
-wd4503 # Decorated name length exceeded, name was truncated
$<$<COMPILE_LANGUAGE:CXX>:
-EHa
-GR
>
$<$<CONFIG:Release>:-Ox>
$<$<AND:$<COMPILE_LANGUAGE:CXX>,$<CONFIG:Debug>>:
-GS
-Zc:forScope
>
# static runtime
$<$<CONFIG:Debug>:-MTd>
$<$<NOT:$<CONFIG:Debug>>:-MT>
$<$<BOOL:${werr}>:-WX>
)
target_compile_definitions (common
INTERFACE
_WIN32_WINNT=0x6000
_SCL_SECURE_NO_WARNINGS
_CRT_SECURE_NO_WARNINGS
WIN32_CONSOLE
WIN32_LEAN_AND_MEAN
NOMINMAX
# TODO: Resolve these warnings, don't just silence them
_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS
$<$<AND:$<COMPILE_LANGUAGE:CXX>,$<CONFIG:Debug>>:_CRTDBG_MAP_ALLOC>)
target_link_libraries (common
INTERFACE
-errorreport:none
-machine:X64)
else ()
# HACK : because these need to come first, before any warning demotion
string (APPEND CMAKE_CXX_FLAGS " -Wall -Wdeprecated")
if (wextra)
string (APPEND CMAKE_CXX_FLAGS " -Wextra -Wno-unused-parameter")
endif ()
# not MSVC
target_compile_options (common
INTERFACE
$<$<BOOL:${werr}>:-Werror>
$<$<COMPILE_LANGUAGE:CXX>:
-frtti
-Wnon-virtual-dtor
>
-Wno-sign-compare
-Wno-char-subscripts
-Wno-format
-Wno-unused-local-typedefs
-fstack-protector
$<$<BOOL:${is_gcc}>:
-Wno-unused-but-set-variable
-Wno-deprecated
>
$<$<NOT:$<CONFIG:Debug>>:-fno-strict-aliasing>
# tweak gcc optimization for debug
$<$<AND:$<BOOL:${is_gcc}>,$<CONFIG:Debug>>:-O0>
# Add debug symbols to release config
$<$<CONFIG:Release>:-g>)
target_link_libraries (common
INTERFACE
-rdynamic
$<$<BOOL:${is_linux}>:-Wl,-z,relro,-z,now>
# link to static libc/c++ iff:
# * static option set and
# * NOT APPLE (AppleClang does not support static libc/c++) and
# * NOT san (sanitizers typically don't work with static libc/c++)
$<$<AND:$<BOOL:${static}>,$<NOT:$<BOOL:${APPLE}>>,$<NOT:$<BOOL:${san}>>>:
-static-libstdc++
-static-libgcc
>)
endif ()
if (use_mold)
# use mold linker if available
execute_process (
COMMAND ${CMAKE_CXX_COMPILER} -fuse-ld=mold -Wl,--version
ERROR_QUIET OUTPUT_VARIABLE LD_VERSION)
if ("${LD_VERSION}" MATCHES "mold")
target_link_libraries (common INTERFACE -fuse-ld=mold)
endif ()
unset (LD_VERSION)
elseif (use_gold AND is_gcc)
# use gold linker if available
execute_process (
COMMAND ${CMAKE_CXX_COMPILER} -fuse-ld=gold -Wl,--version
ERROR_QUIET OUTPUT_VARIABLE LD_VERSION)
#[=========================================================[
NOTE: THE gold linker inserts -rpath as DT_RUNPATH by
default intead of DT_RPATH, so you might have slightly
unexpected runtime ld behavior if you were expecting
DT_RPATH. Specify --disable-new-dtags to gold if you do
not want the default DT_RUNPATH behavior. This rpath
treatment as well as static/dynamic selection means that
gold does not currently have ideal default behavior when
we are using jemalloc. Thus for simplicity we don't use
it when jemalloc is requested. An alternative to
disabling would be to figure out all the settings
required to make gold play nicely with jemalloc.
#]=========================================================]
if (("${LD_VERSION}" MATCHES "GNU gold") AND (NOT jemalloc))
target_link_libraries (common
INTERFACE
-fuse-ld=gold
-Wl,--no-as-needed
#[=========================================================[
see https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/1253638/comments/5
DT_RUNPATH does not work great for transitive
dependencies (of which boost has a few) - so just
switch to DT_RPATH if doing dynamic linking with gold
#]=========================================================]
$<$<NOT:$<BOOL:${static}>>:-Wl,--disable-new-dtags>)
endif ()
unset (LD_VERSION)
elseif (use_lld)
# use lld linker if available
execute_process (
COMMAND ${CMAKE_CXX_COMPILER} -fuse-ld=lld -Wl,--version
ERROR_QUIET OUTPUT_VARIABLE LD_VERSION)
if ("${LD_VERSION}" MATCHES "LLD")
target_link_libraries (common INTERFACE -fuse-ld=lld)
endif ()
unset (LD_VERSION)
endif()
if (assert)
foreach (var_ CMAKE_C_FLAGS_RELEASE CMAKE_CXX_FLAGS_RELEASE)
STRING (REGEX REPLACE "[-/]DNDEBUG" "" ${var_} "${${var_}}")
endforeach ()
endif ()

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,38 @@
#[===================================================================[
coverage report target
#]===================================================================]
if(NOT coverage)
message(FATAL_ERROR "Code coverage not enabled! Aborting ...")
endif()
if(CMAKE_CXX_COMPILER_ID MATCHES "MSVC")
message(WARNING "Code coverage on Windows is not supported, ignoring 'coverage' flag")
return()
endif()
include(CodeCoverage)
# The instructions for these commands come from the `CodeCoverage` module,
# which was copied from https://github.com/bilke/cmake-modules, commit fb7d2a3,
# then locally changed (see CHANGES: section in `CodeCoverage.cmake`)
set(GCOVR_ADDITIONAL_ARGS ${coverage_extra_args})
if(NOT GCOVR_ADDITIONAL_ARGS STREQUAL "")
separate_arguments(GCOVR_ADDITIONAL_ARGS)
endif()
list(APPEND GCOVR_ADDITIONAL_ARGS
--exclude-throw-branches
--exclude-noncode-lines
--exclude-unreachable-branches -s
-j ${coverage_test_parallelism})
setup_target_for_coverage_gcovr(
NAME coverage
FORMAT ${coverage_format}
EXECUTABLE rippled
EXECUTABLE_ARGS --unittest$<$<BOOL:${coverage_test}>:=${coverage_test}> --unittest-jobs ${coverage_test_parallelism} --quiet --unittest-log
EXCLUDE "src/test" "${CMAKE_BINARY_DIR}/proto_gen" "${CMAKE_BINARY_DIR}/proto_gen_grpc"
DEPENDENCIES rippled
)

View File

@@ -0,0 +1,84 @@
#[===================================================================[
docs target (optional)
#]===================================================================]
option(with_docs "Include the docs target?" FALSE)
if(NOT (with_docs OR only_docs))
return()
endif()
find_package(Doxygen)
if(NOT TARGET Doxygen::doxygen)
message(STATUS "doxygen executable not found -- skipping docs target")
return()
endif()
set(doxygen_output_directory "${CMAKE_BINARY_DIR}/docs")
set(doxygen_include_path "${CMAKE_CURRENT_SOURCE_DIR}/src")
set(doxygen_index_file "${doxygen_output_directory}/html/index.html")
set(doxyfile "${CMAKE_CURRENT_SOURCE_DIR}/docs/Doxyfile")
file(GLOB_RECURSE doxygen_input
docs/*.md
src/ripple/*.h
src/ripple/*.cpp
src/ripple/*.md
src/test/*.h
src/test/*.md
Builds/*/README.md)
list(APPEND doxygen_input
README.md
RELEASENOTES.md
src/README.md)
set(dependencies "${doxygen_input}" "${doxyfile}")
function(verbose_find_path variable name)
# find_path sets a CACHE variable, so don't try using a "local" variable.
find_path(${variable} "${name}" ${ARGN})
if(NOT ${variable})
message(NOTICE "could not find ${name}")
else()
message(STATUS "found ${name}: ${${variable}}/${name}")
endif()
endfunction()
verbose_find_path(doxygen_plantuml_jar_path plantuml.jar PATH_SUFFIXES share/plantuml)
verbose_find_path(doxygen_dot_path dot)
# https://en.cppreference.com/w/Cppreference:Archives
# https://stackoverflow.com/questions/60822559/how-to-move-a-file-download-from-configure-step-to-build-step
set(download_script "${CMAKE_BINARY_DIR}/docs/download-cppreference.cmake")
file(WRITE
"${download_script}"
"file(DOWNLOAD \
http://upload.cppreference.com/mwiki/images/b/b2/html_book_20190607.zip \
${CMAKE_BINARY_DIR}/docs/cppreference.zip \
EXPECTED_HASH MD5=82b3a612d7d35a83e3cb1195a63689ab \
)\n \
execute_process( \
COMMAND \"${CMAKE_COMMAND}\" -E tar -xf cppreference.zip \
)\n"
)
set(tagfile "${CMAKE_BINARY_DIR}/docs/cppreference-doxygen-web.tag.xml")
add_custom_command(
OUTPUT "${tagfile}"
COMMAND "${CMAKE_COMMAND}" -P "${download_script}"
WORKING_DIRECTORY "${CMAKE_BINARY_DIR}/docs"
)
set(doxygen_tagfiles "${tagfile}=http://en.cppreference.com/w/")
add_custom_command(
OUTPUT "${doxygen_index_file}"
COMMAND "${CMAKE_COMMAND}" -E env
"DOXYGEN_OUTPUT_DIRECTORY=${doxygen_output_directory}"
"DOXYGEN_INCLUDE_PATH=${doxygen_include_path}"
"DOXYGEN_TAGFILES=${doxygen_tagfiles}"
"DOXYGEN_PLANTUML_JAR_PATH=${doxygen_plantuml_jar_path}"
"DOXYGEN_DOT_PATH=${doxygen_dot_path}"
"${DOXYGEN_EXECUTABLE}" "${doxyfile}"
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}"
DEPENDS "${dependencies}" "${tagfile}")
add_custom_target(docs
DEPENDS "${doxygen_index_file}"
SOURCES "${dependencies}")

View File

@@ -0,0 +1,50 @@
#[===================================================================[
install stuff
#]===================================================================]
install (
TARGETS
common
opts
ripple_syslibs
ripple_boost
xrpl_core
EXPORT RippleExports
LIBRARY DESTINATION lib
ARCHIVE DESTINATION lib
RUNTIME DESTINATION bin
INCLUDES DESTINATION include)
install (EXPORT RippleExports
FILE RippleTargets.cmake
NAMESPACE Ripple::
DESTINATION lib/cmake/ripple)
include (CMakePackageConfigHelpers)
write_basic_package_version_file (
RippleConfigVersion.cmake
VERSION ${rippled_version}
COMPATIBILITY SameMajorVersion)
if (is_root_project)
install (TARGETS rippled RUNTIME DESTINATION bin)
set_target_properties(rippled PROPERTIES INSTALL_RPATH_USE_LINK_PATH ON)
install (
FILES
${CMAKE_CURRENT_SOURCE_DIR}/Builds/CMake/RippleConfig.cmake
${CMAKE_CURRENT_BINARY_DIR}/RippleConfigVersion.cmake
DESTINATION lib/cmake/ripple)
# sample configs should not overwrite existing files
# install if-not-exists workaround as suggested by
# https://cmake.org/Bug/view.php?id=12646
install(CODE "
macro (copy_if_not_exists SRC DEST NEWNAME)
if (NOT EXISTS \"\$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}/\${DEST}/\${NEWNAME}\")
file (INSTALL FILE_PERMISSIONS OWNER_READ OWNER_WRITE DESTINATION \"\${CMAKE_INSTALL_PREFIX}/\${DEST}\" FILES \"\${SRC}\" RENAME \"\${NEWNAME}\")
else ()
message (\"-- Skipping : \$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}/\${DEST}/\${NEWNAME}\")
endif ()
endmacro()
copy_if_not_exists(\"${CMAKE_CURRENT_SOURCE_DIR}/cfg/rippled-example.cfg\" etc rippled.cfg)
copy_if_not_exists(\"${CMAKE_CURRENT_SOURCE_DIR}/cfg/validators-example.txt\" etc validators.txt)
")
endif ()

View File

@@ -0,0 +1,96 @@
#[===================================================================[
rippled compile options/settings via an interface library
#]===================================================================]
add_library (opts INTERFACE)
add_library (Ripple::opts ALIAS opts)
target_compile_definitions (opts
INTERFACE
BOOST_ASIO_DISABLE_HANDLER_TYPE_REQUIREMENTS
$<$<BOOL:${boost_show_deprecated}>:
BOOST_ASIO_NO_DEPRECATED
BOOST_FILESYSTEM_NO_DEPRECATED
>
$<$<NOT:$<BOOL:${boost_show_deprecated}>>:
BOOST_COROUTINES_NO_DEPRECATION_WARNING
BOOST_BEAST_ALLOW_DEPRECATED
BOOST_FILESYSTEM_DEPRECATED
>
$<$<BOOL:${beast_no_unit_test_inline}>:BEAST_NO_UNIT_TEST_INLINE=1>
$<$<BOOL:${beast_disable_autolink}>:BEAST_DONT_AUTOLINK_TO_WIN32_LIBRARIES=1>
$<$<BOOL:${single_io_service_thread}>:RIPPLE_SINGLE_IO_SERVICE_THREAD=1>)
target_compile_options (opts
INTERFACE
$<$<AND:$<BOOL:${is_gcc}>,$<COMPILE_LANGUAGE:CXX>>:-Wsuggest-override>
$<$<BOOL:${perf}>:-fno-omit-frame-pointer>
$<$<AND:$<BOOL:${is_gcc}>,$<BOOL:${coverage}>>:-g --coverage -fprofile-abs-path>
$<$<AND:$<BOOL:${is_clang}>,$<BOOL:${coverage}>>:-g --coverage>
$<$<BOOL:${profile}>:-pg>
$<$<AND:$<BOOL:${is_gcc}>,$<BOOL:${profile}>>:-p>)
target_link_libraries (opts
INTERFACE
$<$<AND:$<BOOL:${is_gcc}>,$<BOOL:${coverage}>>:-g --coverage -fprofile-abs-path>
$<$<AND:$<BOOL:${is_clang}>,$<BOOL:${coverage}>>:-g --coverage>
$<$<BOOL:${profile}>:-pg>
$<$<AND:$<BOOL:${is_gcc}>,$<BOOL:${profile}>>:-p>)
if(jemalloc)
find_package(jemalloc REQUIRED)
target_compile_definitions(opts INTERFACE PROFILE_JEMALLOC)
target_link_libraries(opts INTERFACE jemalloc::jemalloc)
endif ()
if (san)
target_compile_options (opts
INTERFACE
# sanitizers recommend minimum of -O1 for reasonable performance
$<$<CONFIG:Debug>:-O1>
${SAN_FLAG}
-fno-omit-frame-pointer)
target_compile_definitions (opts
INTERFACE
$<$<STREQUAL:${san},address>:SANITIZER=ASAN>
$<$<STREQUAL:${san},thread>:SANITIZER=TSAN>
$<$<STREQUAL:${san},memory>:SANITIZER=MSAN>
$<$<STREQUAL:${san},undefined>:SANITIZER=UBSAN>)
target_link_libraries (opts INTERFACE ${SAN_FLAG} ${SAN_LIB})
endif ()
#[===================================================================[
rippled transitive library deps via an interface library
#]===================================================================]
add_library (ripple_syslibs INTERFACE)
add_library (Ripple::syslibs ALIAS ripple_syslibs)
target_link_libraries (ripple_syslibs
INTERFACE
$<$<BOOL:${MSVC}>:
legacy_stdio_definitions.lib
Shlwapi
kernel32
user32
gdi32
winspool
comdlg32
advapi32
shell32
ole32
oleaut32
uuid
odbc32
odbccp32
crypt32
>
$<$<NOT:$<BOOL:${MSVC}>>:dl>
$<$<NOT:$<OR:$<BOOL:${MSVC}>,$<BOOL:${APPLE}>>>:rt>)
if (NOT MSVC)
set (THREADS_PREFER_PTHREAD_FLAG ON)
find_package (Threads)
target_link_libraries (ripple_syslibs INTERFACE Threads::Threads)
endif ()
add_library (ripple_libs INTERFACE)
add_library (Ripple::libs ALIAS ripple_libs)
target_link_libraries (ripple_libs INTERFACE Ripple::syslibs)

View File

@@ -0,0 +1,39 @@
#[===================================================================[
multiconfig misc
#]===================================================================]
if (is_multiconfig)
# This code finds all source files in the src subdirectory for inclusion
# in the IDE file tree as non-compiled sources. Since this file list will
# have some overlap with files we have already added to our targets to
# be compiled, we explicitly remove any of these target source files from
# this list.
file (GLOB_RECURSE all_sources RELATIVE ${CMAKE_CURRENT_SOURCE_DIR}
CONFIGURE_DEPENDS
src/*.* Builds/*.md docs/*.md src/*.md Builds/*.cmake)
file(GLOB md_files RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} CONFIGURE_DEPENDS
*.md)
LIST(APPEND all_sources ${md_files})
foreach (_target secp256k1::secp256k1 ed25519::ed25519 xrpl_core rippled)
get_target_property (_type ${_target} TYPE)
if(_type STREQUAL "INTERFACE_LIBRARY")
continue()
endif()
get_target_property (_src ${_target} SOURCES)
list (REMOVE_ITEM all_sources ${_src})
endforeach ()
target_sources (rippled PRIVATE ${all_sources})
set_property (
SOURCE ${all_sources}
APPEND
PROPERTY HEADER_FILE_ONLY true)
if (MSVC)
set_property(
DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
PROPERTY VS_STARTUP_PROJECT rippled)
endif ()
group_sources(src)
group_sources(docs)
group_sources(Builds)
endif ()

View File

@@ -0,0 +1,180 @@
#[===================================================================[
package/container targets - (optional)
#]===================================================================]
if (is_root_project)
if (NOT DOCKER)
find_program (DOCKER docker)
endif ()
if (DOCKER)
# if no container label is provided, use current git hash
git_hash (commit_hash)
if (NOT container_label)
set (container_label ${commit_hash})
endif ()
message (STATUS "using [${container_label}] as build container tag...")
file (MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/packages)
#[===================================================================[
rpm
#]===================================================================]
add_custom_target (rpm_container
docker build
--pull
--build-arg GIT_COMMIT=${commit_hash}
-t rippleci/rippled-rpm-builder:${container_label}
$<$<BOOL:${rpm_cache_from}>:--cache-from=${rpm_cache_from}>
-f centos-builder/Dockerfile .
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/Builds/containers
VERBATIM
USES_TERMINAL
COMMAND_EXPAND_LISTS
SOURCES
Builds/containers/centos-builder/Dockerfile
Builds/containers/centos-builder/centos_setup.sh
Builds/containers/shared/update-rippled.sh
Builds/containers/shared/update_sources.sh
Builds/containers/shared/rippled.service
Builds/containers/shared/rippled-reporting.service
Builds/containers/packaging/rpm/rippled.spec
Builds/containers/packaging/rpm/build_rpm.sh
Builds/containers/packaging/rpm/50-rippled.preset
Builds/containers/packaging/rpm/50-rippled-reporting.preset
bin/getRippledInfo
)
exclude_from_default (rpm_container)
add_custom_target (rpm
docker run
-v ${CMAKE_CURRENT_SOURCE_DIR}:/opt/rippled_bld/pkg/rippled
-v ${CMAKE_CURRENT_BINARY_DIR}/packages:/opt/rippled_bld/pkg/out
-t rippled-rpm-builder:${container_label}
/bin/bash -c "cp -fpu rippled/Builds/containers/packaging/rpm/build_rpm.sh . && ./build_rpm.sh"
VERBATIM
USES_TERMINAL
COMMAND_EXPAND_LISTS
SOURCES
Builds/containers/packaging/rpm/rippled.spec
)
exclude_from_default (rpm)
if (NOT have_package_container)
add_dependencies(rpm rpm_container)
endif ()
#[===================================================================[
dpkg
#]===================================================================]
# currently use ubuntu 18.04 as a base b/c it has one of
# the lower versions of libc among ubuntu and debian releases.
# we could change this in the future and build with some other deb
# based system.
add_custom_target (dpkg_container
docker build
--pull
--build-arg DIST_TAG=18.04
--build-arg GIT_COMMIT=${commit_hash}
-t rippled-dpkg-builder:${container_label}
$<$<BOOL:${dpkg_cache_from}>:--cache-from=${dpkg_cache_from}>
-f ubuntu-builder/Dockerfile .
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/Builds/containers
VERBATIM
USES_TERMINAL
COMMAND_EXPAND_LISTS
SOURCES
Builds/containers/packaging/dpkg/debian/rippled-reporting.links
Builds/containers/packaging/dpkg/debian/copyright
Builds/containers/packaging/dpkg/debian/rules
Builds/containers/packaging/dpkg/debian/rippled-reporting.install
Builds/containers/packaging/dpkg/debian/rippled-reporting.postinst
Builds/containers/packaging/dpkg/debian/rippled.links
Builds/containers/packaging/dpkg/debian/rippled.prerm
Builds/containers/packaging/dpkg/debian/rippled.postinst
Builds/containers/packaging/dpkg/debian/rippled-dev.install
Builds/containers/packaging/dpkg/debian/dirs
Builds/containers/packaging/dpkg/debian/rippled.postrm
Builds/containers/packaging/dpkg/debian/rippled.conffiles
Builds/containers/packaging/dpkg/debian/compat
Builds/containers/packaging/dpkg/debian/source/format
Builds/containers/packaging/dpkg/debian/source/local-options
Builds/containers/packaging/dpkg/debian/README.Debian
Builds/containers/packaging/dpkg/debian/rippled.install
Builds/containers/packaging/dpkg/debian/rippled.preinst
Builds/containers/packaging/dpkg/debian/docs
Builds/containers/packaging/dpkg/debian/control
Builds/containers/packaging/dpkg/debian/rippled-reporting.dirs
Builds/containers/packaging/dpkg/build_dpkg.sh
Builds/containers/ubuntu-builder/Dockerfile
Builds/containers/ubuntu-builder/ubuntu_setup.sh
bin/getRippledInfo
Builds/containers/shared/install_cmake.sh
Builds/containers/shared/update-rippled.sh
Builds/containers/shared/update_sources.sh
Builds/containers/shared/rippled.service
Builds/containers/shared/rippled-reporting.service
Builds/containers/shared/rippled-logrotate
Builds/containers/shared/update-rippled-cron
)
exclude_from_default (dpkg_container)
add_custom_target (dpkg
docker run
-v ${CMAKE_CURRENT_SOURCE_DIR}:/opt/rippled_bld/pkg/rippled
-v ${CMAKE_CURRENT_BINARY_DIR}/packages:/opt/rippled_bld/pkg/out
-t rippled-dpkg-builder:${container_label}
/bin/bash -c "cp -fpu rippled/Builds/containers/packaging/dpkg/build_dpkg.sh . && ./build_dpkg.sh"
VERBATIM
USES_TERMINAL
COMMAND_EXPAND_LISTS
SOURCES
Builds/containers/packaging/dpkg/debian/control
)
exclude_from_default (dpkg)
if (NOT have_package_container)
add_dependencies(dpkg dpkg_container)
endif ()
#[===================================================================[
ci container
#]===================================================================]
# now use the same ubuntu image for our travis-ci docker images,
# but we use a newer distro (18.04 vs 16.04).
#
# the following steps assume the github pkg repo, but it's possible to
# adapt these for other docker hub repositories.
#
# steps for publishing a new CI image when you make changes:
#
# mkdir bld.ci && cd bld.ci && cmake -Dpackages_only=ON -Dcontainer_label=CI_LATEST
# cmake --build . --target ci_container --verbose
# docker tag rippled-ci-builder:CI_LATEST <HUB REPO PATH>/rippled-ci-builder:YYYY-MM-DD
# (NOTE: change YYYY-MM-DD to match current date, or use a different
# tag/version scheme if you prefer)
# docker push <HUB REPO PATH>/rippled-ci-builder:YYYY-MM-DD
# (NOTE: <HUB REPO PATH> is probably your user or org name if using
# docker hub, or it might be something like
# docker.pkg.github.com/ripple/rippled if using the github pkg
# registry. for any registry, you will need to be logged-in via
# docker and have push access.)
#
# ...then change the DOCKER_IMAGE line in .travis.yml :
# - DOCKER_IMAGE="<HUB REPO PATH>/rippled-ci-builder:YYYY-MM-DD"
add_custom_target (ci_container
docker build
--pull
--build-arg DIST_TAG=18.04
--build-arg GIT_COMMIT=${commit_hash}
--build-arg CI_USE=true
-t rippled-ci-builder:${container_label}
$<$<BOOL:${ci_cache_from}>:--cache-from=${ci_cache_from}>
-f ubuntu-builder/Dockerfile .
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/Builds/containers
VERBATIM
USES_TERMINAL
COMMAND_EXPAND_LISTS
SOURCES
Builds/containers/ubuntu-builder/Dockerfile
Builds/containers/ubuntu-builder/ubuntu_setup.sh
)
exclude_from_default (ci_container)
else ()
message (STATUS "docker NOT found -- won't be able to build containers for packaging")
endif ()
endif ()

View File

@@ -0,0 +1,80 @@
#[===================================================================[
convenience variables and sanity checks
#]===================================================================]
include(ProcessorCount)
if (NOT ep_procs)
ProcessorCount(ep_procs)
if (ep_procs GREATER 1)
# never use more than half of cores for EP builds
math (EXPR ep_procs "${ep_procs} / 2")
message (STATUS "Using ${ep_procs} cores for ExternalProject builds.")
endif ()
endif ()
get_property(is_multiconfig GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)
set (CMAKE_CONFIGURATION_TYPES "Debug;Release" CACHE STRING "" FORCE)
if (NOT is_multiconfig)
if (NOT CMAKE_BUILD_TYPE)
message (STATUS "Build type not specified - defaulting to Release")
set (CMAKE_BUILD_TYPE Release CACHE STRING "build type" FORCE)
elseif (NOT (CMAKE_BUILD_TYPE STREQUAL Debug OR CMAKE_BUILD_TYPE STREQUAL Release))
# for simplicity, these are the only two config types we care about. Limiting
# the build types simplifies dealing with external project builds especially
message (FATAL_ERROR " *** Only Debug or Release build types are currently supported ***")
endif ()
endif ()
get_directory_property(has_parent PARENT_DIRECTORY)
if (has_parent)
set (is_root_project OFF)
else ()
set (is_root_project ON)
endif ()
if ("${CMAKE_CXX_COMPILER_ID}" MATCHES ".*Clang") # both Clang and AppleClang
set (is_clang TRUE)
if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang" AND
CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8.0)
message (FATAL_ERROR "This project requires clang 8 or later")
endif ()
# TODO min AppleClang version check ?
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
set (is_gcc TRUE)
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8.0)
message (FATAL_ERROR "This project requires GCC 8 or later")
endif ()
endif ()
if (CMAKE_SYSTEM_NAME STREQUAL "Linux")
set (is_linux TRUE)
else ()
set (is_linux FALSE)
endif ()
if ("$ENV{CI}" STREQUAL "true" OR "$ENV{CONTINUOUS_INTEGRATION}" STREQUAL "true")
set (is_ci TRUE)
else ()
set (is_ci FALSE)
endif ()
# check for in-source build and fail
if ("${CMAKE_CURRENT_SOURCE_DIR}" STREQUAL "${CMAKE_BINARY_DIR}")
message (FATAL_ERROR "Builds (in-source) are not allowed in "
"${CMAKE_CURRENT_SOURCE_DIR}. Please remove CMakeCache.txt and the CMakeFiles "
"directory from ${CMAKE_CURRENT_SOURCE_DIR} and try building in a separate directory.")
endif ()
if (MSVC AND CMAKE_GENERATOR_PLATFORM STREQUAL "Win32")
message (FATAL_ERROR "Visual Studio 32-bit build is not supported.")
endif ()
if (NOT CMAKE_SIZEOF_VOID_P EQUAL 8)
message (FATAL_ERROR "Rippled requires a 64 bit target architecture.\n"
"The most likely cause of this warning is trying to build rippled with a 32-bit OS.")
endif ()
if (APPLE AND NOT HOMEBREW)
find_program (HOMEBREW brew)
endif ()

View File

@@ -0,0 +1,130 @@
#[===================================================================[
declare user options/settings
#]===================================================================]
include(ProcessorCount)
ProcessorCount(PROCESSOR_COUNT)
option(assert "Enables asserts, even in release builds" OFF)
option(reporting "Build rippled with reporting mode enabled" OFF)
option(tests "Build tests" ON)
option(unity "Creates a build using UNITY support in cmake. This is the default" ON)
if(unity)
if(NOT is_ci)
set(CMAKE_UNITY_BUILD_BATCH_SIZE 15 CACHE STRING "")
endif()
endif()
if(is_gcc OR is_clang)
option(coverage "Generates coverage info." OFF)
option(profile "Add profiling flags" OFF)
set(coverage_test_parallelism "${PROCESSOR_COUNT}" CACHE STRING
"Unit tests parallelism for the purpose of coverage report.")
set(coverage_format "html-details" CACHE STRING
"Output format of the coverage report.")
set(coverage_extra_args "" CACHE STRING
"Additional arguments to pass to gcovr.")
set(coverage_test "" CACHE STRING
"On gcc & clang, the specific unit test(s) to run for coverage. Default is all tests.")
if(coverage_test AND NOT coverage)
set(coverage ON CACHE BOOL "gcc/clang only" FORCE)
endif()
option(wextra "compile with extra gcc/clang warnings enabled" ON)
else()
set(profile OFF CACHE BOOL "gcc/clang only" FORCE)
set(coverage OFF CACHE BOOL "gcc/clang only" FORCE)
set(wextra OFF CACHE BOOL "gcc/clang only" FORCE)
endif()
if(is_linux)
option(BUILD_SHARED_LIBS "build shared ripple libraries" OFF)
option(static "link protobuf, openssl, libc++, and boost statically" ON)
option(perf "Enables flags that assist with perf recording" OFF)
option(use_gold "enables detection of gold (binutils) linker" ON)
option(use_mold "enables detection of mold (binutils) linker" ON)
else()
# we are not ready to allow shared-libs on windows because it would require
# export declarations. On macos it's more feasible, but static openssl
# produces odd linker errors, thus we disable shared lib builds for now.
set(BUILD_SHARED_LIBS OFF CACHE BOOL "build shared ripple libraries - OFF for win/macos" FORCE)
set(static ON CACHE BOOL "static link, linux only. ON for WIN/macos" FORCE)
set(perf OFF CACHE BOOL "perf flags, linux only" FORCE)
set(use_gold OFF CACHE BOOL "gold linker, linux only" FORCE)
set(use_mold OFF CACHE BOOL "mold linker, linux only" FORCE)
endif()
if(is_clang)
option(use_lld "enables detection of lld linker" ON)
else()
set(use_lld OFF CACHE BOOL "try lld linker, clang only" FORCE)
endif()
option(jemalloc "Enables jemalloc for heap profiling" OFF)
option(werr "treat warnings as errors" OFF)
option(local_protobuf
"Force a local build of protobuf instead of looking for an installed version." OFF)
option(local_grpc
"Force a local build of gRPC instead of looking for an installed version." OFF)
# this one is a string and therefore can't be an option
set(san "" CACHE STRING "On gcc & clang, add sanitizer instrumentation")
set_property(CACHE san PROPERTY STRINGS ";undefined;memory;address;thread")
if(san)
string(TOLOWER ${san} san)
set(SAN_FLAG "-fsanitize=${san}")
set(SAN_LIB "")
if(is_gcc)
if(san STREQUAL "address")
set(SAN_LIB "asan")
elseif(san STREQUAL "thread")
set(SAN_LIB "tsan")
elseif(san STREQUAL "memory")
set(SAN_LIB "msan")
elseif(san STREQUAL "undefined")
set(SAN_LIB "ubsan")
endif()
endif()
set(_saved_CRL ${CMAKE_REQUIRED_LIBRARIES})
set(CMAKE_REQUIRED_LIBRARIES "${SAN_FLAG};${SAN_LIB}")
check_cxx_compiler_flag(${SAN_FLAG} COMPILER_SUPPORTS_SAN)
set(CMAKE_REQUIRED_LIBRARIES ${_saved_CRL})
if(NOT COMPILER_SUPPORTS_SAN)
message(FATAL_ERROR "${san} sanitizer does not seem to be supported by your compiler")
endif()
endif()
set(container_label "" CACHE STRING "tag to use for package building containers")
option(packages_only
"ONLY generate package building targets. This is special use-case and almost \
certainly not what you want. Use with caution as you won't be able to build \
any compiled targets locally." OFF)
option(have_package_container
"Sometimes you already have the tagged container you want to use for package \
building and you don't want docker to rebuild it. This flag will detach the \
dependency of the package build from the container build. It's an advanced \
use case and most likely you should not be touching this flag." OFF)
# the remaining options are obscure and rarely used
option(beast_no_unit_test_inline
"Prevents unit test definitions from being inserted into global table"
OFF)
option(single_io_service_thread
"Restricts the number of threads calling io_service::run to one. \
This can be useful when debugging."
OFF)
option(boost_show_deprecated
"Allow boost to fail on deprecated usage. Only useful if you're trying\
to find deprecated calls."
OFF)
option(beast_hashers
"Use local implementations for sha/ripemd hashes (experimental, not recommended)"
OFF)
if(WIN32)
option(beast_disable_autolink "Disables autolinking of system libraries on WIN32" OFF)
else()
set(beast_disable_autolink OFF CACHE BOOL "WIN32 only" FORCE)
endif()
if(coverage)
message(STATUS "coverage build requested - forcing Debug build")
set(CMAKE_BUILD_TYPE Debug CACHE STRING "build type" FORCE)
endif()

View File

@@ -0,0 +1,22 @@
option (validator_keys "Enables building of validator-keys-tool as a separate target (imported via FetchContent)" OFF)
if (validator_keys)
git_branch (current_branch)
# default to tracking VK develop branch unless we are on master/release
if (NOT (current_branch STREQUAL "master" OR current_branch STREQUAL "release"))
set (current_branch "develop")
endif ()
message (STATUS "tracking ValidatorKeys branch: ${current_branch}")
FetchContent_Declare (
validator_keys_src
GIT_REPOSITORY https://github.com/ripple/validator-keys-tool.git
GIT_TAG "${current_branch}"
)
FetchContent_GetProperties (validator_keys_src)
if (NOT validator_keys_src_POPULATED)
message (STATUS "Pausing to download ValidatorKeys...")
FetchContent_Populate (validator_keys_src)
endif ()
add_subdirectory (${validator_keys_src_SOURCE_DIR} ${CMAKE_BINARY_DIR}/validator-keys)
endif ()

View File

@@ -0,0 +1,15 @@
#[===================================================================[
read version from source
#]===================================================================]
file (STRINGS src/ripple/protocol/impl/BuildInfo.cpp BUILD_INFO)
foreach (line_ ${BUILD_INFO})
if (line_ MATCHES "versionString[ ]*=[ ]*\"(.+)\"")
set (rippled_version ${CMAKE_MATCH_1})
endif ()
endforeach ()
if (rippled_version)
message (STATUS "rippled version: ${rippled_version}")
else ()
message (FATAL_ERROR "unable to determine rippled version")
endif ()

View File

@@ -0,0 +1,106 @@
################################################################################
# SociConfig.cmake - CMake build configuration of SOCI library
################################################################################
# Copyright (C) 2010 Mateusz Loskot <mateusz@loskot.net>
#
# Distributed under the Boost Software License, Version 1.0.
# (See accompanying file LICENSE_1_0.txt or copy at
# http://www.boost.org/LICENSE_1_0.txt)
################################################################################
include(CheckCXXSymbolExists)
if(WIN32)
check_cxx_symbol_exists("_M_AMD64" "" SOCI_TARGET_ARCH_X64)
if(NOT RTC_ARCH_X64)
check_cxx_symbol_exists("_M_IX86" "" SOCI_TARGET_ARCH_X86)
endif(NOT RTC_ARCH_X64)
# add check for arm here
# see http://msdn.microsoft.com/en-us/library/b0084kay.aspx
else(WIN32)
check_cxx_symbol_exists("__i386__" "" SOCI_TARGET_ARCH_X86)
check_cxx_symbol_exists("__x86_64__" "" SOCI_TARGET_ARCH_X64)
check_cxx_symbol_exists("__arm__" "" SOCI_TARGET_ARCH_ARM)
endif(WIN32)
if(NOT DEFINED LIB_SUFFIX)
if(SOCI_TARGET_ARCH_X64)
set(_lib_suffix "64")
else()
set(_lib_suffix "")
endif()
set(LIB_SUFFIX ${_lib_suffix} CACHE STRING "Specifies suffix for the lib directory")
endif()
#
# C++11 Option
#
if(NOT SOCI_CXX_C11)
set (SOCI_CXX_C11 OFF CACHE BOOL "Build to the C++11 standard")
endif()
#
# Force compilation flags and set desired warnings level
#
if (MSVC)
add_definitions(-D_CRT_SECURE_NO_DEPRECATE)
add_definitions(-D_CRT_SECURE_NO_WARNINGS)
add_definitions(-D_CRT_NONSTDC_NO_WARNING)
add_definitions(-D_SCL_SECURE_NO_WARNINGS)
if(CMAKE_CXX_FLAGS MATCHES "/W[0-4]")
string(REGEX REPLACE "/W[0-4]" "/W4" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /W4 /we4266")
endif()
else()
set(SOCI_GCC_CLANG_COMMON_FLAGS "")
# "-pedantic -Werror -Wno-error=parentheses -Wall -Wextra -Wpointer-arith -Wcast-align -Wcast-qual -Wfloat-equal -Woverloaded-virtual -Wredundant-decls -Wno-long-long")
if (SOCI_CXX_C11)
set(SOCI_CXX_VERSION_FLAGS "-std=c++11")
else()
set(SOCI_CXX_VERSION_FLAGS "-std=gnu++98")
endif()
if("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang" OR "${CMAKE_CXX_COMPILER}" MATCHES "clang")
if(NOT CMAKE_CXX_COMPILER_VERSION LESS 3.1 AND SOCI_ASAN)
set(SOCI_GCC_CLANG_COMMON_FLAGS "${SOCI_GCC_CLANG_COMMON_FLAGS} -fsanitize=address")
endif()
# enforce C++11 for Clang
set(SOCI_CXX_C11 ON)
set(SOCI_CXX_VERSION_FLAGS "-std=c++11")
add_definitions(-DCATCH_CONFIG_CPP11_NO_IS_ENUM)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SOCI_GCC_CLANG_COMMON_FLAGS} ${SOCI_CXX_VERSION_FLAGS}")
elseif(CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_GNUCXX)
if(NOT CMAKE_CXX_COMPILER_VERSION LESS 4.8 AND SOCI_ASAN)
set(SOCI_GCC_CLANG_COMMON_FLAGS "${SOCI_GCC_CLANG_COMMON_FLAGS} -fsanitize=address")
endif()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SOCI_GCC_CLANG_COMMON_FLAGS} ${SOCI_CXX_VERSION_FLAGS} ")
if (CMAKE_COMPILER_IS_GNUCXX)
if (CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-variadic-macros")
endif()
endif()
else()
message(WARNING "Unknown toolset - using default flags to build SOCI")
endif()
endif()
# Set SOCI_HAVE_* variables for soci-config.h generator
set(SOCI_HAVE_CXX_C11 ${SOCI_CXX_C11} CACHE INTERNAL "Enables C++11 support")

View File

@@ -0,0 +1,53 @@
find_package(Boost 1.82 REQUIRED
COMPONENTS
chrono
container
context
coroutine
date_time
filesystem
json
program_options
regex
system
thread
)
add_library(ripple_boost INTERFACE)
add_library(Ripple::boost ALIAS ripple_boost)
if(XCODE)
target_include_directories(ripple_boost BEFORE INTERFACE ${Boost_INCLUDE_DIRS})
target_compile_options(ripple_boost INTERFACE --system-header-prefix="boost/")
else()
target_include_directories(ripple_boost SYSTEM BEFORE INTERFACE ${Boost_INCLUDE_DIRS})
endif()
target_link_libraries(ripple_boost
INTERFACE
Boost::boost
Boost::chrono
Boost::container
Boost::coroutine
Boost::date_time
Boost::filesystem
Boost::json
Boost::program_options
Boost::regex
Boost::system
Boost::thread)
if(Boost_COMPILER)
target_link_libraries(ripple_boost INTERFACE Boost::disable_autolinking)
endif()
if(san AND is_clang)
# TODO: gcc does not support -fsanitize-blacklist...can we do something else
# for gcc ?
if(NOT Boost_INCLUDE_DIRS AND TARGET Boost::headers)
get_target_property(Boost_INCLUDE_DIRS Boost::headers INTERFACE_INCLUDE_DIRECTORIES)
endif()
message(STATUS "Adding [${Boost_INCLUDE_DIRS}] to sanitizer blacklist")
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/san_bl.txt "src:${Boost_INCLUDE_DIRS}/*")
target_compile_options(opts
INTERFACE
# ignore boost headers for sanitizing
-fsanitize-blacklist=${CMAKE_CURRENT_BINARY_DIR}/san_bl.txt)
endif()

View File

@@ -0,0 +1,27 @@
find_package(Protobuf 3.8)
set(output_dir ${CMAKE_BINARY_DIR}/proto_gen)
file(MAKE_DIRECTORY ${output_dir})
set(ccbd ${CMAKE_CURRENT_BINARY_DIR})
set(CMAKE_CURRENT_BINARY_DIR ${output_dir})
protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS src/ripple/proto/ripple.proto)
set(CMAKE_CURRENT_BINARY_DIR ${ccbd})
target_include_directories(xrpl_core SYSTEM PUBLIC
# The generated implementation imports the header relative to the output
# directory.
$<BUILD_INTERFACE:${output_dir}>
$<BUILD_INTERFACE:${output_dir}/src>
)
target_sources(xrpl_core PRIVATE ${output_dir}/src/ripple/proto/ripple.pb.cc)
install(
FILES ${output_dir}/src/ripple/proto/ripple.pb.h
DESTINATION include/ripple/proto)
target_link_libraries(xrpl_core PUBLIC protobuf::libprotobuf)
target_compile_options(xrpl_core
PUBLIC
$<$<BOOL:${XCODE}>:
--system-header-prefix="google/protobuf"
-Wno-deprecated-dynamic-exception-spec
>
)

View File

@@ -0,0 +1,82 @@
find_package(gRPC 1.23)
#[=================================[
generate protobuf sources for
grpc defs and bundle into a
static lib
#]=================================]
set(output_dir "${CMAKE_BINARY_DIR}/proto_gen_grpc")
set(GRPC_GEN_DIR "${output_dir}/ripple/proto")
file(MAKE_DIRECTORY ${GRPC_GEN_DIR})
set(GRPC_PROTO_SRCS)
set(GRPC_PROTO_HDRS)
set(GRPC_PROTO_ROOT "${CMAKE_CURRENT_SOURCE_DIR}/src/ripple/proto/org")
file(GLOB_RECURSE GRPC_DEFINITION_FILES "${GRPC_PROTO_ROOT}/*.proto")
foreach(file ${GRPC_DEFINITION_FILES})
# /home/user/rippled/src/ripple/proto/org/.../v1/get_ledger.proto
get_filename_component(_abs_file ${file} ABSOLUTE)
# /home/user/rippled/src/ripple/proto/org/.../v1
get_filename_component(_abs_dir ${_abs_file} DIRECTORY)
# get_ledger
get_filename_component(_basename ${file} NAME_WE)
# /home/user/rippled/src/ripple/proto
get_filename_component(_proto_inc ${GRPC_PROTO_ROOT} DIRECTORY) # updir one level
# org/.../v1/get_ledger.proto
file(RELATIVE_PATH _rel_root_file ${_proto_inc} ${_abs_file})
# org/.../v1
get_filename_component(_rel_root_dir ${_rel_root_file} DIRECTORY)
# src/ripple/proto/org/.../v1
file(RELATIVE_PATH _rel_dir ${CMAKE_CURRENT_SOURCE_DIR} ${_abs_dir})
# .cmake/proto_gen_grpc/ripple/proto/org/.../v1/get_ledger.grpc.pb.cc
set(src_1 "${GRPC_GEN_DIR}/${_rel_root_dir}/${_basename}.grpc.pb.cc")
set(src_2 "${GRPC_GEN_DIR}/${_rel_root_dir}/${_basename}.pb.cc")
set(hdr_1 "${GRPC_GEN_DIR}/${_rel_root_dir}/${_basename}.grpc.pb.h")
set(hdr_2 "${GRPC_GEN_DIR}/${_rel_root_dir}/${_basename}.pb.h")
add_custom_command(
OUTPUT ${src_1} ${src_2} ${hdr_1} ${hdr_2}
COMMAND protobuf::protoc
ARGS --grpc_out=${GRPC_GEN_DIR}
--cpp_out=${GRPC_GEN_DIR}
--plugin=protoc-gen-grpc=$<TARGET_FILE:gRPC::grpc_cpp_plugin>
-I ${_proto_inc} -I ${_rel_dir}
${_abs_file}
DEPENDS ${_abs_file} protobuf::protoc gRPC::grpc_cpp_plugin
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
COMMENT "Running gRPC C++ protocol buffer compiler on ${file}"
VERBATIM)
set_source_files_properties(${src_1} ${src_2} ${hdr_1} ${hdr_2} PROPERTIES
GENERATED TRUE
SKIP_UNITY_BUILD_INCLUSION ON
)
list(APPEND GRPC_PROTO_SRCS ${src_1} ${src_2})
list(APPEND GRPC_PROTO_HDRS ${hdr_1} ${hdr_2})
endforeach()
target_include_directories(xrpl_core SYSTEM PUBLIC
$<BUILD_INTERFACE:${output_dir}>
$<BUILD_INTERFACE:${output_dir}/ripple/proto>
# The generated sources include headers relative to this path. Fix it later.
$<INSTALL_INTERFACE:include/ripple/proto>
)
target_sources(xrpl_core PRIVATE ${GRPC_PROTO_SRCS})
install(
DIRECTORY ${output_dir}/ripple
DESTINATION include/
FILES_MATCHING PATTERN "*.h"
)
target_link_libraries(xrpl_core PUBLIC
"gRPC::grpc++"
# libgrpc is missing references.
absl::random_random
)
target_compile_options(xrpl_core
PRIVATE
$<$<BOOL:${MSVC}>:-wd4065>
$<$<NOT:$<BOOL:${MSVC}>>:-Wno-deprecated-declarations>
PUBLIC
$<$<BOOL:${MSVC}>:-wd4996>
$<$<BOOL:${XCODE}>:
--system-header-prefix="google/protobuf"
-Wno-deprecated-dynamic-exception-spec
>)

View File

@@ -0,0 +1,17 @@
#[=========================================================[
This is a CMake script file that is used to write
the contents of a file to stdout (using the cmake
echo command). The input file is passed via the
IN_FILE variable.
#]=========================================================]
if (EXISTS ${IN_FILE})
file (READ ${IN_FILE} contents)
## only print files that actually have some text in them
if (contents MATCHES "[a-z0-9A-Z]+")
execute_process(
COMMAND
${CMAKE_COMMAND} -E echo "${contents}")
endif ()
endif ()

View File

@@ -1,30 +0,0 @@
# rippled
# use the ubuntu base image
FROM ubuntu
MAINTAINER Roberto Catini roberto.catini@gmail.com
# make sure the package repository is up to date
RUN apt-get update
RUN apt-get -y upgrade
# install the dependencies
RUN apt-get -y install git scons pkg-config protobuf-compiler libprotobuf-dev libssl-dev libboost1.55-all-dev
# download source code from official repository
RUN git clone https://github.com/ripple/rippled.git src; cd src/; git checkout master
# compile
RUN cd src/; scons build/rippled
# move to root directory and strip
RUN cp src/build/rippled rippled; strip rippled
# copy default config
RUN cp src/doc/rippled-example.cfg rippled.cfg
# clean source
RUN rm -r src
# launch rippled when launching the container
ENTRYPOINT ./rippled

View File

@@ -1,23 +0,0 @@
FROM ubuntu
MAINTAINER Torrie Fischer <torrie@ripple.com>
RUN apt-get update -qq &&\
apt-get install -qq software-properties-common &&\
apt-add-repository -y ppa:ubuntu-toolchain-r/test &&\
apt-add-repository -y ppa:afrank/boost &&\
apt-get update -qq
RUN apt-get purge -qq libboost1.48-dev &&\
apt-get install -qq libprotobuf8 libboost1.57-all-dev
RUN mkdir -p /srv/rippled/data
VOLUME /srv/rippled/data/
ENTRYPOINT ["/srv/rippled/bin/rippled"]
CMD ["--conf", "/srv/rippled/data/rippled.cfg"]
EXPOSE 51235/udp
EXPOSE 5005/tcp
ADD ./rippled.cfg /srv/rippled/data/rippled.cfg
ADD ./rippled /srv/rippled/bin/

View File

@@ -1,13 +0,0 @@
set -e
mkdir -p build/docker/
cp doc/rippled-example.cfg build/clang.debug/rippled build/docker/
cp Builds/Docker/Dockerfile-testnet build/docker/Dockerfile
mv build/docker/rippled-example.cfg build/docker/rippled.cfg
strip build/docker/rippled
docker build -t ripple/rippled:$CIRCLE_SHA1 build/docker/
docker tag ripple/rippled:$CIRCLE_SHA1 ripple/rippled:latest
if [ -z "$CIRCLE_PR_NUMBER" ]; then
docker tag ripple/rippled:$CIRCLE_SHA1 ripple/rippled:$CIRCLE_BRANCH
fi

View File

@@ -1,16 +0,0 @@
set -e
if [ -z "$DOCKER_EMAIL" -o -z "$DOCKER_USERNAME" -o -z "$DOCKER_PASSWORD" ];then
echo "Docker credentials are not set. Can't login to docker, no containers will be pushed."
exit 0
fi
if [ -n "$CIRCLE_PR_NUMBER" ]; then
echo "Not pushing results of a pull request build."
exit 0
fi
docker login -e $DOCKER_EMAIL -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
docker push ripple/rippled:$CIRCLE_SHA1
docker push ripple/rippled:$CIRCLE_BRANCH
docker push ripple/rippled:latest

View File

@@ -1,31 +0,0 @@
**Requirements**
1. Java Runtime Environment (JRE)
2. Eclipse with CDT (tested on Luna):
http://www.eclipse.org/downloads/packages/eclipse-ide-cc-developers/lunasr2
3. Eclipse SCons plugin: http://sconsolidator.com/
**WARNING**: by default the SCons plugin uses 16 threads. Go to
*Window->Preferences->SCons->Build Settings* in Eclipse and make it
use only 4-8 jobs(threads) or whatever you feel confortable with. It will
positively freeze your system if you run with 16 threads/jobs.
![scons](scons.png)
**Getting Started**
After setting up Eclipse just do a File->New->Other...
Select: C/C++ / New SCons project from existing source
Point the importer to the folder where the SConstruct resides (the root
folder of your git workspace normally)
**Build**
Just hit Project->Build All in Eclipse to get started. And remember to not
let it run 16 threads!
**Debug**
Start a new Eclipse debug configuration and set binary to run to build/rippled
(assuming you have built it).
![debug](debug.png)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 17 KiB

View File

@@ -1,22 +0,0 @@
#!/usr/bin/env bash
#
# This scripts installs the dependencies needed by rippled. It should be run
# with sudo.
#
if [ ! -f /etc/fedora-release ]; then
echo "This script is meant to be run on fedora"
exit 1
fi
fedora_release=$(grep -o '[0-9]*' /etc/fedora-release)
if (( $(bc <<< "${fedora_release} < 22") )); then
echo "This script is meant to run on fedora 22 or greater"
exit 1
fi
yum -y update
yum -y group install "Development Tools"
yum -y install gcc-c++ scons openssl-devel openssl-static protobuf-devel protobuf-static boost-devel boost-static libstdc++-static

View File

@@ -1,5 +0,0 @@
# QTCreator
Makefile
*.user

View File

@@ -1,112 +0,0 @@
# Ripple protocol buffers
PROTOS = ../../src/ripple_data/protocol/ripple.proto
PROTOS_DIR = ../../build/proto
# Google Protocol Buffers support
protobuf_h.name = protobuf header
protobuf_h.input = PROTOS
protobuf_h.output = $${PROTOS_DIR}/${QMAKE_FILE_BASE}.pb.h
protobuf_h.depends = ${QMAKE_FILE_NAME}
protobuf_h.commands = protoc --cpp_out=$${PROTOS_DIR} --proto_path=${QMAKE_FILE_PATH} ${QMAKE_FILE_NAME}
protobuf_h.variable_out = HEADERS
QMAKE_EXTRA_COMPILERS += protobuf_h
protobuf_cc.name = protobuf implementation
protobuf_cc.input = PROTOS
protobuf_cc.output = $${PROTOS_DIR}/${QMAKE_FILE_BASE}.pb.cc
protobuf_cc.depends = $${PROTOS_DIR}/${QMAKE_FILE_BASE}.pb.h
protobuf_cc.commands = $$escape_expand(\\n)
#protobuf_cc.variable_out = SOURCES
QMAKE_EXTRA_COMPILERS += protobuf_cc
# Ripple compilation
DESTDIR = ../../build/QtCreator
OBJECTS_DIR = ../../build/QtCreator/obj
TEMPLATE = app
CONFIG += console thread warn_off
CONFIG -= qt gui
DEFINES += _DEBUG
linux-g++:QMAKE_CXXFLAGS += \
-Wall \
-Wno-sign-compare \
-Wno-char-subscripts \
-Wno-invalid-offsetof \
-Wno-unused-parameter \
-Wformat \
-O0 \
-std=c++11 \
-pthread
INCLUDEPATH += \
"../../src/leveldb/" \
"../../src/leveldb/port" \
"../../src/leveldb/include" \
$${PROTOS_DIR}
OTHER_FILES += \
# $$files(../../src/*, true) \
# $$files(../../src/beast/*) \
# $$files(../../src/beast/modules/beast_basics/diagnostic/*)
# $$files(../../src/beast/modules/beast_core/, true)
UI_HEADERS_DIR += ../../src/ripple_basics
# ---------
# New style
#
SOURCES += \
../../src/ripple/beast/ripple_beast.unity.cpp \
../../src/ripple/beast/ripple_beastc.c \
../../src/ripple/common/ripple_common.unity.cpp \
../../src/ripple/http/ripple_http.unity.cpp \
../../src/ripple/json/ripple_json.unity.cpp \
../../src/ripple/peerfinder/ripple_peerfinder.unity.cpp \
../../src/ripple/radmap/ripple_radmap.unity.cpp \
../../src/ripple/resource/ripple_resource.unity.cpp \
../../src/ripple/sitefiles/ripple_sitefiles.unity.cpp \
../../src/ripple/sslutil/ripple_sslutil.unity.cpp \
../../src/ripple/testoverlay/ripple_testoverlay.unity.cpp \
../../src/ripple/types/ripple_types.unity.cpp \
../../src/ripple/validators/ripple_validators.unity.cpp
# ---------
# Old style
#
SOURCES += \
../../src/ripple_app/ripple_app.unity.cpp \
../../src/ripple_app/ripple_app_pt1.unity.cpp \
../../src/ripple_app/ripple_app_pt2.unity.cpp \
../../src/ripple_app/ripple_app_pt3.unity.cpp \
../../src/ripple_app/ripple_app_pt4.unity.cpp \
../../src/ripple_app/ripple_app_pt5.unity.cpp \
../../src/ripple_app/ripple_app_pt6.unity.cpp \
../../src/ripple_app/ripple_app_pt7.unity.cpp \
../../src/ripple_app/ripple_app_pt8.unity.cpp \
../../src/ripple_basics/ripple_basics.unity.cpp \
../../src/ripple_core/ripple_core.unity.cpp \
../../src/ripple_data/ripple_data.unity.cpp \
../../src/ripple_hyperleveldb/ripple_hyperleveldb.unity.cpp \
../../src/ripple_leveldb/ripple_leveldb.unity.cpp \
../../src/ripple_net/ripple_net.unity.cpp \
../../src/ripple_overlay/ripple_overlay.unity.cpp \
../../src/ripple_rpc/ripple_rpc.unity.cpp \
../../src/ripple_websocket/ripple_websocket.unity.cpp
LIBS += \
-lboost_date_time-mt\
-lboost_filesystem-mt \
-lboost_program_options-mt \
-lboost_regex-mt \
-lboost_system-mt \
-lboost_thread-mt \
-lboost_random-mt \
-lprotobuf \
-lssl \
-lrt

1
Builds/README.md Normal file
View File

@@ -0,0 +1 @@
[Please see the BUILD instructions here](../BUILD.md)

View File

@@ -1,7 +1,7 @@
#!/usr/bin/env python
# This file is part of rippled: https://github.com/ripple/rippled
# Copyright (c) 2012 - 2015 Ripple Labs Inc.
# Copyright (c) 2012 - 2017 Ripple Labs Inc.
#
# Permission to use, copy, modify, and/or distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
@@ -22,10 +22,10 @@ Invocation:
The build must succeed without shell aliases for this to work.
To pass flags to scons, put them at the very end of the command line, after
To pass flags to cmake, put them at the very end of the command line, after
the -- flag - like this:
./Builds/Test.py -- -j4 # Pass -j4 to scons.
./Builds/Test.py -- -j4 # Pass -j4 to cmake --build
Common problems:
@@ -34,11 +34,7 @@ Common problems:
2) OpenSSL not found. Solution: export OPENSSL_ROOT=[path to OpenSSL folder]
3) scons is an alias. Solution: Create a script named "scons" somewhere in
your $PATH (eg. ~/bin/scons will often work).
#!/bin/sh
python /C/Python27/Scripts/scons.py "${@}"
3) cmake is not found. Solution: Be sure cmake directory is on your $PATH
"""
from __future__ import absolute_import, division, print_function, unicode_literals
@@ -61,26 +57,36 @@ def powerset(iterable):
IS_WINDOWS = platform.system().lower() == 'windows'
IS_OS_X = platform.system().lower() == 'darwin'
if IS_WINDOWS or IS_OS_X:
ALL_TARGETS = [('debug',), ('release',)]
# CMake
if IS_WINDOWS:
CMAKE_UNITY_CONFIGS = ['Debug', 'Release']
CMAKE_NONUNITY_CONFIGS = ['Debug', 'Release']
else:
ALL_TARGETS = [(cc + "." + target,)
CMAKE_UNITY_CONFIGS = []
CMAKE_NONUNITY_CONFIGS = []
CMAKE_UNITY_COMBOS = { '' : [['rippled'], CMAKE_UNITY_CONFIGS],
'.nounity' : [['rippled'], CMAKE_NONUNITY_CONFIGS] }
if IS_WINDOWS:
CMAKE_DIR_TARGETS = { ('msvc' + unity,) : targets for unity, targets in
CMAKE_UNITY_COMBOS.items() }
elif IS_OS_X:
CMAKE_DIR_TARGETS = { (build + unity,) : targets
for build in ['debug', 'release']
for unity, targets in CMAKE_UNITY_COMBOS.items() }
else:
CMAKE_DIR_TARGETS = { (cc + "." + build + unity,) : targets
for cc in ['gcc', 'clang']
for target in ['debug', 'release', 'coverage', 'profile',
'debug.nounity', 'release.nounity', 'coverage.nounity', 'profile.nounity']]
for build in ['debug', 'release', 'coverage', 'profile']
for unity, targets in CMAKE_UNITY_COMBOS.items() }
# list of tuples of all possible options
if IS_WINDOWS or IS_OS_X:
ALL_OPTIONS = [tuple(x) for x in powerset(['--assert'])]
CMAKE_ALL_GENERATE_OPTIONS = [tuple(x) for x in powerset(['-GNinja', '-Dassert=true'])]
else:
ALL_OPTIONS = list(set(
[tuple(x) for x in powerset(['--ninja', '--static', '--assert', '--sanitize=address'])] +
[tuple(x) for x in powerset(['--ninja', '--static', '--assert', '--sanitize=thread'])]))
# list of tuples of all possible options + all possible targets
ALL_BUILDS = [options + target
for target in ALL_TARGETS
for options in ALL_OPTIONS]
CMAKE_ALL_GENERATE_OPTIONS = list(set(
[tuple(x) for x in powerset(['-GNinja', '-Dstatic=true', '-Dassert=true', '-Dsan=address'])] +
[tuple(x) for x in powerset(['-GNinja', '-Dstatic=true', '-Dassert=true', '-Dsan=thread'])]))
parser = argparse.ArgumentParser(
description='Test.py - run ripple tests'
@@ -117,6 +123,19 @@ parser.add_argument(
help='Add a prefix for unit tests',
)
parser.add_argument(
'--testjobs',
default='0',
type=int,
help='Run tests in parallel'
)
parser.add_argument(
'--ipv6',
action='store_true',
help='Use IPv6 localhost when running unit tests.',
)
parser.add_argument(
'--clean', '-c',
action='store_true',
@@ -130,15 +149,62 @@ parser.add_argument(
)
parser.add_argument(
'scons_args',
'--dir', '-d',
default=(),
nargs='*'
nargs='*',
help='Specify one or more CMake dir names. '
'Will also be used as -Dtarget=<dir> running cmake.'
)
parser.add_argument(
'--target',
default=(),
nargs='*',
help='Specify one or more CMake build targets. '
'Will be used as --target <target> running cmake --build.'
)
parser.add_argument(
'--config',
default=(),
nargs='*',
help='Specify one or more CMake build configs. '
'Will be used as --config <config> running cmake --build.'
)
parser.add_argument(
'--generator_option',
action='append',
help='Specify a CMake generator option. Repeat for multiple options. '
'Will be passed to the cmake generator. '
'Due to limits of the argument parser, arguments starting with \'-\' '
'must be attached to this option. e.g. --generator_option=-GNinja.')
parser.add_argument(
'--build_option',
action='append',
help='Specify a build option. Repeat for multiple options. '
'Will be passed to the build tool via cmake --build. '
'Due to limits of the argument parser, arguments starting with \'-\' '
'must be attached to this option. e.g. --build_option=-j8.')
parser.add_argument(
'extra_args',
default=(),
nargs='*',
help='Extra arguments are passed through to the tools'
)
ARGS = parser.parse_args()
def decodeString(line):
# Python 2 vs. Python 3
if isinstance(line, str):
return line
else:
return line.decode()
def shell(cmd, args=(), silent=False):
def shell(cmd, args=(), silent=False, cust_env=None):
""""Execute a shell command and return the output."""
silent = ARGS.silent or silent
verbose = not silent and ARGS.verbose
@@ -147,125 +213,189 @@ def shell(cmd, args=(), silent=False):
command = (cmd,) + args
# shell is needed in Windows to find executable in the path
process = subprocess.Popen(
command,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
env=cust_env,
shell=IS_WINDOWS)
lines = []
count = 0
for line in process.stdout:
# Python 2 vs. Python 3
if isinstance(line, str):
decoded = line
# readline returns '' at EOF
for line in iter(process.stdout.readline, ''):
if process.poll() is None:
decoded = decodeString(line)
lines.append(decoded)
if verbose:
print(decoded, end='')
elif not silent:
count += 1
if count >= 80:
print()
count = 0
else:
print('.', end='')
else:
decoded = line.decode()
lines.append(decoded)
if verbose:
print(decoded, end='')
elif not silent:
count += 1
if count >= 80:
print()
count = 0
else:
print('.', end='')
break
if not verbose and count:
print()
process.wait()
return process.returncode, lines
def get_cmake_dir(cmake_dir):
return os.path.join('build' , 'cmake' , cmake_dir)
def run_tests(args):
failed = []
if IS_WINDOWS:
binary_re = re.compile(r'build\\([^\\]+)\\rippled.exe')
else:
binary_re = re.compile(r'build/([^/]+)/rippled')
_, lines = shell('scons', ('-n', '--tree=derived',) + args, silent=True)
for line in lines:
match = binary_re.search(line)
if match:
executable, target = match.group(0, 1)
def run_cmake(directory, cmake_dir, args):
print('Generating build in', directory, 'with', *args or ('default options',))
old_dir = os.getcwd()
if not os.path.exists(directory):
os.makedirs(directory)
os.chdir(directory)
if IS_WINDOWS and not any(arg.startswith("-G") for arg in args) and not os.path.exists("CMakeCache.txt"):
if '--ninja' in args:
args += ( '-GNinja', )
else:
args += ( '-GVisual Studio 14 2015 Win64', )
# hack to extract cmake options/args from the legacy target format
if re.search('\.unity', cmake_dir):
args += ( '-Dunity=ON', )
if re.search('\.nounity', cmake_dir):
args += ( '-Dunity=OFF', )
if re.search('coverage', cmake_dir):
args += ( '-Dcoverage=ON', )
if re.search('profile', cmake_dir):
args += ( '-Dprofile=ON', )
if re.search('debug', cmake_dir):
args += ( '-DCMAKE_BUILD_TYPE=Debug', )
if re.search('release', cmake_dir):
args += ( '-DCMAKE_BUILD_TYPE=Release', )
m = re.search('gcc(-[^.]*)', cmake_dir)
if m:
args += ( '-DCMAKE_C_COMPILER=' + m.group(0),
'-DCMAKE_CXX_COMPILER=g++' + m.group(1), )
elif re.search('gcc', cmake_dir):
args += ( '-DCMAKE_C_COMPILER=gcc', '-DCMAKE_CXX_COMPILER=g++', )
m = re.search('clang(-[^.]*)', cmake_dir)
if m:
args += ( '-DCMAKE_C_COMPILER=' + m.group(0),
'-DCMAKE_CXX_COMPILER=clang++' + m.group(1), )
elif re.search('clang', cmake_dir):
args += ( '-DCMAKE_C_COMPILER=clang', '-DCMAKE_CXX_COMPILER=clang++', )
print('Unit tests for', target)
testflag = '--unittest'
quiet = ''
if ARGS.test:
testflag += ('=' + ARGS.test)
if ARGS.quiet:
quiet = '-q'
resultcode, lines = shell(executable, (testflag, quiet,))
args += ( os.path.join('..', '..', '..'), )
resultcode, lines = shell('cmake', args)
if resultcode:
if not ARGS.verbose:
print('ERROR:', *lines, sep='')
failed.append([target, 'unittest'])
if not ARGS.keep_going:
break
if resultcode:
print('Generating FAILED:')
if not ARGS.verbose:
print(*lines, sep='')
sys.exit(1)
return failed
os.chdir(old_dir)
def run_build(args=None):
print('Building:', *args or ('(default)',))
resultcode, lines = shell('scons', args)
def run_cmake_build(directory, target, config, args):
print('Building', target, config, 'in', directory, 'with', *args or ('default options',))
build_args=('--build', directory)
if target:
build_args += ('--target', target)
if config:
build_args += ('--config', config)
if args:
build_args += ('--',)
build_args += tuple(args)
resultcode, lines = shell('cmake', build_args)
if resultcode:
print('Build FAILED:')
if not ARGS.verbose:
print(*lines, sep='')
sys.exit(1)
if '--ninja' in args:
resultcode, lines = shell('ninja')
if resultcode:
print('Ninja build FAILED:')
if not ARGS.verbose:
print(*lines, sep='')
sys.exit(1)
def run_cmake_tests(directory, target, config):
failed = []
if IS_WINDOWS:
target += '.exe'
executable = os.path.join(directory, config if config else 'Debug', target)
if(not os.path.exists(executable)):
executable = os.path.join(directory, target)
print('Unit tests for', executable)
testflag = '--unittest'
quiet = ''
testjobs = ''
ipv6 = ''
if ARGS.test:
testflag += ('=' + ARGS.test)
if ARGS.quiet:
quiet = '-q'
if ARGS.ipv6:
ipv6 = '--unittest-ipv6'
if ARGS.testjobs:
testjobs = ('--unittest-jobs=' + str(ARGS.testjobs))
resultcode, lines = shell(executable, (testflag, quiet, testjobs, ipv6))
if resultcode:
if not ARGS.verbose:
print('ERROR:', *lines, sep='')
failed.append([target, 'unittest'])
return failed
def main():
if ARGS.all:
to_build = ALL_BUILDS
else:
to_build = [tuple(ARGS.scons_args)]
all_failed = []
for build in to_build:
args = ()
# additional arguments come first
for arg in list(ARGS.scons_args):
if arg not in build:
args += (arg,)
args += build
run_build(args)
failed = run_tests(args)
if failed:
print('FAILED:', *(':'.join(f) for f in failed))
if not ARGS.keep_going:
sys.exit(1)
else:
all_failed.extend([','.join(build), ':'.join(f)]
for f in failed)
if ARGS.all:
build_dir_targets = CMAKE_DIR_TARGETS
generator_options = CMAKE_ALL_GENERATE_OPTIONS
else:
build_dir_targets = { tuple(ARGS.dir) : [ARGS.target, ARGS.config] }
if ARGS.generator_option:
generator_options = [tuple(ARGS.generator_option)]
else:
print('Success')
generator_options = [tuple()]
if ARGS.clean:
shutil.rmtree('build')
if '--ninja' in args:
os.remove('build.ninja')
os.remove('.ninja_deps')
os.remove('.ninja_log')
if not build_dir_targets:
# Let CMake choose the build tool.
build_dir_targets = { () : [] }
if ARGS.build_option:
ARGS.build_option = ARGS.build_option + list(ARGS.extra_args)
else:
ARGS.build_option = list(ARGS.extra_args)
for args in generator_options:
for build_dirs, (build_targets, build_configs) in build_dir_targets.items():
if not build_dirs:
build_dirs = ('default',)
if not build_targets:
build_targets = ('rippled',)
if not build_configs:
build_configs = ('',)
for cmake_dir in build_dirs:
cmake_full_dir = get_cmake_dir(cmake_dir)
run_cmake(cmake_full_dir, cmake_dir, args)
for target in build_targets:
for config in build_configs:
run_cmake_build(cmake_full_dir, target, config, ARGS.build_option)
failed = run_cmake_tests(cmake_full_dir, target, config)
if failed:
print('FAILED:', *(':'.join(f) for f in failed))
if not ARGS.keep_going:
sys.exit(1)
else:
all_failed.extend([decodeString(cmake_dir +
"." + target + "." + config), ':'.join(f)]
for f in failed)
else:
print('Success')
if ARGS.clean:
shutil.rmtree(cmake_full_dir)
if all_failed:
if len(to_build) > 1:
if len(all_failed) > 1:
print()
print('FAILED:', *(':'.join(f) for f in all_failed))
sys.exit(1)

View File

@@ -1,82 +0,0 @@
#!/usr/bin/env bash
#
# This scripts installs boost and protobuf built with clang. This is needed on
# ubuntu 15.10 when building with clang
# It will build these in a 'clang' subdirectory that it creates below the directory
# this script is run from. If a clang directory already exists the script will refuse
# to run.
if hash lsb_release 2>/dev/null; then
if [ $(lsb_release -si) == "Ubuntu" ]; then
ubuntu_release=$(lsb_release -sr)
fi
fi
if [ -z "${ubuntu_release}" ]; then
echo "System not supported"
exit 1
fi
if ! hash clang 2>/dev/null; then
clang_version=3.7
if [ ${ubuntu_release} == "16.04" ]; then
clang_version=3.8
fi
sudo apt-get -y install clang-${clang_version}
update-alternatives --install /usr/bin/clang clang /usr/bin/clang-${clang_version} 99 clang++
hash -r
if ! hash clang 2>/dev/null; then
echo "Please install clang"
exit 1
fi
fi
if [ ${ubuntu_release} != "16.04" ] && [ ${ubuntu_release} != "15.10" ]; then
echo "clang specific boost and protobuf not needed"
exit 0
fi
if [ -d clang ]; then
echo "clang directory already exists. Cowardly refusing to run"
exit 1
fi
if ! hash wget 2>/dev/null; then
sudo apt-get -y install wget
hash -r
if ! hash wget 2>/dev/null; then
echo "Please install wget"
exit 1
fi
fi
num_procs=$(lscpu -p | grep -v '^#' | sort -u -t, -k 2,4 | wc -l) # pysical cores
mkdir clang
pushd clang > /dev/null
# Install protobuf
pb=protobuf-2.6.1
pb_tar=${pb}.tar.gz
wget -O ${pb_tar} https://github.com/google/protobuf/releases/download/v2.6.1/${pb_tar}
tar xf ${pb_tar}
rm ${pb_tar}
pushd ${pb} > /dev/null
./configure CC=clang CXX=clang++ CXXFLAGS='-std=c++14 -O3 -g'
make -j${num_procs}
popd > /dev/null
# Install boost
boost_ver=1.60.0
bd=boost_${boost_ver//./_}
bd_tar=${bd}.tar.gz
wget -O ${bd_tar} http://sourceforge.net/projects/boost/files/boost/${boost_ver}/${bd_tar}
tar xf ${bd_tar}
rm ${bd_tar}
pushd ${bd} > /dev/null
./bootstrap.sh
./b2 toolset=clang -j${num_procs}
popd > /dev/null
popd > /dev/null

View File

@@ -1,39 +0,0 @@
#!/usr/bin/env bash
#
# This script builds boost with the correct ABI flags for ubuntu
#
version=63
patch=0
if hash lsb_release 2>/dev/null; then
if [ $(lsb_release -si) == "Ubuntu" ]; then
ubuntu_release=$(lsb_release -sr)
fi
fi
if [ -z "${ubuntu_release}" ]; then
echo "System not supported"
exit 1
fi
extra_defines=""
if (( $(bc <<< "${ubuntu_release} < 15.1") )); then
extra_defines="define=_GLIBCXX_USE_CXX11_ABI=0"
fi
num_procs=$(lscpu -p | grep -v '^#' | sort -u -t, -k 2,4 | wc -l) # pysical cores
printf "\nBuild command will be: ./b2 -j${num_procs} ${extra_defines}\n\n"
boost_dir="boost_1_${version}_${patch}"
boost_tag="boost-1.${version}.${patch}"
git clone -b "${boost_tag}" --recursive https://github.com/boostorg/boost.git "${boost_dir}"
cd ${boost_dir}
git checkout --force ${boost_tag}
git submodule foreach git checkout --force ${boost_tag}
./bootstrap.sh
./b2 headers
./b2 -j${num_procs} ${extra_defines}
echo "Build command was: ./b2 -j${num_procs} ${extra_defines}"
echo "Don't forget to set BOOST_ROOT!"

View File

@@ -1,59 +0,0 @@
#!/usr/bin/env bash
#
# This scripts installs the dependencies needed by rippled. It should be run
# with sudo. For ubuntu < 15.10, it installs gcc 5 as the default compiler. gcc
# 5 is ABI incompatable with gcc 4. If needed, the following will switch back to
# gcc-4: `sudo update-alternatives --config gcc` and choosing the gcc-4
# option.
#
if hash lsb_release 2>/dev/null; then
if [ $(lsb_release -si) == "Ubuntu" ]; then
ubuntu_release=$(lsb_release -sr)
fi
fi
if [ -z "${ubuntu_release}" ]; then
echo "System not supported"
exit 1
fi
if [ ${ubuntu_release} == "12.04" ]; then
apt-get install python-software-properties
add-apt-repository ppa:afrank/boost
add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get update
apt-get -y upgrade
apt-get -y install curl git scons ctags pkg-config protobuf-compiler libprotobuf-dev libssl-dev python-software-properties boost1.57-all-dev g++-5 g++-4.9
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 99 --slave /usr/bin/g++ g++ /usr/bin/g++-5
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 99 --slave /usr/bin/g++ g++ /usr/bin/g++-4.9
exit 0
fi
if [ ${ubuntu_release} == "14.04" ] || [ ${ubuntu_release} == "15.04" ]; then
apt-get install python-software-properties
echo "deb [arch=amd64] https://mirrors.ripple.com/ubuntu/ trusty stable contrib" | sudo tee /etc/apt/sources.list.d/ripple.list
wget -O- -q https://mirrors.ripple.com/mirrors.ripple.com.gpg.key | sudo apt-key add -
add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get update
apt-get -y upgrade
apt-get -y install curl git scons ctags pkg-config protobuf-compiler libprotobuf-dev libssl-dev python-software-properties boost-all-dev g++-5 g++-4.9
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 99 --slave /usr/bin/g++ g++ /usr/bin/g++-5
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 99 --slave /usr/bin/g++ g++ /usr/bin/g++-4.9
exit 0
fi
# Test if 0th parameter has a version number greater than or equal to the 1st param
function version_check() { test "$(printf '%s\n' "$@" | sort -V | tail -n 1)" == "$1"; }
# this should work for versions greater than 15.10
if version_check ${ubuntu_release} 15.10; then
apt-get update
apt-get -y upgrade
apt-get -y install python-software-properties curl git scons ctags pkg-config protobuf-compiler libprotobuf-dev libssl-dev python-software-properties libboost-all-dev
exit 0
fi
echo "System not supported"
exit 1

View File

@@ -1,4 +0,0 @@
RippleD.vcxproj -text
RippleD.vcxproj.filters -text

View File

@@ -1,255 +0,0 @@
# Visual Studio 2015 Build Instructions
## Important
We do not recommend Windows for rippled production use at this time. Currently, the Ubuntu
platform has received the highest level of quality assurance, testing, and support.
Additionally, 32-bit Windows versions are not supported.
## Prerequisites
To clone the source code repository, create branches for inspection or modification,
build rippled under Visual Studio, and run the unit tests you will need these
software components:
* [Visual Studio 2015](README.md#install-visual-studio-2015)
* [Git for Windows](README.md#install-git-for-windows)
* [Google Protocol Buffers Compiler](README.md#install-google-protocol-buffers-compiler)
* (Optional) [Python and Scons](README.md#optional-install-python-and-scons)
* [OpenSSL Library](README.md#install-openssl)
* [Boost library](README.md#build-boost)
## Install Software
### Install Visual Studio 2015
If not already installed on your system, download your choice of installer from the
[Visual Studio 2015 Download](https://www.visualstudio.com/downloads/download-visual-studio-vs)
page, run the installer, and follow the directions. You may need to choose a "Custom"
installation and ensure that "Visual C++" is selected under "Programming Languages".
Any version of Visual Studio 2015 may be used to build rippled.
The **Visual Studio 2015 Community** edition is available free of charge (see
[the product page](https://www.visualstudio.com/products/visual-studio-community-vs)
for licensing details), while paid editions may be used for an free initial trial period.
### Install Git for Windows
Git is a distributed revision control system. The Windows version also provides the
bash shell and many Windows versions of Unix commands. While there are other
varieties of Git (such as TortoiseGit, which has a native Windows interface and
integrates with the Explorer shell), we recommend installing
[Git for Windows](https://git-scm.com/) since
it provides a Unix-like command line environment useful for running shell scripts.
Use of the bash shell under Windows is mandatory for running the unit tests.
* NOTE: To gain full featured access to the
[git-subtree](https://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/)
functionality used in the rippled repository we suggest Git version 2.6.2 or later.
### Install Google Protocol Buffers Compiler
Building rippled requires **protoc.exe** version 2.5.1 or later. At your option you
may build it yourself from the sources in the
[Google Protocol Buffers](https://github.com/google/protobuf) repository,
or you may download a
[protoc.exe](https://ripple.github.io/Downloads/protoc/2.5.1/protoc.exe)
([alternate link](https://github.com/ripple/Downloads/raw/gh-pages/protoc/2.5.1/protoc.exe))
precompiled Windows executable from the
[Ripple Organization](https://github.com/ripple).
Either way, once you have the required version of **protoc.exe**, copy it into
a folder in your command line `%PATH%`.
* **NOTE:** If you use an older version of the compiler, the build will
fail with errors related to a mismatch of the version of protocol
buffer headers versus the compiler.
### (Optional) Install Python and Scons
[Python](https://www.python.org/downloads/) and
[Scons](http://scons.org/download.php) are not required to build
rippled with Visual Studio, but can be used to build from the
command line and in scripts, and are required to properly update
the `RippleD.vcxproj` file.
If you wish to build with scons, a version after 2.3.5 is required
for Visual Studio 2015 support.
## Configure Dependencies
### Install OpenSSL
[Download OpenSSL.](http://slproweb.com/products/Win32OpenSSL.html)
There will be four variants available:
1. 64-bit. Use this if you are running 64-bit windows. As of this writing, the link is called: "Win64 OpenSSL v1.0.2j".
2. 64-bit light - Don't use this. It is missing files needed to build rippled. As of this writing, the link is called: "Win64 OpenSSL v1.0.2j Light"
Run the installer, and choose an appropriate location for your OpenSSL
installation. In this guide we use **C:\lib\OpenSSL-Win64** as the
destination location.
You may be informed on running the installer that "Visual C++ 2008
Redistributables" must first be installed first. If so, download it
from the [same page](http://slproweb.com/products/Win32OpenSSL.html),
again making sure to get the correct 32-/64-bit variant.
* NOTE: Since rippled links statically to OpenSSL, it does not matter
where the OpenSSL .DLL files are placed, or what version they are.
rippled does not use or require any external .DLL files to run
other than the standard operating system ones.
### Build Boost
After [downloading boost](http://www.boost.org/users/download/) and
unpacking it, open a **Developer Command Prompt** for
Visual Studio, change to the directory containing boost, then
bootstrap the build tools:
(As of this writing, the most recent version of boost is 1.62.0, which
will unpack into a directory named `boost_1_62_0`. For higher versions
of boost, adjust the directories provided in these examples as
appropriate.)
```powershell
cd C:\lib\boost_1_62_0
bootstrap
```
The rippled application is linked statically to the standard runtimes and external
dependencies on Windows, to ensure that the behavior of the executable is not
affected by changes in outside files. Therefore, it is necessary to build the
required boost static libraries using this command:
```powershell
bjam --toolset=msvc-14.0 address-model=64 architecture=x86 link=static threading=multi runtime-link=shared,static stage --stagedir=stage64
```
Building the boost libraries may take considerable time. When the build process
is completed, take note of both the reported compiler include paths and linker
library paths as they will be required later.
* NOTE: If older versions of Visual Studio are also installed, the build may fail.
If this happens, make sure that only Visual Studio 2015 is installed. Due to
defects in the uninstallation procedures of these Microsoft products, it may
be necessary to start with a fresh install of the operating system with only
the necessary development environment components installed to have a successful build.
### Clone the rippled repository
If you are familiar with cloning github repositories, just follow your normal process
and clone `git@github.com:ripple/rippled.git`. Otherwise follow this section for instructions.
1. If you don't have a github account, sign up for one at
[github.com](https://github.com/).
2. Make sure you have Github ssh keys. For help see
[generating-ssh-keys](https://help.github.com/articles/generating-ssh-keys).
Open the "Git Bash" shell that was installed with "Git for Windows" in the
step above. Navigate to the directory where you want to clone rippled (git
bash uses `/c` for windows's `C:` and forward slash where windows uses
backslash, so `C:\Users\joe\projs` would be `/c/Users/joe/projs` in git bash).
Now clone the repository and optionally switch to the *master* branch.
Type the following at the bash prompt:
```powershell
git clone git@github.com:ripple/rippled.git
cd rippled
git checkout master
```
* If you receive an error about not having the "correct access rights"
make sure you have Github ssh keys, as described above.
### Configure Library Paths
Open the solution file located at **Builds/Visual Studio 2015/ripple.sln**
and select the "View->Property Manager" to bring up the Property Manager.
Expand the *debug | x64* section and
double click the *Microsoft.Cpp.x64.user* property sheet to bring up the
*Property Pages* dialog. These are global properties applied to all
64-bit build targets:
![Visual Studio 2015 Global Properties](images/VS2015x64Properties.png)
Go to *C/C++, General, Additional Include Directories* and add the
location of the boost installation:
![Visual Studio 2015 Include Directories](images/VS2015x64IncludeDirs.png)
Then, go to *Linker, General, Additional Library Directories* and add
the location of the compiled boost libraries reported at the completion
of building the boost libraries:
![Visual Studio 2015 Library Directories](images/VS2015x64LibraryDirs.png)
Follow the same procedure for adding the `Additional Include Directories`
and `Additional Library Directories` required for OpenSSL. In our example
these directories are **C:\lib\OpenSSL-Win64\include** and
**C:\lib\OpenSSL-Win64\lib** respectively.
# Setup Environment
## Create a working directory for rippled.cfg
The rippled server uses the [Rippled.cfg](https://wiki.ripple.com/Rippled.cfg)
file to read its configuration parameters. This section describes setting up
a directory to hold the config file. The next sections describe how to tell
the rippled server where that file is.
1. Create a directory to hold the configuration file. In this example, the
ripple config directory was created in `C:\Users\joe\ripple\config`.
2. Copy the example config file located in `doc\rippled-example.cfg` to the
new directory and rename it "rippled.cfg".
3. Read the rippled.cfg file and edit as appropriate.
## Change the Visual Studio Projects Debugging Properties
1. If not already open, open the solution file located at **Builds/Visual Studio 2015/Ripple.sln**
2. Select the correct solution platform in the solution platform dropdown (either *x64*
or *Win32* depending on machine type).
3. Select the "Project->Properties" menu item to bring up RippleD's Properties Pages
4. In "Configuration Properties" select "Debugging".
5. In the upper-left Configurations drop down, select "All Configurations".
6. In "Debugger to Launch" select "Local Windows Debugger".
### Tell rippled where to find the configuration file.
The `--conf` command-line switch to tell rippled where to find this file.
In the "Command Arguments" field in the properties dialog (that you opened
in the above section), add: `--conf="C:/Users/joe/ripple/config/rippled.cfg"`
(of course replacing that path with the path you set up above).
![Visual Studio 2013 Command Args Prop Page](images/VSCommandArgsPropPage.png)
### Set the _NO_DEBUG_HEAP Environment Variable
Rippled can run very slowly in the debugger when using the Windows Debug Heap.
Set the `_NO_DEBUG_HEAP` environment variable to one to disable the debug heap.
In the "Environment" field (that you opened in the above section), add:
`_NO_DEBUG_HEAP=1`
![Visual Studio 2013 No Debug Heap Prop Page](images/NoDebugHeapPropPage.png)
# Build
After these steps are complete, rippled should be ready to build. Simply
set rippled as the startup project by right clicking on it in the
Visual Studio Solution Explorer, choose **Set as Startup Project**,
and then choose the **Build->Build Solution** menu item.
# Unit Tests (Recommended)
The rippled unit tests are written in C++ and are part
of the rippled executable.
From a Windows console, run the unit tests:
```
./build/msvc.debug/rippled.exe --unittest
```
Substitute the correct path to the executable to test different builds.

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 19 KiB

View File

@@ -1,36 +0,0 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 14
VisualStudioVersion = 14.0.25123.0
MinimumVisualStudioVersion = 10.0.40219.1
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "RippleD", "RippleD.vcxproj", "{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
debug.classic|x64 = debug.classic|x64
debug.classic|x86 = debug.classic|x86
debug|x64 = debug|x64
debug|x86 = debug|x86
release.classic|x64 = release.classic|x64
release.classic|x86 = release.classic|x86
release|x64 = release|x64
release|x86 = release|x86
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.debug.classic|x64.ActiveCfg = debug.classic|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.debug.classic|x64.Build.0 = debug.classic|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.debug.classic|x86.ActiveCfg = debug.classic|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.debug|x64.ActiveCfg = debug|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.debug|x64.Build.0 = debug|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.debug|x86.ActiveCfg = debug|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.release.classic|x64.ActiveCfg = release.classic|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.release.classic|x64.Build.0 = release.classic|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.release.classic|x86.ActiveCfg = release.classic|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.release|x64.ActiveCfg = release|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.release|x64.Build.0 = release|x64
{26B7D9AC-1A80-8EF8-6703-D061F1BECB75}.release|x86.ActiveCfg = release|x64
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
EndGlobal

View File

@@ -1,177 +0,0 @@
# macos Build Instructions
## Important
We don't recommend OS X for rippled production use at this time. Currently, the
Ubuntu platform has received the highest level of quality assurance and
testing.
## Prerequisites
You'll need OSX 10.8 or later
To clone the source code repository, create branches for inspection or
modification, build rippled using clang, and run the system tests you will need
these software components:
* [XCode](https://developer.apple.com/xcode/)
* [Homebrew](http://brew.sh/)
* [Git](http://git-scm.com/)
* [Scons](http://www.scons.org/)
## Install Software
### Install XCode
If not already installed on your system, download and install XCode using the
appstore or by using [this link](https://developer.apple.com/xcode/).
For more info, see "Step 1: Download and Install the Command Line Tools"
[here](http://www.moncefbelyamani.com/how-to-install-xcode-homebrew-git-rvm-ruby-on-mac)
The command line tools can be installed through the terminal with the command:
```
xcode-select --install
```
### Install Homebrew
> "[Homebrew](http://brew.sh/) installs the stuff you need that Apple didnt."
Open a terminal and type:
```
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
```
For more info, see "Step 3: Install Homebrew"
[here](http://www.moncefbelyamani.com/how-to-install-xcode-homebrew-git-rvm-ruby-on-mac)
### Install Git
```
brew update brew install git
```
For more info, see "Step 4: Install Git"
[here](http://www.moncefbelyamani.com/how-to-install-xcode-homebrew-git-rvm-ruby-on-mac)
**NOTE**: To gain full featured access to the
[git-subtree](http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/)
functionality used in the rippled repository, we suggest Git version 1.8.3.2 or
later.
### Install Scons
Requires version 2.3.0 or later
```
brew install scons
```
`brew` will generally install the latest stable version of any package, which
will satisfy the scons minimum version requirement for rippled.
### Install Package Config
```
brew install pkg-config
```
## Install/Build/Configure Dependencies
### Build Google Protocol Buffers Compiler
Building rippled on osx requires `protoc` version 2.5.x or 2.6.x (later versions
do not work with rippled at this time).
Download [this](https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.bz2)
We want to compile protocol buffers with clang/libc++:
```
tar xfvj protobuf-2.6.1.tar.bz2
cd protobuf-2.6.1
./configure CC=clang CXX=clang++ CXXFLAGS='-std=c++11 -stdlib=libc++ -O3 -g' LDFLAGS='-stdlib=libc++' LIBS="-lc++ -lc++abi"
make -j 4
sudo make install
```
If you have installed `protobuf` via brew - either directly or indirectly as a
dependency of some other package - this is likely to conflict with our specific
version requirements. The simplest way to avoid conflicts is to uninstall it.
`brew ls --versions protobuf` will list any versions of protobuf
you currently have installed.
### Install OpenSSL
```
brew install openssl
```
### Build Boost
We want to compile boost with clang/libc++
Download [a release](https://sourceforge.net/projects/boost/files/boost/1.61.0/boost_1_61_0.tar.bz2)
Extract it to a folder, making note of where, open a terminal, then:
```
./bootstrap.sh ./b2 toolset=clang threading=multi runtime-link=static link=static cxxflags="-stdlib=libc++" linkflags="-stdlib=libc++" address-model=64
```
Create an environment variable `BOOST_ROOT` in one of your `rc` files, pointing
to the root of the extracted directory.
### Clone the rippled repository
From the terminal
```
git clone git@github.com:ripple/rippled.git
cd rippled
```
Choose the master branch or one of the tagged releases listed on
[GitHub](https://github.com/ripple/rippled/releases GitHub).
```
git checkout master
```
or to test the latest release candidate, choose the `release` branch.
```
git checkout release
```
### Configure Library Paths
If you didn't persistently set the `BOOST_ROOT` environment variable to the
root of the extracted directory above, then you should set it temporarily.
For example, assuming your username were `Abigail` and you extracted Boost
1.61.0 in `/Users/Abigail/Downloads/boost_1_61_0`, you would do for any
shell in which you want to build:
```
export BOOST_ROOT=/Users/Abigail/Downloads/boost_1_61_0
```
## Build
```
scons
```
See: [here](https://ripple.com/wiki/Rippled_build_instructions#Building)
## Unit Tests (Recommended)
rippled builds a set of unit tests into the server executable. To run these unit
tests after building, pass the `--unittest` option to the compiled `rippled`
executable. The executable will exit after running the unit tests.

5
Builds/build_all.sh Normal file → Executable file
View File

@@ -2,5 +2,6 @@
num_procs=$(lscpu -p | grep -v '^#' | sort -u -t, -k 2,4 | wc -l) # number of physical cores
cd ..
./Builds/Test.py -a -c -- -j${num_procs}
path=$(cd $(dirname $0) && pwd)
cd $(dirname $path)
${path}/Test.py -a -c --testjobs=${num_procs} -- -j${num_procs}

View File

@@ -0,0 +1,31 @@
# rippled Packaging and Containers
This folder contains docker container definitions and configuration
files to support building rpm and deb packages of rippled. The container
definitions include some additional software/packages that are used
for general build/test CI workflows of rippled but are not explicitly
needed for the package building workflow.
## CMake Targets
If you have docker installed on your local system, then the main
CMake file will enable several targets related to building packages:
`rpm_container`, `rpm`, `dpkg_container`, and `dpkg`. The package targets
depend on the container targets and will trigger a build of those first.
The container builds can take several dozen minutes to complete (depending
on hardware specs), so quick build cycles are not possible currently. As
such, these targets are often best suited to CI/automated build systems.
The package build can be invoked like any other cmake target from the
rippled root folder:
```
mkdir -p build/pkg && cd build/pkg
cmake -Dpackages_only=ON ../..
cmake --build . --target rpm
```
Upon successful completion, the generated package files will be in
the `build/pkg/packages` directory. For deb packages, simply replace
`rpm` with `dpkg` in the build command above.

View File

@@ -0,0 +1,26 @@
FROM rippleci/centos:7
ARG GIT_COMMIT=unknown
ARG CI_USE=false
LABEL git-commit=$GIT_COMMIT
COPY centos-builder/centos_setup.sh /tmp/
COPY shared/install_cmake.sh /tmp/
RUN chmod +x /tmp/centos_setup.sh && \
chmod +x /tmp/install_cmake.sh
RUN /tmp/centos_setup.sh
RUN /tmp/install_cmake.sh 3.16.3 /opt/local/cmake-3.16
RUN ln -s /opt/local/cmake-3.16 /opt/local/cmake
ENV PATH="/opt/local/cmake/bin:$PATH"
# TODO: Install latest CMake for testing
RUN if [ "${CI_USE}" = true ] ; then /tmp/install_cmake.sh 3.16.3 /opt/local/cmake-3.16; fi
RUN mkdir -m 777 -p /opt/rippled_bld/pkg
WORKDIR /opt/rippled_bld/pkg
RUN mkdir -m 777 ./rpmbuild
RUN mkdir -m 777 ./rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
COPY packaging/rpm/build_rpm.sh ./
CMD ./build_rpm.sh

View File

@@ -0,0 +1,22 @@
#!/usr/bin/env bash
set -ex
source /etc/os-release
yum -y upgrade
yum -y update
yum -y install epel-release centos-release-scl
yum -y install \
wget curl time gcc-c++ yum-utils autoconf automake pkgconfig libtool \
libstdc++-static rpm-build gnupg which make cmake \
devtoolset-11 devtoolset-11-gdb devtoolset-11-binutils devtoolset-11-libstdc++-devel \
devtoolset-11-libasan-devel devtoolset-11-libtsan-devel devtoolset-11-libubsan-devel devtoolset-11-liblsan-devel \
flex flex-devel bison bison-devel parallel \
ncurses ncurses-devel ncurses-libs graphviz graphviz-devel \
lzip p7zip bzip2 bzip2-devel lzma-sdk lzma-sdk-devel xz-devel \
zlib zlib-devel zlib-static texinfo openssl openssl-static \
jemalloc jemalloc-devel \
libicu-devel htop \
rh-python38 \
ninja-build git svn \
swig perl-Digest-MD5

View File

@@ -0,0 +1,28 @@
#!/usr/bin/env sh
set -ex
pkgtype=$1
if [ "${pkgtype}" = "rpm" ] ; then
container_name="${RPM_CONTAINER_NAME}"
elif [ "${pkgtype}" = "dpkg" ] ; then
container_name="${DPKG_CONTAINER_NAME}"
else
echo "invalid package type"
exit 1
fi
if docker pull "${ARTIFACTORY_HUB}/${container_name}:latest_${CI_COMMIT_REF_SLUG}"; then
echo "found container for latest - using as cache."
docker tag \
"${ARTIFACTORY_HUB}/${container_name}:latest_${CI_COMMIT_REF_SLUG}" \
"${container_name}:latest_${CI_COMMIT_REF_SLUG}"
CMAKE_EXTRA="-D${pkgtype}_cache_from=${container_name}:latest_${CI_COMMIT_REF_SLUG}"
fi
cmake --version
test -d build && rm -rf build
mkdir -p build/container && cd build/container
eval time \
cmake -Dpackages_only=ON -DCMAKE_VERBOSE_MAKEFILE=ON ${CMAKE_EXTRA} \
-G Ninja ../..
time cmake --build . --target "${pkgtype}_container" -- -v

View File

@@ -0,0 +1,28 @@
#!/usr/bin/env sh
set -ex
pkgtype=$1
if [ "${pkgtype}" = "rpm" ] ; then
container_name="${RPM_CONTAINER_FULLNAME}"
container_tag="${RPM_CONTAINER_TAG}"
elif [ "${pkgtype}" = "dpkg" ] ; then
container_name="${DPKG_CONTAINER_FULLNAME}"
container_tag="${DPKG_CONTAINER_TAG}"
else
echo "invalid package type"
exit 1
fi
time docker pull "${ARTIFACTORY_HUB}/${container_name}"
docker tag \
"${ARTIFACTORY_HUB}/${container_name}" \
"${container_name}"
docker images
test -d build && rm -rf build
mkdir -p build/${pkgtype} && cd build/${pkgtype}
time cmake \
-Dpackages_only=ON \
-Dcontainer_label="${container_tag}" \
-Dhave_package_container=ON \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-Dunity=OFF \
-G Ninja ../..
time cmake --build . --target ${pkgtype} -- -v

View File

@@ -0,0 +1,15 @@
#!/usr/bin/env sh
set -e
# used as a before/setup script for docker steps in gitlab-ci
# expects to be run in standard alpine/dind image
echo $(nproc)
docker login -u rippled \
-p ${ARTIFACTORY_DEPLOY_KEY_RIPPLED} ${ARTIFACTORY_HUB}
apk add --update py-pip
apk add \
bash util-linux coreutils binutils grep \
make ninja cmake build-base gcc g++ abuild git \
python3 python3-dev
pip3 install awscli --break-system-packages
# list curdir contents to build log:
ls -la

View File

@@ -0,0 +1,16 @@
#!/usr/bin/env sh
case ${CI_COMMIT_REF_NAME} in
develop)
export COMPONENT="nightly"
;;
release)
export COMPONENT="unstable"
;;
master)
export COMPONENT="stable"
;;
*)
export COMPONENT="_unknown_"
;;
esac

View File

@@ -0,0 +1,771 @@
#########################################################################
## ##
## gitlab CI defintition for rippled build containers and distro ##
## packages (rpm and dpkg). ##
## ##
#########################################################################
# NOTE: these are sensible defaults for Ripple pipelines. These
# can be overridden by project or group variables as needed.
variables:
# these containers are built manually using the rippled
# cmake build (container targets) and tagged/pushed so they
# can be used here
RPM_CONTAINER_TAG: "2023-02-13"
RPM_CONTAINER_NAME: "rippled-rpm-builder"
RPM_CONTAINER_FULLNAME: "${RPM_CONTAINER_NAME}:${RPM_CONTAINER_TAG}"
DPKG_CONTAINER_TAG: "2023-07-31"
DPKG_CONTAINER_NAME: "rippled-dpkg-builder"
DPKG_CONTAINER_FULLNAME: "${DPKG_CONTAINER_NAME}:${DPKG_CONTAINER_TAG}"
ARTIFACTORY_HOST: "artifactory.ops.ripple.com"
ARTIFACTORY_HUB: "${ARTIFACTORY_HOST}:6555"
GIT_SIGN_PUBKEYS_URL: "https://gitlab.ops.ripple.com/xrpledger/rippled-packages/snippets/49/raw"
PUBLIC_REPO_ROOT: "https://repos.ripple.com/repos"
# also need to define this variable ONLY for the primary
# build/publish pipeline on the mainline repo:
# IS_PRIMARY_REPO = "true"
stages:
- build_packages
- sign_packages
- smoketest
- verify_sig
- tag_images
- push_to_test
- verify_from_test
- wait_approval_prod
- push_to_prod
- verify_from_prod
- get_final_hashes
- build_containers
.dind_template: &dind_param
before_script:
- . ./Builds/containers/gitlab-ci/docker_alpine_setup.sh
variables:
docker_driver: overlay2
DOCKER_TLS_CERTDIR: ""
image:
name: artifactory.ops.ripple.com/docker:latest
services:
# workaround for TLS issues - consider going back
# back to unversioned `dind` when issues are resolved
- name: artifactory.ops.ripple.com/docker:stable-dind
alias: docker
tags:
- 4xlarge
.only_primary_template: &only_primary
only:
refs:
- /^(master|release|develop)$/
variables:
- $IS_PRIMARY_REPO == "true"
.smoketest_local_template: &run_local_smoketest
tags:
- xlarge
script:
- . ./Builds/containers/gitlab-ci/smoketest.sh local
.smoketest_repo_template: &run_repo_smoketest
tags:
- xlarge
script:
- . ./Builds/containers/gitlab-ci/smoketest.sh repo
#########################################################################
## ##
## stage: build_packages ##
## ##
## build packages using containers from previous stage. ##
## ##
#########################################################################
rpm_build:
timeout: "1h 30m"
stage: build_packages
<<: *dind_param
artifacts:
paths:
- build/rpm/packages/
script:
- . ./Builds/containers/gitlab-ci/build_package.sh rpm
dpkg_build:
timeout: "1h 30m"
stage: build_packages
<<: *dind_param
artifacts:
paths:
- build/dpkg/packages/
script:
- . ./Builds/containers/gitlab-ci/build_package.sh dpkg
#########################################################################
## ##
## stage: sign_packages ##
## ##
## build packages using containers from previous stage. ##
## ##
#########################################################################
rpm_sign:
stage: sign_packages
dependencies:
- rpm_build
image:
name: artifactory.ops.ripple.com/centos:7
<<: *only_primary
before_script:
- |
# Make sure GnuPG is installed
yum -y install gnupg rpm-sign
# checking GPG signing support
if [ -n "$GPG_KEY_B64" ]; then
echo "$GPG_KEY_B64"| base64 -d | gpg --batch --no-tty --allow-secret-key-import --import -
unset GPG_KEY_B64
export GPG_PASSPHRASE=$(echo $GPG_KEY_PASS_B64 | base64 -di)
unset GPG_KEY_PASS_B64
export GPG_KEYID=$(gpg --with-colon --list-secret-keys | head -n1 | cut -d : -f 5)
else
echo -e "\033[0;31m****** GPG signing disabled ******\033[0m"
exit 1
fi
artifacts:
paths:
- build/rpm/packages/
script:
- ls -alh build/rpm/packages
- . ./Builds/containers/gitlab-ci/sign_package.sh rpm
dpkg_sign:
stage: sign_packages
dependencies:
- dpkg_build
image:
name: artifactory.ops.ripple.com/ubuntu:18.04
<<: *only_primary
before_script:
- |
# make sure we have GnuPG
apt update
apt install -y gpg dpkg-sig
# checking GPG signing support
if [ -n "$GPG_KEY_B64" ]; then
echo "$GPG_KEY_B64"| base64 -d | gpg --batch --no-tty --allow-secret-key-import --import -
unset GPG_KEY_B64
export GPG_PASSPHRASE=$(echo $GPG_KEY_PASS_B64 | base64 -di)
unset GPG_KEY_PASS_B64
export GPG_KEYID=$(gpg --with-colon --list-secret-keys | head -n1 | cut -d : -f 5)
else
echo -e "\033[0;31m****** GPG signing disabled ******\033[0m"
exit 1
fi
artifacts:
paths:
- build/dpkg/packages/
script:
- ls -alh build/dpkg/packages
- . ./Builds/containers/gitlab-ci/sign_package.sh dpkg
#########################################################################
## ##
## stage: smoketest ##
## ##
## install unsigned packages from previous step and run unit tests. ##
## ##
#########################################################################
centos_7_smoketest:
stage: smoketest
dependencies:
- rpm_build
image:
name: artifactory.ops.ripple.com/centos:7
<<: *run_local_smoketest
rocky_8_smoketest:
stage: smoketest
dependencies:
- rpm_build
image:
name: artifactory.ops.ripple.com/rockylinux/rockylinux:8
<<: *run_local_smoketest
rocky_9_smoketest:
stage: smoketest
dependencies:
- rpm_build
image:
name: artifactory.ops.ripple.com/rockylinux/rockylinux:9
<<: *run_local_smoketest
alma_8_smoketest:
stage: smoketest
dependencies:
- rpm_build
image:
name: artifactory.ops.ripple.com/almalinux:8
<<: *run_local_smoketest
alma_9_smoketest:
stage: smoketest
dependencies:
- rpm_build
image:
name: artifactory.ops.ripple.com/almalinux:9
<<: *run_local_smoketest
fedora_38_smoketest:
stage: smoketest
dependencies:
- rpm_build
image:
name: artifactory.ops.ripple.com/fedora:38
<<: *run_local_smoketest
fedora_39_smoketest:
stage: smoketest
dependencies:
- rpm_build
image:
name: artifactory.ops.ripple.com/fedora:39
<<: *run_local_smoketest
ubuntu_18_smoketest:
stage: smoketest
dependencies:
- dpkg_build
image:
name: artifactory.ops.ripple.com/ubuntu:18.04
<<: *run_local_smoketest
ubuntu_20_smoketest:
stage: smoketest
dependencies:
- dpkg_build
image:
name: artifactory.ops.ripple.com/ubuntu:20.04
<<: *run_local_smoketest
ubuntu_22_smoketest:
stage: smoketest
dependencies:
- dpkg_build
image:
name: artifactory.ops.ripple.com/ubuntu:22.04
<<: *run_local_smoketest
debian_10_smoketest:
stage: smoketest
dependencies:
- dpkg_build
image:
name: artifactory.ops.ripple.com/debian:10
<<: *run_local_smoketest
debian_11_smoketest:
stage: smoketest
dependencies:
- dpkg_build
image:
name: artifactory.ops.ripple.com/debian:11
<<: *run_local_smoketest
debian_12_smoketest:
stage: smoketest
dependencies:
- dpkg_build
image:
name: artifactory.ops.ripple.com/debian:12
<<: *run_local_smoketest
#########################################################################
## ##
## stage: verify_sig ##
## ##
## use git/gpg to verify that HEAD is signed by an approved ##
## committer. The whitelist of pubkeys is manually mantained ##
## and fetched from GIT_SIGN_PUBKEYS_URL (currently a snippet ##
## link). ##
## ONLY RUNS FOR PRIMARY BRANCHES/REPO ##
## ##
#########################################################################
verify_head_signed:
stage: verify_sig
image:
name: artifactory.ops.ripple.com/ubuntu:latest
<<: *only_primary
script:
- . ./Builds/containers/gitlab-ci/verify_head_commit.sh
#########################################################################
## ##
## stage: tag_images ##
## ##
## apply rippled version tag to containers from previous stage. ##
## ONLY RUNS FOR PRIMARY BRANCHES/REPO ##
## ##
#########################################################################
tag_bld_images:
stage: tag_images
variables:
docker_driver: overlay2
DOCKER_TLS_CERTDIR: ""
image:
name: artifactory.ops.ripple.com/docker:latest
services:
# workaround for TLS issues - consider going back
# back to unversioned `dind` when issues are resolved
- name: artifactory.ops.ripple.com/docker:stable-dind
alias: docker
tags:
- large
dependencies:
- rpm_sign
- dpkg_sign
<<: *only_primary
script:
- . ./Builds/containers/gitlab-ci/tag_docker_image.sh
#########################################################################
## ##
## stage: push_to_test ##
## ##
## push packages to artifactory repositories (test) ##
## ONLY RUNS FOR PRIMARY BRANCHES/REPO ##
## ##
#########################################################################
push_test:
stage: push_to_test
variables:
DEB_REPO: "rippled-deb-test-mirror"
RPM_REPO: "rippled-rpm-test-mirror"
image:
name: artifactory.ops.ripple.com/alpine:latest
artifacts:
paths:
- files.info
dependencies:
- rpm_sign
- dpkg_sign
<<: *only_primary
script:
- . ./Builds/containers/gitlab-ci/push_to_artifactory.sh "PUT" "."
#########################################################################
## ##
## stage: verify_from_test ##
## ##
## install/test packages from test repos. ##
## ONLY RUNS FOR PRIMARY BRANCHES/REPO ##
## ##
#########################################################################
centos_7_verify_repo_test:
stage: verify_from_test
variables:
RPM_REPO: "rippled-rpm-test-mirror"
image:
name: artifactory.ops.ripple.com/centos:7
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
rocky_8_verify_repo_test:
stage: verify_from_test
variables:
RPM_REPO: "rippled-rpm-test-mirror"
image:
name: artifactory.ops.ripple.com/rockylinux/rockylinux:8
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
rocky_9_verify_repo_test:
stage: verify_from_test
variables:
RPM_REPO: "rippled-rpm-test-mirror"
image:
name: artifactory.ops.ripple.com/rockylinux/rockylinux:9
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
almalinux_8_verify_repo_test:
stage: verify_from_test
variables:
RPM_REPO: "rippled-rpm-test-mirror"
image:
name: artifactory.ops.ripple.com/almalinux:8
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
almalinux_9_verify_repo_test:
stage: verify_from_test
variables:
RPM_REPO: "rippled-rpm-test-mirror"
image:
name: artifactory.ops.ripple.com/almalinux:9
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
fedora_38_verify_repo_test:
stage: verify_from_test
variables:
RPM_REPO: "rippled-rpm-test-mirror"
image:
name: artifactory.ops.ripple.com/fedora:38
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
fedora_39_verify_repo_test:
stage: verify_from_test
variables:
RPM_REPO: "rippled-rpm-test-mirror"
image:
name: artifactory.ops.ripple.com/fedora:39
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
ubuntu_18_verify_repo_test:
stage: verify_from_test
variables:
DISTRO: "bionic"
DEB_REPO: "rippled-deb-test-mirror"
image:
name: artifactory.ops.ripple.com/ubuntu:18.04
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
ubuntu_20_verify_repo_test:
stage: verify_from_test
variables:
DISTRO: "focal"
DEB_REPO: "rippled-deb-test-mirror"
image:
name: artifactory.ops.ripple.com/ubuntu:20.04
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
ubuntu_22_verify_repo_test:
stage: verify_from_test
variables:
DISTRO: "jammy"
DEB_REPO: "rippled-deb-test-mirror"
image:
name: artifactory.ops.ripple.com/ubuntu:22.04
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
debian_10_verify_repo_test:
stage: verify_from_test
variables:
DISTRO: "buster"
DEB_REPO: "rippled-deb-test-mirror"
image:
name: artifactory.ops.ripple.com/debian:10
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
debian_11_verify_repo_test:
stage: verify_from_test
variables:
DISTRO: "bullseye"
DEB_REPO: "rippled-deb-test-mirror"
image:
name: artifactory.ops.ripple.com/debian:11
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
debian_12_verify_repo_test:
stage: verify_from_test
variables:
DISTRO: "bookworm"
DEB_REPO: "rippled-deb-test-mirror"
image:
name: artifactory.ops.ripple.com/debian:12
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
#########################################################################
## ##
## stage: wait_approval_prod ##
## ##
## wait for manual approval before proceeding to next stage ##
## which pushes to prod repo. ##
## ONLY RUNS FOR PRIMARY BRANCHES/REPO ##
## ##
#########################################################################
wait_before_push_prod:
stage: wait_approval_prod
image:
name: artifactory.ops.ripple.com/alpine:latest
<<: *only_primary
script:
- echo "proceeding to next stage"
when: manual
allow_failure: false
#########################################################################
## ##
## stage: push_to_prod ##
## ##
## push packages to artifactory repositories (prod) ##
## ONLY RUNS FOR PRIMARY BRANCHES/REPO ##
## ##
#########################################################################
push_prod:
variables:
DEB_REPO: "rippled-deb"
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/alpine:latest
stage: push_to_prod
artifacts:
paths:
- files.info
dependencies:
- rpm_sign
- dpkg_sign
<<: *only_primary
script:
- . ./Builds/containers/gitlab-ci/push_to_artifactory.sh "PUT" "."
#########################################################################
## ##
## stage: verify_from_prod ##
## ##
## install/test packages from prod repos. ##
## ONLY RUNS FOR PRIMARY BRANCHES/REPO ##
## ##
#########################################################################
centos_7_verify_repo_prod:
stage: verify_from_prod
variables:
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/centos:7
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
rocky_8_verify_repo_prod:
stage: verify_from_prod
variables:
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/rockylinux/rockylinux:8
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
rocky_9_verify_repo_prod:
stage: verify_from_prod
variables:
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/rockylinux/rockylinux:9
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
alma_8_verify_repo_prod:
stage: verify_from_prod
variables:
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/almalinux:8
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
alma_9_verify_repo_prod:
stage: verify_from_prod
variables:
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/almalinux:9
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
fedora_37_verify_repo_prod:
stage: verify_from_prod
variables:
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/fedora:37
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
fedora_38_verify_repo_prod:
stage: verify_from_prod
variables:
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/fedora:38
dependencies:
- rpm_sign
<<: *only_primary
<<: *run_repo_smoketest
ubuntu_18_verify_repo_prod:
stage: verify_from_prod
variables:
DISTRO: "bionic"
DEB_REPO: "rippled-deb"
image:
name: artifactory.ops.ripple.com/ubuntu:18.04
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
ubuntu_20_verify_repo_prod:
stage: verify_from_prod
variables:
DISTRO: "focal"
DEB_REPO: "rippled-deb"
image:
name: artifactory.ops.ripple.com/ubuntu:20.04
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
ubuntu_22_verify_repo_prod:
stage: verify_from_prod
variables:
DISTRO: "jammy"
DEB_REPO: "rippled-deb"
image:
name: artifactory.ops.ripple.com/ubuntu:22.04
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
debian_10_verify_repo_prod:
stage: verify_from_prod
variables:
DISTRO: "buster"
DEB_REPO: "rippled-deb"
image:
name: artifactory.ops.ripple.com/debian:10
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
debian_11_verify_repo_prod:
stage: verify_from_prod
variables:
DISTRO: "bullseye"
DEB_REPO: "rippled-deb"
image:
name: artifactory.ops.ripple.com/debian:11
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
debian_12_verify_repo_prod:
stage: verify_from_prod
variables:
DISTRO: "bookworm"
DEB_REPO: "rippled-deb"
image:
name: artifactory.ops.ripple.com/debian:12
dependencies:
- dpkg_sign
<<: *only_primary
<<: *run_repo_smoketest
#########################################################################
## ##
## stage: get_final_hashes ##
## ##
## fetch final hashes from artifactory. ##
## ONLY RUNS FOR PRIMARY BRANCHES/REPO ##
## ##
#########################################################################
get_prod_hashes:
variables:
DEB_REPO: "rippled-deb"
RPM_REPO: "rippled-rpm"
image:
name: artifactory.ops.ripple.com/alpine:latest
stage: get_final_hashes
artifacts:
paths:
- files.info
dependencies:
- rpm_sign
- dpkg_sign
<<: *only_primary
script:
- . ./Builds/containers/gitlab-ci/push_to_artifactory.sh "GET" ".checksums"
#########################################################################
## ##
## stage: build_containers ##
## ##
## build containers from docker definitions. These containers are NOT ##
## used for the package build. This step is only used to ensure that ##
## the package build targets and files are still working properly. ##
## ##
#########################################################################
build_centos_container:
stage: build_containers
<<: *dind_param
script:
- . ./Builds/containers/gitlab-ci/build_container.sh rpm
build_ubuntu_container:
stage: build_containers
<<: *dind_param
script:
- . ./Builds/containers/gitlab-ci/build_container.sh dpkg

View File

@@ -0,0 +1,92 @@
#!/usr/bin/env sh
set -e
action=$1
filter=$2
. ./Builds/containers/gitlab-ci/get_component.sh
apk add curl jq coreutils util-linux
TOPDIR=$(pwd)
# DPKG
cd $TOPDIR
cd build/dpkg/packages
CURLARGS="-sk -X${action} -urippled:${ARTIFACTORY_DEPLOY_KEY_RIPPLED}"
RIPPLED_PKG=$(ls rippled_*.deb)
RIPPLED_REPORTING_PKG=$(ls rippled-reporting_*.deb)
RIPPLED_DBG_PKG=$(ls rippled-dbgsym_*.*deb)
RIPPLED_REPORTING_DBG_PKG=$(ls rippled-reporting-dbgsym_*.*deb)
# TODO - where to upload src tgz?
RIPPLED_SRC=$(ls rippled_*.orig.tar.gz)
DEB_MATRIX=";deb.component=${COMPONENT};deb.architecture=amd64"
for dist in bookworm buster bullseye bionic focal jammy; do
DEB_MATRIX="${DEB_MATRIX};deb.distribution=${dist}"
done
echo "{ \"debs\": {" > "${TOPDIR}/files.info"
for deb in ${RIPPLED_PKG} ${RIPPLED_DBG_PKG} ${RIPPLED_REPORTING_PKG} ${RIPPLED_REPORTING_DBG_PKG}; do
# first item doesn't get a comma separator
if [ $deb != $RIPPLED_PKG ] ; then
echo "," >> "${TOPDIR}/files.info"
fi
echo "\"${deb}\"": | tee -a "${TOPDIR}/files.info"
ca="${CURLARGS}"
if [ "${action}" = "PUT" ] ; then
url="https://${ARTIFACTORY_HOST}/artifactory/${DEB_REPO}/pool/${COMPONENT}/${deb}${DEB_MATRIX}"
ca="${ca} -T${deb}"
elif [ "${action}" = "GET" ] ; then
url="https://${ARTIFACTORY_HOST}/artifactory/api/storage/${DEB_REPO}/pool/${COMPONENT}/${deb}"
fi
echo "file info request url --> ${url}"
eval "curl ${ca} \"${url}\"" | jq -M "${filter}" | tee -a "${TOPDIR}/files.info"
done
echo "}," >> "${TOPDIR}/files.info"
# RPM
cd $TOPDIR
cd build/rpm/packages
RIPPLED_PKG=$(ls rippled-[0-9]*.x86_64.rpm)
RIPPLED_DEV_PKG=$(ls rippled-devel*.rpm)
RIPPLED_DBG_PKG=$(ls rippled-debuginfo*.rpm)
RIPPLED_REPORTING_PKG=$(ls rippled-reporting*.rpm)
# TODO - where to upload src rpm ?
RIPPLED_SRC=$(ls rippled-[0-9]*.src.rpm)
echo "\"rpms\": {" >> "${TOPDIR}/files.info"
for rpm in ${RIPPLED_PKG} ${RIPPLED_DEV_PKG} ${RIPPLED_DBG_PKG} ${RIPPLED_REPORTING_PKG}; do
# first item doesn't get a comma separator
if [ $rpm != $RIPPLED_PKG ] ; then
echo "," >> "${TOPDIR}/files.info"
fi
echo "\"${rpm}\"": | tee -a "${TOPDIR}/files.info"
ca="${CURLARGS}"
if [ "${action}" = "PUT" ] ; then
url="https://${ARTIFACTORY_HOST}/artifactory/${RPM_REPO}/${COMPONENT}/"
ca="${ca} -T${rpm}"
elif [ "${action}" = "GET" ] ; then
url="https://${ARTIFACTORY_HOST}/artifactory/api/storage/${RPM_REPO}/${COMPONENT}/${rpm}"
fi
echo "file info request url --> ${url}"
eval "curl ${ca} \"${url}\"" | jq -M "${filter}" | tee -a "${TOPDIR}/files.info"
done
echo "}}" >> "${TOPDIR}/files.info"
jq '.' "${TOPDIR}/files.info" > "${TOPDIR}/files.info.tmp"
mv "${TOPDIR}/files.info.tmp" "${TOPDIR}/files.info"
if [ ! -z "${SLACK_NOTIFY_URL}" ] && [ "${action}" = "GET" ] ; then
# extract files.info content to variable and sanitize so it can
# be interpolated into a slack text field below
finfo=$(cat ${TOPDIR}/files.info | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g' | sed -E 's/"/\\"/g')
# try posting file info to slack.
# can add channel field to payload if the
# default channel is incorrect. Get rid of
# newlines in payload json since slack doesn't accept them
CONTENT=$(tr -d '[\n]' <<JSON
payload={
"username": "GitlabCI",
"text": "The package build for branch \`${CI_COMMIT_REF_NAME}\` is complete. File hashes are: \`\`\`${finfo}\`\`\`",
"icon_emoji": ":package:"}
JSON
)
curl ${SLACK_NOTIFY_URL} --data-urlencode "${CONTENT}"
fi

View File

@@ -0,0 +1,38 @@
#!/usr/bin/env bash
set -eo pipefail
sign_dpkg() {
if [ -n "${GPG_KEYID}" ]; then
dpkg-sig \
-g "--no-tty --digest-algo 'sha512' --passphrase '${GPG_PASSPHRASE}' --pinentry-mode=loopback" \
-k "${GPG_KEYID}" \
--sign builder \
"build/dpkg/packages/*.deb"
fi
}
sign_rpm() {
if [ -n "${GPG_KEYID}" ] ; then
find build/rpm/packages -name "*.rpm" -exec bash -c '
echo "yes" | setsid rpm \
--define "_gpg_name ${GPG_KEYID}" \
--define "_signature gpg" \
--define "__gpg_check_password_cmd /bin/true" \
--define "__gpg_sign_cmd %{__gpg} gpg --batch --no-armor --digest-algo 'sha512' --passphrase '${GPG_PASSPHRASE}' --no-secmem-warning -u '%{_gpg_name}' --sign --detach-sign --output %{__signature_filename} %{__plaintext_filename}" \
--addsign '{} \;
fi
}
case "${1}" in
dpkg)
sign_dpkg
;;
rpm)
sign_rpm
;;
*)
echo "Usage: ${0} (dpkg|rpm)"
;;
esac

View File

@@ -0,0 +1,101 @@
#!/usr/bin/env sh
set -e
install_from=$1
use_private=${2:-0} # this option not currently needed by any CI scripts,
# reserved for possible future use
if [ "$use_private" -gt 0 ] ; then
REPO_ROOT="https://rippled:${ARTIFACTORY_DEPLOY_KEY_RIPPLED}@${ARTIFACTORY_HOST}/artifactory"
else
REPO_ROOT="${PUBLIC_REPO_ROOT}"
fi
. ./Builds/containers/gitlab-ci/get_component.sh
. /etc/os-release
case ${ID} in
ubuntu|debian)
pkgtype="dpkg"
;;
fedora|centos|rhel|scientific|rocky|almalinux)
pkgtype="rpm"
;;
*)
echo "unrecognized distro!"
exit 1
;;
esac
# this script provides info variables about pkg version
. build/${pkgtype}/packages/build_vars
if [ "${pkgtype}" = "dpkg" ] ; then
# sometimes update fails and requires a cleanup
updateWithRetry()
{
if ! apt-get -y update ; then
rm -rvf /var/lib/apt/lists/*
apt-get -y clean
apt-get -y update
fi
}
if [ "${install_from}" = "repo" ] ; then
apt-get -y upgrade
updateWithRetry
apt-get -y install apt apt-transport-https ca-certificates coreutils util-linux wget gnupg
wget -q -O - "${REPO_ROOT}/api/gpg/key/public" | apt-key add -
echo "deb ${REPO_ROOT}/${DEB_REPO} ${DISTRO} ${COMPONENT}" >> /etc/apt/sources.list
updateWithRetry
# uncomment this next line if you want to see the available package versions
# apt-cache policy rippled
apt-get -y install rippled=${dpkg_full_version}
elif [ "${install_from}" = "local" ] ; then
# cached pkg install
updateWithRetry
apt-get -y install libprotobuf-dev libprotoc-dev protobuf-compiler libssl-dev
rm -f build/dpkg/packages/rippled-dbgsym*.*
dpkg --no-debsig -i build/dpkg/packages/*.deb
else
echo "unrecognized pkg source!"
exit 1
fi
else
yum -y update
if [ "${install_from}" = "repo" ] ; then
pkgs=("yum-utils coreutils util-linux")
case "$ID" in
rocky|almalinux)
pkgs="${pkgs[@]/coreutils}"
esac
yum install -y $pkgs
REPOFILE="/etc/yum.repos.d/artifactory.repo"
echo "[Artifactory]" > ${REPOFILE}
echo "name=Artifactory" >> ${REPOFILE}
echo "baseurl=${REPO_ROOT}/${RPM_REPO}/${COMPONENT}/" >> ${REPOFILE}
echo "enabled=1" >> ${REPOFILE}
echo "gpgcheck=0" >> ${REPOFILE}
echo "gpgkey=${REPO_ROOT}/${RPM_REPO}/${COMPONENT}/repodata/repomd.xml.key" >> ${REPOFILE}
echo "repo_gpgcheck=1" >> ${REPOFILE}
yum -y update
# uncomment this next line if you want to see the available package versions
# yum --showduplicates list rippled
yum -y install ${rpm_version_release}
elif [ "${install_from}" = "local" ] ; then
rm -f build/rpm/packages/rippled-debug*.rpm
rm -f build/rpm/packages/rippled-devel*.rpm
rm -f build/rpm/packages/*.src.rpm
rpm -i build/rpm/packages/*.rpm
else
echo "unrecognized pkg source!"
exit 1
fi
fi
# verify installed version
INSTALLED=$(/opt/ripple/bin/rippled --version | awk '{print $NF}')
if [ "${rippled_version}" != "${INSTALLED}" ] ; then
echo "INSTALLED version ${INSTALLED} does not match ${rippled_version}"
exit 1
fi
# run unit tests
/opt/ripple/bin/rippled --unittest --unittest-jobs $(nproc)
/opt/ripple/bin/validator-keys --unittest

View File

@@ -0,0 +1,21 @@
#!/usr/bin/env sh
set -e
docker login -u rippled \
-p ${ARTIFACTORY_DEPLOY_KEY_RIPPLED} "${ARTIFACTORY_HUB}"
# this gives us rippled_version :
source build/rpm/packages/build_vars
docker pull "${ARTIFACTORY_HUB}/${RPM_CONTAINER_FULLNAME}"
docker pull "${ARTIFACTORY_HUB}/${DPKG_CONTAINER_FULLNAME}"
# tag/push two labels...one using the current rippled version and one just using "latest"
for label in ${rippled_version} latest ; do
docker tag \
"${ARTIFACTORY_HUB}/${RPM_CONTAINER_FULLNAME}" \
"${ARTIFACTORY_HUB}/${RPM_CONTAINER_NAME}:${label}_${CI_COMMIT_REF_SLUG}"
docker push \
"${ARTIFACTORY_HUB}/${RPM_CONTAINER_NAME}:${label}_${CI_COMMIT_REF_SLUG}"
docker tag \
"${ARTIFACTORY_HUB}/${DPKG_CONTAINER_FULLNAME}" \
"${ARTIFACTORY_HUB}/${DPKG_CONTAINER_NAME}:${label}_${CI_COMMIT_REF_SLUG}"
docker push \
"${ARTIFACTORY_HUB}/${DPKG_CONTAINER_NAME}:${label}_${CI_COMMIT_REF_SLUG}"
done

View File

@@ -0,0 +1,17 @@
#!/usr/bin/env sh
set -ex
apt -y update
DEBIAN_FRONTEND="noninteractive" apt-get -y install tzdata
apt -y install software-properties-common curl git gnupg
curl -sk -o rippled-pubkeys.txt "${GIT_SIGN_PUBKEYS_URL}"
gpg --import rippled-pubkeys.txt
if git verify-commit HEAD; then
echo "git commit signature check passed"
else
echo "git commit signature check failed"
git log -n 5 --color \
--pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an> [%G?]%Creset' \
--abbrev-commit
exit 1
fi

View File

@@ -0,0 +1,99 @@
#!/usr/bin/env bash
set -ex
# make sure pkg source files are up to date with repo
cd /opt/rippled_bld/pkg
cp -fpru rippled/Builds/containers/packaging/dpkg/debian/. debian/
cp -fpu rippled/Builds/containers/shared/rippled*.service debian/
cp -fpu rippled/Builds/containers/shared/update_sources.sh .
source update_sources.sh
# Build the dpkg
#dpkg uses - as separator, so we need to change our -bN versions to tilde
RIPPLED_DPKG_VERSION=$(echo "${RIPPLED_VERSION}" | sed 's!-!~!g')
# TODO - decide how to handle the trailing/release
# version here (hardcoded to 1). Does it ever need to change?
RIPPLED_DPKG_FULL_VERSION="${RIPPLED_DPKG_VERSION}-1"
git config --global --add safe.directory /opt/rippled_bld/pkg/rippled
cd /opt/rippled_bld/pkg/rippled
if [[ -n $(git status --porcelain) ]]; then
git status
error "Unstaged changes in this repo - please commit first"
fi
git archive --format tar.gz --prefix rippled-${RIPPLED_DPKG_VERSION}/ -o ../rippled-${RIPPLED_DPKG_VERSION}.tar.gz HEAD
cd ..
# dpkg debmake would normally create this link, but we do it manually
ln -s ./rippled-${RIPPLED_DPKG_VERSION}.tar.gz rippled_${RIPPLED_DPKG_VERSION}.orig.tar.gz
tar xvf rippled-${RIPPLED_DPKG_VERSION}.tar.gz
cd rippled-${RIPPLED_DPKG_VERSION}
cp -pr ../debian .
# dpkg requires a changelog. We don't currently maintain
# a useable one, so let's just fake it with our current version
# TODO : not sure if the "unstable" will need to change for
# release packages (?)
NOWSTR=$(TZ=UTC date -R)
cat << CHANGELOG > ./debian/changelog
rippled (${RIPPLED_DPKG_FULL_VERSION}) unstable; urgency=low
* see RELEASENOTES
-- Ripple Labs Inc. <support@ripple.com> ${NOWSTR}
CHANGELOG
# PATH must be preserved for our more modern cmake in /opt/local
# TODO : consider allowing lintian to run in future ?
export DH_BUILD_DDEBS=1
debuild --no-lintian --preserve-envvar PATH --preserve-env -us -uc
rc=$?; if [[ $rc != 0 ]]; then
error "error building dpkg"
fi
cd ..
# copy artifacts
cp rippled-reporting_${RIPPLED_DPKG_FULL_VERSION}_amd64.deb ${PKG_OUTDIR}
cp rippled_${RIPPLED_DPKG_FULL_VERSION}_amd64.deb ${PKG_OUTDIR}
cp rippled_${RIPPLED_DPKG_FULL_VERSION}.dsc ${PKG_OUTDIR}
# dbgsym suffix is ddeb under newer debuild, but just deb under earlier
cp rippled-dbgsym_${RIPPLED_DPKG_FULL_VERSION}_amd64.* ${PKG_OUTDIR}
cp rippled-reporting-dbgsym_${RIPPLED_DPKG_FULL_VERSION}_amd64.* ${PKG_OUTDIR}
cp rippled_${RIPPLED_DPKG_FULL_VERSION}_amd64.changes ${PKG_OUTDIR}
cp rippled_${RIPPLED_DPKG_FULL_VERSION}_amd64.build ${PKG_OUTDIR}
cp rippled_${RIPPLED_DPKG_VERSION}.orig.tar.gz ${PKG_OUTDIR}
cp rippled_${RIPPLED_DPKG_FULL_VERSION}.debian.tar.xz ${PKG_OUTDIR}
# buildinfo is only generated by later version of debuild
if [ -e rippled_${RIPPLED_DPKG_FULL_VERSION}_amd64.buildinfo ] ; then
cp rippled_${RIPPLED_DPKG_FULL_VERSION}_amd64.buildinfo ${PKG_OUTDIR}
fi
pushd ${PKG_OUTDIR}
for f in *.ddeb; do mv -- "$f" "${f%.ddeb}.deb"; done
popd
cat rippled_${RIPPLED_DPKG_FULL_VERSION}_amd64.changes
# extract the text in the .changes file that appears between
# Checksums-Sha256: ...
# and
# Files: ...
awk '/Checksums-Sha256:/{hit=1;next}/Files:/{hit=0}hit' \
rippled_${RIPPLED_DPKG_VERSION}-1_amd64.changes | \
sed -E 's!^[[:space:]]+!!' > shasums
DEB_SHA256=$(cat shasums | \
grep "rippled_${RIPPLED_DPKG_VERSION}-1_amd64.deb" | cut -d " " -f 1)
DBG_SHA256=$(cat shasums | \
grep "rippled-dbgsym_${RIPPLED_DPKG_VERSION}-1_amd64.*" | cut -d " " -f 1)
REPORTING_DBG_SHA256=$(cat shasums | \
grep "rippled-reporting-dbgsym_${RIPPLED_DPKG_VERSION}-1_amd64.*" | cut -d " " -f 1)
REPORTING_SHA256=$(cat shasums | \
grep "rippled-reporting_${RIPPLED_DPKG_VERSION}-1_amd64.deb" | cut -d " " -f 1)
SRC_SHA256=$(cat shasums | \
grep "rippled_${RIPPLED_DPKG_VERSION}.orig.tar.gz" | cut -d " " -f 1)
echo "deb_sha256=${DEB_SHA256}" >> ${PKG_OUTDIR}/build_vars
echo "dbg_sha256=${DBG_SHA256}" >> ${PKG_OUTDIR}/build_vars
echo "reporting_sha256=${REPORTING_SHA256}" >> ${PKG_OUTDIR}/build_vars
echo "reporting_dbg_sha256=${REPORTING_DBG_SHA256}" >> ${PKG_OUTDIR}/build_vars
echo "src_sha256=${SRC_SHA256}" >> ${PKG_OUTDIR}/build_vars
echo "rippled_version=${RIPPLED_VERSION}" >> ${PKG_OUTDIR}/build_vars
echo "dpkg_version=${RIPPLED_DPKG_VERSION}" >> ${PKG_OUTDIR}/build_vars
echo "dpkg_full_version=${RIPPLED_DPKG_FULL_VERSION}" >> ${PKG_OUTDIR}/build_vars

View File

@@ -0,0 +1,3 @@
rippled daemon
-- Mike Ellery <mellery451@gmail.com> Tue, 04 Dec 2018 18:19:03 +0000

View File

@@ -0,0 +1 @@
10

View File

@@ -0,0 +1,19 @@
Source: rippled
Section: misc
Priority: extra
Maintainer: Ripple Labs Inc. <support@ripple.com>
Build-Depends: cmake, debhelper (>=9), zlib1g-dev, dh-systemd, ninja-build
Standards-Version: 3.9.7
Homepage: http://ripple.com/
Package: rippled
Architecture: any
Multi-Arch: foreign
Depends: ${misc:Depends}, ${shlibs:Depends}
Description: rippled daemon
Package: rippled-reporting
Architecture: any
Multi-Arch: foreign
Depends: ${misc:Depends}, ${shlibs:Depends}
Description: rippled reporting daemon

View File

@@ -0,0 +1,86 @@
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: rippled
Source: https://github.com/ripple/rippled
Files: *
Copyright: 2012-2019 Ripple Labs Inc.
License: __UNKNOWN__
The accompanying files under various copyrights.
Copyright (c) 2012, 2013, 2014 Ripple Labs Inc.
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
The accompanying files incorporate work covered by the following copyright
and previous license notice:
Copyright (c) 2011 Arthur Britto, David Schwartz, Jed McCaleb,
Vinnie Falco, Bob Way, Eric Lombrozo, Nikolaos D. Bougalis, Howard Hinnant
Some code from Raw Material Software, Ltd., provided under the terms of the
ISC License. See the corresponding source files for more details.
Copyright (c) 2013 - Raw Material Software Ltd.
Please visit http://www.juce.com
Some code from ASIO examples:
// Copyright (c) 2003-2011 Christopher M. Kohlhoff (chris at kohlhoff dot com)
//
// Distributed under the Boost Software License, Version 1.0. (See accompanying
// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
Some code from Bitcoin:
// Copyright (c) 2009-2010 Satoshi Nakamoto
// Copyright (c) 2011 The Bitcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file license.txt or http://www.opensource.org/licenses/mit-license.php.
Some code from Tom Wu:
This software is covered under the following copyright:
/*
* Copyright (c) 2003-2005 Tom Wu
* All Rights Reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining
* a copy of this software and associated documentation files (the
* "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish,
* distribute, sublicense, and/or sell copies of the Software, and to
* permit persons to whom the Software is furnished to do so, subject to
* the following conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS-IS" AND WITHOUT WARRANTY OF ANY KIND,
* EXPRESS, IMPLIED OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
* WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
*
* IN NO EVENT SHALL TOM WU BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
* INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES WHATSOEVER
* RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR NOT ADVISED OF
* THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF LIABILITY, ARISING OUT
* OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*
* In addition, the following condition applies:
*
* All redistributions must retain an intact copy of this copyright notice
* and disclaimer.
*/
Address all questions regarding this license to:
Tom Wu
tjw@cs.Stanford.EDU

View File

@@ -0,0 +1,3 @@
/var/log/rippled/
/var/lib/rippled/
/etc/systemd/system/rippled.service.d/

View File

@@ -0,0 +1,3 @@
README.md
LICENSE.md
RELEASENOTES.md

View File

@@ -0,0 +1,3 @@
opt/ripple/include
opt/ripple/lib/*.a
opt/ripple/lib/cmake/ripple

View File

@@ -0,0 +1,3 @@
/var/log/rippled-reporting/
/var/lib/rippled-reporting/
/etc/systemd/system/rippled-reporting.service.d/

View File

@@ -0,0 +1,8 @@
build.rippled-reporting/rippled-reporting opt/rippled-reporting/bin
cfg/rippled-reporting.cfg opt/rippled-reporting/etc
debian/tmp/opt/rippled-reporting/etc/validators.txt opt/rippled-reporting/etc
opt/rippled-reporting/bin/update-rippled-reporting.sh
opt/rippled-reporting/bin/getRippledReportingInfo
opt/rippled-reporting/etc/update-rippled-reporting-cron
etc/logrotate.d/rippled-reporting

View File

@@ -0,0 +1,3 @@
opt/rippled-reporting/etc/rippled-reporting.cfg etc/opt/rippled-reporting/rippled-reporting.cfg
opt/rippled-reporting/etc/validators.txt etc/opt/rippled-reporting/validators.txt
opt/rippled-reporting/bin/rippled-reporting usr/local/bin/rippled-reporting

View File

@@ -0,0 +1,33 @@
#!/bin/sh
set -e
USER_NAME=rippled-reporting
GROUP_NAME=rippled-reporting
case "$1" in
configure)
id -u $USER_NAME >/dev/null 2>&1 || \
adduser --system --quiet \
--home /nonexistent --no-create-home \
--disabled-password \
--group "$GROUP_NAME"
chown -R $USER_NAME:$GROUP_NAME /var/log/rippled-reporting/
chown -R $USER_NAME:$GROUP_NAME /var/lib/rippled-reporting/
chmod 755 /var/log/rippled-reporting/
chmod 755 /var/lib/rippled-reporting/
chown -R $USER_NAME:$GROUP_NAME /opt/rippled-reporting
;;
abort-upgrade|abort-remove|abort-deconfigure)
;;
*)
echo "postinst called with unknown argument \`$1'" >&2
exit 1
;;
esac
#DEBHELPER#
exit 0

View File

@@ -0,0 +1,2 @@
/opt/ripple/etc/rippled.cfg
/opt/ripple/etc/validators.txt

View File

@@ -0,0 +1,8 @@
opt/ripple/bin/rippled
opt/ripple/bin/validator-keys
opt/ripple/bin/update-rippled.sh
opt/ripple/bin/getRippledInfo
opt/ripple/etc/rippled.cfg
opt/ripple/etc/validators.txt
opt/ripple/etc/update-rippled-cron
etc/logrotate.d/rippled

View File

@@ -0,0 +1,3 @@
opt/ripple/etc/rippled.cfg etc/opt/ripple/rippled.cfg
opt/ripple/etc/validators.txt etc/opt/ripple/validators.txt
opt/ripple/bin/rippled usr/local/bin/rippled

View File

@@ -0,0 +1,35 @@
#!/bin/sh
set -e
USER_NAME=rippled
GROUP_NAME=rippled
case "$1" in
configure)
id -u $USER_NAME >/dev/null 2>&1 || \
adduser --system --quiet \
--home /nonexistent --no-create-home \
--disabled-password \
--group "$GROUP_NAME"
chown -R $USER_NAME:$GROUP_NAME /var/log/rippled/
chown -R $USER_NAME:$GROUP_NAME /var/lib/rippled/
chown -R $USER_NAME:$GROUP_NAME /opt/ripple
chmod 755 /var/log/rippled/
chmod 755 /var/lib/rippled/
chmod 644 /opt/ripple/etc/update-rippled-cron
chmod 644 /etc/logrotate.d/rippled
chown -R root:$GROUP_NAME /opt/ripple/etc/update-rippled-cron
;;
abort-upgrade|abort-remove|abort-deconfigure)
;;
*)
echo "postinst called with unknown argument \`$1'" >&2
exit 1
;;
esac
#DEBHELPER#
exit 0

View File

@@ -0,0 +1,17 @@
#!/bin/sh
set -e
case "$1" in
purge|remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear)
;;
*)
echo "postrm called with unknown argument \`$1'" >&2
exit 1
;;
esac
#DEBHELPER#
exit 0

Some files were not shown because too many files have changed in this diff Show More