ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2025-11-20 02:55:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

NuDB: Performance improvements (RIPD-793,796):

Browse Source 
This introduces changes in nudb to improve speed, reduce database size,
and enhance correctness. The most significant change is to store hashes
rather than entire keys in the key file. The output of the hash function
is reduced to 48 bits, and stored directly in buckets.

The API is also modified to introduce a Codec parameter allowing for
compression and decompression to be supported in the implementation
itself rather than callers.

THe data file no longer contains a salt, as the salt is applicable
only to the key and log files. This allows a data file to have multiple
key files with different salt values. To distinguish physical files
belonging to the same logical database, a new field UID is introduced.
The UID is a 64-bit random value generated once on creation and stored
in all three files.

Buckets are zero filled to the end of each block, this is a security
measure to prevent unintended contents of memory getting stored to
disk. NuDB offers the varint integer type, this is identical to
the varint described by Google.

* Add varint
* Add Codec template argument
* Add "api" convenience traits
* Store hash in buckets
* istream can throw short read errors
* Support std::uint8_t format in streams
* Make file classes part of the public interface
* Remove buffers pessimization, replace with buffer
* Consolidate creation utility functions to the same header
* Zero fill unused areas of buckets on disk
* More coverage and improvements to the recover test
* Fix file read/write to loop until all bytes processed
* Add verify_fast, faster verify for large databases

The database version number is incremented to 2; older databases can
no longer be opened and should be deleted.
This commit is contained in:
Vinnie Falco
2015-02-03 07:46:24 -08:00
parent 62c5b5e570
commit e2a5535ed6
37 changed files with 2098 additions and 1300 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
Builds/VisualStudio2013/RippleD.vcxproj
View File

@@ -804,20 +804,22 @@
</ClCompile>
<ClInclude Include="..\..\src\beast\beast\nudb.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\api.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\common.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\create.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\arena.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\bucket.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\buffers.h">
<ClInclude Include="..\..\src\beast\beast\nudb\detail\buffer.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\bulkio.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\cache.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\config.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\field.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\format.h">
@@ -826,21 +828,19 @@
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\pool.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\posix_file.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\stream.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\win32_file.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\error.h">
<ClInclude Include="..\..\src\beast\beast\nudb\detail\varint.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\file.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\mode.h">
<ClInclude Include="..\..\src\beast\beast\nudb\identity_codec.h">
</ClInclude>
<ClCompile Include="..\..\src\beast\beast\nudb\nudb.cpp">
<ExcludedFromBuild>True</ExcludedFromBuild>
</ClCompile>
<ClInclude Include="..\..\src\beast\beast\nudb\posix_file.h">
</ClInclude>
<None Include="..\..\src\beast\beast\nudb\README.md">
</None>
<ClInclude Include="..\..\src\beast\beast\nudb\recover.h">
@@ -860,6 +860,9 @@
<ClCompile Include="..\..\src\beast\beast\nudb\tests\store_test.cpp">
<ExcludedFromBuild>True</ExcludedFromBuild>
</ClCompile>
<ClCompile Include="..\..\src\beast\beast\nudb\tests\varint_test.cpp">
<ExcludedFromBuild>True</ExcludedFromBuild>
</ClCompile>
<ClCompile Include="..\..\src\beast\beast\nudb\tests\verify_test.cpp">
<ExcludedFromBuild>True</ExcludedFromBuild>
</ClCompile>
@@ -867,6 +870,8 @@
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\visit.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\win32_file.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\random\rngfill.h">
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\random\xor_shift_engine.h">

30
Builds/VisualStudio2013/RippleD.vcxproj.filters
View File

@@ -1437,6 +1437,12 @@
<ClInclude Include="..\..\src\beast\beast\nudb.h">
<Filter>beast</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\api.h">
<Filter>beast\nudb</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\common.h">
<Filter>beast\nudb</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\create.h">
<Filter>beast\nudb</Filter>
</ClInclude>
@@ -1446,7 +1452,7 @@
<ClInclude Include="..\..\src\beast\beast\nudb\detail\bucket.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\buffers.h">
<ClInclude Include="..\..\src\beast\beast\nudb\detail\buffer.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\bulkio.h">
@@ -1455,9 +1461,6 @@
<ClInclude Include="..\..\src\beast\beast\nudb\detail\cache.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\config.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\field.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
@@ -1470,27 +1473,24 @@
<ClInclude Include="..\..\src\beast\beast\nudb\detail\pool.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\posix_file.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\stream.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\detail\win32_file.h">
<ClInclude Include="..\..\src\beast\beast\nudb\detail\varint.h">
<Filter>beast\nudb\detail</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\error.h">
<Filter>beast\nudb</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\file.h">
<Filter>beast\nudb</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\mode.h">
<ClInclude Include="..\..\src\beast\beast\nudb\identity_codec.h">
<Filter>beast\nudb</Filter>
</ClInclude>
<ClCompile Include="..\..\src\beast\beast\nudb\nudb.cpp">
<Filter>beast\nudb</Filter>
</ClCompile>
<ClInclude Include="..\..\src\beast\beast\nudb\posix_file.h">
<Filter>beast\nudb</Filter>
</ClInclude>
<None Include="..\..\src\beast\beast\nudb\README.md">
<Filter>beast\nudb</Filter>
</None>
@@ -1515,6 +1515,9 @@
<ClCompile Include="..\..\src\beast\beast\nudb\tests\store_test.cpp">
<Filter>beast\nudb\tests</Filter>
</ClCompile>
<ClCompile Include="..\..\src\beast\beast\nudb\tests\varint_test.cpp">
<Filter>beast\nudb\tests</Filter>
</ClCompile>
<ClCompile Include="..\..\src\beast\beast\nudb\tests\verify_test.cpp">
<Filter>beast\nudb\tests</Filter>
</ClCompile>
@@ -1524,6 +1527,9 @@
<ClInclude Include="..\..\src\beast\beast\nudb\visit.h">
<Filter>beast\nudb</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\nudb\win32_file.h">
<Filter>beast\nudb</Filter>
</ClInclude>
<ClInclude Include="..\..\src\beast\beast\random\rngfill.h">
<Filter>beast\random</Filter>
</ClInclude>