ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2025-12-06 17:27:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Reduce write amplification by merging files in L0 back into L0

Browse Source 
Summary:
There is a new option called hybrid_mode which, when switched on,
causes HBase style compactions.  Files from L0 are
compacted back into L0. This meat of this compaction algorithm
is in PickCompactionHybrid().

All files reside in L0. That means all files have overlapping
keys. Each file has a time-bound, i.e. each file contains a
range of keys that were inserted around the same time. The
start-seqno and the end-seqno refers to the timeframe when
these keys were inserted.  Files that have contiguous seqno
are compacted together into a larger file. All files are
ordered from most recent to the oldest.

The current compaction algorithm starts to look for
candidate files starting from the most recent file. It continues to
add more files to the same compaction run as long as the
sum of the files chosen till now is smaller than the next
candidate file size. This logic needs to be debated
and validated.

The above logic should reduce write amplification to a
large extent... will publish numbers shortly.

Test Plan: dbstress runs for 6 hours with no data corruption (tested so far).

Differential Revision: https://reviews.facebook.net/D11289
This commit is contained in:
Dhruba Borthakur
2013-06-13 22:09:08 -07:00
parent 71e0f695c1
commit 47c4191fe8
15 changed files with 406 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
include/leveldb/options.h
View File

@@ -476,6 +476,17 @@ struct Options {
// Default: 0
uint64_t bytes_per_sync;
// Hybrid Mode. There is only a single level and files in L0 are
// compacted back into L0. Default: false
bool hybrid_mode;
// Percentage flexibilty while comparing file size. If the candidate file(s)
// size is 1% smaller than the next file's size, then include next file into
// this candidate set. // Default: 1
int hybrid_size_ratio;
// The minimum number of files in a single compaction run. Default: 2
int hybrid_min_numfiles_in_single_compaction;
};
// Options that control read operations

6
include/leveldb/statistics.h
View File

@@ -106,7 +106,8 @@ enum Histograms {
READ_BLOCK_COMPACTION_MICROS = 9,
READ_BLOCK_GET_MICROS = 10,
WRITE_RAW_BLOCK_MICROS = 11,
HISTOGRAM_ENUM_MAX = 12
NUM_FILES_IN_SINGLE_COMPACTION = 12,
HISTOGRAM_ENUM_MAX = 13
};
const std::vector<std::pair<Histograms, std::string>> HistogramsNameMap = {
@@ -121,7 +122,8 @@ const std::vector<std::pair<Histograms, std::string>> HistogramsNameMap = {
{ DB_MULTIGET, "rocksdb.db.multiget.micros" },
{ READ_BLOCK_COMPACTION_MICROS, "rocksdb.read.block.compaction.micros" },
{ READ_BLOCK_GET_MICROS, "rocksdb.read.block.get.micros" },
{ WRITE_RAW_BLOCK_MICROS, "rocksdb.write.raw.block.micros" }
{ WRITE_RAW_BLOCK_MICROS, "rocksdb.write.raw.block.micros" },
{ NUM_FILES_IN_SINGLE_COMPACTION, "rocksdb.numfiles.in.singlecompaction" }
};
struct HistogramData {