[/
    Copyright (c) 2013-2016 Vinnie Falco (vinnie dot falco at gmail dot com)

    Distributed under the Boost Software License, Version 1.0. (See accompanying
    file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
]

[library NuDB
    [quickbook 1.6]
    [copyright 2015 - 2016 Vinnie Falco]
    [purpose C++ Library]
    [license
        Distributed under the Boost Software License, Version 1.0.
        (See accompanying file LICENSE_1_0.txt or copy at
        [@http://www.boost.org/LICENSE_1_0.txt])
    ]
    [authors [Falco, Vinnie]]
    [category template]
    [category generic]
]

[template mdash[] '''&mdash; ''']
[template indexterm1[term1] '''<indexterm><primary>'''[term1]'''</primary></indexterm>''']
[template indexterm2[term1 term2] '''<indexterm><primary>'''[term1]'''</primary><secondary>'''[term2]'''</secondary></indexterm>''']

[variablelist
  [[
    [link nudb.overview Overview]
  ][
    An overview of features, requirements, and credits, plus
    rationale and design information.
  ]]
  [[
    [link nudb.example Example]
  ][
    An example that illustrates the use of NuDB.
  ]]
  [[
    [link nudb.usage Usage]
  ][
    An explanation of operations on the database.
  ]]
  [[
    [link nudb.ref Reference]
  ][
    Detailed class and function reference.
  ]]
  [[
    [link nudb.index Index]
  ][
    Book-style text index of the documentation.
  ]]
]

[section:overview Overview]

NuDB is an append only, key/value store specifically optimized for random
read performance on modern SSDs or equivalent high-IOPS decices. The most
common application for NuDB is content addressible storage where a
cryptographic digest of the data is used as the key. The read performance
and memory usage are independent of the size of the database. These are
some other features:

[heading History]

The first versions of rippled, the application behind the Ripple consensus
network, used SQLite as their back end for unstructured data. The
performance quickly became a limiting factor.

Then rippled then went through a series of back ends including LMDB, LevelDB, and
RocksDB. Each of these databases performed well at first, but as the data
size increased, memory usage increased and performance dropped off drastically.

The problem is caching. Each of these databases relies on some O(n) data
structure, such as a Bloom filter, to improve their performance. These work
well until the structures no longer fit in memory. In addition, many virtual
machines are memory constrained.

To address this issue, the developers performed a thought experiment -- if
you assume the data size is so large that no O(n) caching is effective, what
is the best read performance you could expect? They reached the following
conclusions:

1) Writes should not block reads.
2) Reads should be limited only by the SSD's IOPS limit.
3) A read for a non-present key should require one IOP.
4) A read for a present key whose data can be read in a single IOP should
only require two IOPs, one to figure out where it is and one to read it in.

NuDB is designed to come as close to this ideal as possible.

[heading Design]

NuDB uses three files to hold the data and indexes. The data file is append
only and contains sufficient information to rebuild the index. The index
file is random access and contains hash buckets. When an update is in
progress, a temporary journal file is used to roll the update back if
needed.

NuDB uses linear hashing to dynamically increase the number of buckets in
the index file as the data size grows. Bucket overflows are handled by
adding "overflow" records to the data file. Bucket overflows can be
minimized by increasing the number of buckets, leading to a size/speed
tradeoff. Typical databases keep the average bucket half full (or half
empty, depending on your point of view) resulting in spill records
accounting for less than 1% of reads.

Inserts are buffered in memory and appended to the data file immediately.
Updates to the index file are performed as an atomic operation. Fetch
operations retrieve records in the process of being modified from memory
during the update operation so that writes do not block fetches.

Before the index file is modified, a journal file is created to recover
consistency in the event of a crash during the update. The recovery process
will index all records written to the data file, so the aggregation of index
updates does not increase the time which a crash would result in loss of
data.

Iteration can be performed on the data file directly. Since it is append
only, there is no risk of other operations corrupting an iteration in
progress.

[heading Performance]

Writes do not block reads. Read rates are typically around 90% of the SSD's
IOPS limit. An average fetch for a non-present key typically requires fewer
than 1.01 IOPs. An average fetch for a present key requires fewer than 1.01
IOPs plus however many IOPs it takes to read the data.

[heading Applications]

Content addressable storage associates data with its cryptographic digest.
This type of storage is commonly used in decentralized blockchain applications.

Often these applications require following hash chains -- where one object
contains the hash of another object that ultimately leads to the object
desired. NuDB's low latency and high speed are particularly advantageous
in these kinds of applications.

NuDB is append only and does not support a delete operation. To support
retaining limited historical information, NuDB is often used in a dual
database configuration. One database is older and is read only, the other
is newer and is read/write. Periodically, the older database is discarded and
the newer database becomes the new read only database and a new read/write
database is created.

[endsect]



[section:example Example]

This complete program creates a database, opens the database, inserts several
key/value pairs, fetches the key/value pairs, closes the database, then erases
the database files. Source code for this program is located in the examples
directory.

```
#include <nudb/nudb.hpp>
#include <cstddef>
#include <cstdint>

int main()
{
    using namespace nudb;
    std::size_t constexpr N = 1000;
    using key_type = std::uint32_t;
    error_code ec;
    auto const dat_path = "db.dat";
    auto const key_path = "db.key";
    auto const log_path = "db.log";
    create<xxhasher>(
        dat_path, key_path, log_path,
        1,
        make_salt(),
        sizeof(key_type),
        block_size("."),
        0.5f,
        ec);
    store db;
    db.open(dat_path, key_path, log_path,
        16 * 1024 * 1024, ec);
    char data = 0;
    // Insert
    for(key_type i = 0; i < N; ++i)
        db.insert(&i, &data, sizeof(data), ec);
    // Fetch
    for(key_type i = 0; i < N; ++i)
        db.fetch(&i,
            [&](void const* buffer, std::size_t size)
        {
            // do something with buffer, size
        }, ec);
    db.close(ec);
    erase_file(dat_path);
    erase_file(key_path);
    erase_file(log_path);
}
```

[endsect]



[section:usage Usage]

[heading Files]

A database is represented by three files: the data file, the key file,
and the log file. Each file has a distinct header in a well known format.
The data file holds all of the key/value pairs and is serially iterable. The
key file holds a hash table indexing all of the contents in the data file.
The log file holds information used to roll the database back in the event
of a failure.

[heading Create/Open]

The [link nudb.ref.nudb__create create] function creates a new data file and key
file for a database with the specified parameters. The caller specifies
the hash function to use as a template argument, the file paths,
and the database constants:

[note
    Sample code and identifiers mentioned in this section are written
    as if the following declarations are in effect:
    ```
    #include <nudb/nudb.hpp>
    using namespace nudb;
    error_code ec;
    ```
]

```
create<xxhasher>(
    "nudb.dat",             // Path to data file
    "nudb.key",             // Path to key file
    "nudb.log",             // Path to log file
    1,                      // Application-defined constant
    make_salt(),            // A random integer
    4,                      // The size of keys
    block_size(".")         // Block size in key file
    0.5f                    // The load factor
    ec);
```

The application-defined constant is a 64-bit unsigned integer which the
caller may set to any value. This value can be retrieved from an open
database, where it will be equal to the value used at creation time. This
constant can be used for any purpose. For example, to inform the application
of what application-specific version was used to create the database.

The salt is a 64-bit unsigned integer used to prevent algorithmic complexity
attacks. Hash functions used during database operations are constructed with
the salt, providing an opportunity to permute the hash function. This feature
is useful when inserted database keys come from untrusted sources, such as the
network.

The key size is specified when the database is created, and cannot be changed.
All key files indexing the same data file will use the key size of the data
file.

The block size indicates the size of buckets in the key file. The best choice
for the block size is the natural sector size of the device. For most SSDs
in production today this is 4096, or less often 8192 or 16384. The function
[link nudb.ref.nudb__block_size block_size] returns the best guess of the block
size used by the device mounted at the specified path.

The load factor determines the target bucket occupancy fraction. There is
almost never a need to specify anything other than the recommended value of
0.5, which strikes the perfect balance of space-efficiency and fast lookup.

An open database is represented by objects of type
[link nudb.ref.nudb__basic_store basic_store], templated on the hasher. The type
alias [link nudb.ref.nudb__store store] represents a database using
[link nudb.ref.nudb__xxhasher xxhasher], the default hash function. To open
a database, declare a database object and then call the
[link nudb.ref.nudb__basic_store.open open] member function:

```
store db;
db.open("nudb.dat", "nudb.key", "nudb.log", ec);
```

When opening a database that was previously opened by a program that was
terminated abnormally, the implementation automatically invokes the
recovery process. This process restores the integrity of the database by
replaying the log file if it is present.

[heading Insert/Fetch]

Once a database is open, it becomes possible to insert new key/value pairs
and look them up. Insertions are straightforward:

```
db.insert(key, data, bytes, ec);
```

If the key already exists, the error is set to
[link nudb.ref.nudb__error.key_exists error::key_exists]. All keys in a NuDB
database must be unique. Multiple threads can call insert at the same time.
Internally however, insertions are serialized to present a consistent view
of the database to callers.

Retrieving a key/value pair if it exists is similary straightforward:

```
db.fetch(key,
    [&](void const* buffer, std::size_t size)
    {
        ...
    }, ec);
```

To give callers control over memory allocation strategies, the fetch
function takes a callback object as a parameter. The callback is invoked
with a pointer to the data and size, if the item exists in the database.
The callback can decide how to store this information, if at all.

[endsect]

[section Command Line Tool]

To allow administration, NuDB comes with the "nudb" command line tool,
which may be built using b2 or CMake. Files for the tool are located in
the "tools" directory. Once the tool is built, and located in your path,
execute this command for additional instructions:

```
nudb help
```

[endsect]

[section:ref Reference]
[xinclude quickref.xml]
[include types/File.qbk]
[include types/Hasher.qbk]
[include types/Progress.qbk]
[include reference.qbk]
[endsect]

[xinclude index.xml]
