rippled/src/ripple/peerfinder

PeerFinder

Introduction

The Ripple payment network consists of a collection of peers running the rippled software. Each peer maintains multiple outgoing connections and optional incoming connections to other peers. These connections are made over both the public Internet and private local area networks. This network defines a fully connected directed graph of nodes. Peers send and receive messages to other connected peers. This peer to peer network, layered on top of the public and private Internet, forms an overlay network.

Bootstrapping

When a peer comes online it needs a set of IP addresses to connect to in order to gain initial entry into the overlay in a process called bootstrapping. Once they have established an initial set of these outbound peer connections, they need to gain additional addresses to establish more outbound peer connections until the desired limit is reached. Furthermore, they need a mechanism to advertise their IP address to new or existing peers in the overlay so they may receive inbound connections up to some desired limit. And finally, they need a mechanism to provide inbound connection requests with an alternate set of IP addresses to try when they have already reached their desired maximum number of inbound connections.

PeerFinder is a self contained module that provides these services, along with some additional overlay network management services such as fixed slots and cluster slots.

Features

PeerFinder has these responsibilities

• Maintain a persistent set of endpoint addresses suitable for bootstrapping into the peer to peer overlay, ranked by relative locally observed utility.

• Send and receive protocol messages for discovery of endpoint addresses.

• Provide endpoint addresses to new peers that need them.

• Maintain connections to a configured set of fixed peers.

• Impose limits on the various slots consumed by peer connections.

• Initiate outgoing connection attempts to endpoint addresses to maintain the overlay connectivity and fixed peer policies.

• Verify the connectivity of neighbors who advertise inbound connection slots.

• Prevent duplicate connections and connections to self.

---

Concepts

Manager

The Manager is an application singleton which provides the primary interface to interaction with the PeerFinder.

Autoconnect

The Autoconnect feature of PeerFinder automatically establishes outgoing connections using addresses learned from various sources including the configuration file, the result of domain name lookups, and messages received from the overlay itself.

Callback

PeerFinder is an isolated code module with few external dependencies. To perform socket specific activities such as establishing outgoing connections or sending messages to connected peers, the Manager is constructed with an abstract interface called the Callback. An instance of this interface performs the actual required operations, making PeerFinder independent of the calling code.

Config

The Config structure defines the operational parameters of the PeerFinder. Some values come from the configuration file while others are calculated via tuned heuristics. The fields are as follows:

• autoConnect

  A flag indicating whether or not the Autoconnect feature is enabled.

• wantIncoming

  A flag indicating whether or not the peer desires inbound connections. When this flag is turned off, a peer will not advertise itself in Endpoint messages.

• listeningPort

  The port number to use when creating the listening socket for peer connections.

• maxPeers

  The largest number of active peer connections to allow. This includes inbound and outbound connections, but excludes fixed and cluster peers. There is an implementation defined floor on this value.

• outPeers

  The number of automatic outbound connections that PeerFinder will maintain when the Autoconnect feature is enabled. The value is computed with fractional precision as an implementation defined percentage of maxPeers subject to an implementation defined floor. An instance of the PeerFinder rounds the fractional part up or down using a uniform random number generated at program startup. This allows the outdegree of the overlay network to be controlled with fractional precision, ensuring that all inbound network connection slots are not consumed (which would it difficult for new participants to enter the network).

Here's an example of how the network might be structured with a fractional value for outPeers:

(Need example here)

Livecache

The Livecache holds relayed IP addresses that have been received recently in the form of Endpoint messages via the peer to peer overlay. A peer periodically broadcasts the Endpoint message to its neighbors when it has open inbound connection slots. Peers store these messages in the Livecache and periodically forward their neighbors a handful of random entries from their Livecache, with an incremented hop count for each forwarded entry.

The algorithm for sending a neighbor a set of Endpoint messages chooses evenly from all available hop counts on each send. This ensures that each peer will see some entries with the farthest hops at each iteration. The result is to expand a peer's horizon with respect to which overlay endpoints are visible. This is designed to force the overlay to become highly connected and reduce the network diameter with each connection establishment.

When a peer receives an Endpoint message that originates from a neighbor (identified by a hop count of zero) for the first time, it performs an incoming connection test on that neighbor by initiating an outgoing connection to the remote IP address as seen on the connection combined with the port advertised in the Endpoint message. If the test fails, then the peer considers its neighbor firewalled (intentionally or due to misconfiguration) and no longer forwards Endpoint messages for that peer. This prevents poor quality unconnectible addresses from landing in the caches. If the incoming connection test passes, then the peer fills in the Endpoint message with the remote address as seen on the connection before storing it in its cache and forwarding it to other peers. This relieves the neighbor from the responsibility of knowing its own IP address before it can start receiving incoming connections.

Livecache entries expire quickly. Since a peer stops advertising itself when it no longer has available inbound slots, its address will shortly after stop being handed out by other peers. Livecache entries are very likely to result in both a successful connection establishment and the acquisition of an active outbound slot. Compare this with Bootcache addresses, which are very likely to be connectible but unlikely to have an open slot.

Because entries in the Livecache are ephemeral, they are not persisted across launches in the database. The Livecache is continually updated and expired as Endpoint messages are received from the overlay over time.

Bootcache

The Bootcache stores IP addresses useful for gaining initial connections. Each address is associated with the following metadata:

• Valence

  A signed integer which represents the number of successful consecutive connection attempts when positive, and the number of failed consecutive connection attempts when negative. If an outgoing connection attempt to the corresponding IP address fails to complete the handshake the valence is reset to negative one. This harsh penalty is intended to prevent popular servers from forever remaining top ranked in all peer databases.

When choosing addresses from the boot cache for the purpose of establishing outgoing connections, addresses are ranked in decreasing order of valence. The Bootcache is persistent. Entries are periodically inserted and updated in the corresponding SQLite database during program operation. When rippled is launched, the existing Bootcache database data is accessed and loaded to accelerate the bootstrap process.

Desirable entries in the Bootcache are addresses for servers which are known to have high uptimes, and for which connection attempts usually succeed. However, these servers do not necessarily have available inbound connection slots. However, it is assured that these servers will have a well populated Livecache since they will have moved towards the core of the overlay over their high uptime. When a connected server is full it will return a handful of new addresses from its Livecache and gracefully close the connection. Addresses from the Livecache are highly likely to have inbound connection slots and be connectible.

For security, all information that contributes to the ranking of Bootcache entries is observed locally. PeerFinder never trusts external sources of information.

Slot

Each TCP/IP socket that can participate in the peer to peer overlay occupies a slot. Slots have properties and state associated with them:

State (Slot)

The slot state represents the current stage of the connection as it passes through the business logic for establishing peer connections.

• accept

  The accept state is an initial state resulting from accepting an incoming connection request on a listening socket. The remote IP address and port are known, and a handshake is expected next.

• connect

  The connect state is an initial state used when actively establishing outbound connection attempts. The desired remote IP address and port are known.

• connected

  When an outbound connection attempt succeeds, it moves to the connected state. The handshake is initiated but not completed.

• active

  The state becomes Active when a connection in either the Accepted or Connected state completes the handshake process, and a slot is available based on the properties. If no slot is available when the handshake completes, the socket is gracefully closed.

• closing

  The Closing state represents a connected socket in the process of being gracefully closed.

Properties (Slot)

Slot properties may be combined and are not mutually exclusive.

• Inbound

  An inbound slot is the condition of a socket which has accepted an incoming connection request. A connection which is not inbound is by definition outbound.

• Fixed

  A fixed slot is a desired connection to a known peer identified by IP address, usually entered manually in the configuration file. For the purpose of establishing outbound connections, the peer also has an associated port number although only the IP address is checked to determine if the fixed peer is already connected. Fixed slots do not count towards connection limits.

• Cluster

  A cluster slot is a connection which has completed the handshake stage, whose public key matches a known public key usually entered manually in the configuration file or learned through overlay messages from other trusted peers. Cluster slots do not count towards connection limits.

• Superpeer (forthcoming)

  A superpeer slot is a connection to a peer which can accept incoming connections, meets certain resource availaibility requirements (such as bandwidth, CPU, and storage capacity), and operates full duplex in the overlay. Connections which are not superpeers are by definition leaves. A leaf slot is a connection to a peer which does not route overlay messages to other peers, and operates in a partial half duplex fashion in the overlay.

Fixed Slots

Fixed slots are identified by IP address and set up during the initialization of the Manager, usually from the configuration file. The Logic will always make outgoing connection attempts to each fixed slot which is not currently connected. If we receive an inbound connection from an endpoint whose address portion (without port) matches a fixed slot address, we consider the fixed slot to be connected.

Cluster Slots

Cluster slots are identified by the public key and set up during the initialization of the manager or discovered upon receipt of messages in the overlay from trusted connections.

---

Algorithms

Connection Strategy

The Connection Strategy applies the configuration settings to establish desired outbound connections. It runs periodically and progresses through a series of stages, remaining in each stage until a condition is met

Stage 1: Fixed Slots

This stage is invoked when the number of active fixed connections is below the number of fixed connections specified in the configuration, and one of the following is true:

• There are eligible fixed addresses to try
• Any outbound connection attempts are in progress

Each fixed address is associated with a retry timer. On a fixed connection failure, the timer is reset so that the address is not tried for some amount of time, which increases according to a scheduled sequence up to some maximum which is currently set to approximately one hour between retries. A fixed address is considered eligible if we are not currently connected or attempting the address, and its retry timer has expired.

The PeerFinder makes its best effort to become fully connected to the fixed addresses specified in the configuration file before moving on to establish outgoing connections to foreign peers. This security feature helps rippled establish itself with a trusted set of peers first before accepting untrusted data from the network.

Stage 2: Livecache

The Livecache is invoked when Stage 1 is not active, autoconnect is enabled, and the number of active outbound connections is below the number desired. The stage remains active while:

• The Livecache has addresses to try
• Any outbound connection attempts are in progress

PeerFinder makes its best effort to exhaust addresses in the Livecache before moving on to the Bootcache, because Livecache addresses are highly likely to be connectible (since they are known to have been online within the last minute), and highly likely to have an open slot for an incoming connection (because peers only advertise themselves in the Livecache when they have open slots).

Stage 3: Bootcache

The Bootcache is invoked when Stage 1 and Stage 2 are not active, autoconnect is enabled, and the number of active outbound connections is below the number desired. The stage remains active while:

• There are addresses in the cache that have not been tried recently.

Entries in the Bootcache are ranked, with highly connectible addresses preferred over others. Connection attempts to Bootcache addresses are very likely to succeed but unlikely to produce an active connection since the peers likely do not have open slots. Before the remote peer closes the connection it will send a handful of addresses from its Livecache to help the new peer coming online obtain connections.

---

References

Much of the work in PeerFinder was inspired by earlier work in Gnutella:

Revised Gnutella Ping Pong Scheme
By Christopher Rohrs and Vincent Falco

Gnutella 0.6 Protocol: Sections:

• 2.2.2 Ping (0x00)
• 2.2.3 Pong (0x01)
• 2.2.4 Use of Ping and Pong messages
• 2.2.4.1 A simple pong caching scheme
• 2.2.4.2 Other pong caching schemes