From f05e24e3566a0dd53542e56ab44e15413767c6f3 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Fri, 21 Feb 2020 19:09:27 -0800 Subject: [PATCH 1/4] Rate limiting: incomplete draft --- .../api-conventions/rate-limiting.md | 43 +++++++++++++++++++ .../api-conventions/response-formatting.md | 1 + dactyl-config.yml | 11 +++++ 3 files changed, 55 insertions(+) create mode 100644 content/references/rippled-api/api-conventions/rate-limiting.md diff --git a/content/references/rippled-api/api-conventions/rate-limiting.md b/content/references/rippled-api/api-conventions/rate-limiting.md new file mode 100644 index 0000000000..d5052b9c54 --- /dev/null +++ b/content/references/rippled-api/api-conventions/rate-limiting.md @@ -0,0 +1,43 @@ +# Rate Limiting + +The `rippled` server limits the rate at which API clients can make requests on public APIs. Rate limiting is based on the IP address of the client, so clients behind [network address translation](https://en.wikipedia.org/wiki/Network_address_translation) share a limit based on their public IP address. + +**Tip:** Rate limiting does not apply when the client is connected [as an admin](get-started-with-the-rippled-api.html#admin-access). + +When a client is approaching the rate limit, the server adds the field `"warning": "load"` at the top level of an [API response](response-formatting.html). This warning is not added to every response, but the server may send several such warnings before it disconnects a client. + +If a client goes past the rate limit, the server disconnects that client and does not serve further requests from the client's API address for a while. The disconnect message is different for the WebSocket and JSON-RPC APIs. + +## WebSocket API Disconnect Message + +For a WebSocket API connection, the server closes the connection and provides a close message and code with the close message. The way you access these messages depends on your WebSocket client implementation. For example, using the [Node.js ws library](https://github.com/websockets/ws), the following code prints the close reason when disconnected: + +```js +const WebSocket = require('ws') +const ws = new WebSocket('ws://localhost:6007/') +ws.on('close', (code,reason) => { + console.log("Disconnected. \ncode: ", code, "\nreason: ", reason) +}) + +// If rate limited, prints: +// Disconnected. +// code: 1008 +// reason: threshold exceeded +``` + +If the connection is closed because of rate limiting, the close code is `1008` and the reason is the string `threshold exceeded`. + +## JSON-RPC Rate Limited Error + +For a JSON-RPC API request, the server rejects a request with a ***TODO*** error when the client is over the rate limit. For example: + +```json +TODO +``` + +## Rate Per Request +[[Source]](https://github.com/ripple/rippled/blob/master/src/ripple/resource/Fees.h "Source") + +The server calculates a client's usage rate based on the number of requests made over time, and weighs different types of requests differently based on approximately how much work the server must do to serve them. Follow-up messages from the server for the [subscribe method][] and [path_find method][] also count towards a client's usage rate. + +The usage rate drops off exponentially over time, so a client that does not make requests automatically has its access restored after a period of seconds to minutes. diff --git a/content/references/rippled-api/api-conventions/response-formatting.md b/content/references/rippled-api/api-conventions/response-formatting.md index 3445a42a40..b5fb9ff9bb 100644 --- a/content/references/rippled-api/api-conventions/response-formatting.md +++ b/content/references/rippled-api/api-conventions/response-formatting.md @@ -11,6 +11,7 @@ The fields of a successful response include: | `result.status` | String | (JSON-RPC and Commandline) The value `success` indicates the request was successfully received and understood by the server. | | `type` | String | (WebSocket only) The value `response` indicates a successful response to a command. [Asynchronous notifications](subscribe.html) use a different value such as `ledgerClosed` or `transaction`. | | `result` | Object | The result of the query; contents vary depending on the command. | +| `warning` | String | _(May be omitted)_ If this field is provided, the value is the string `load`. This means the client is approaching the [rate limiting](rate-limiting.html) threshold where the server will disconnect this client. | ## Example Successful Response diff --git a/dactyl-config.yml b/dactyl-config.yml index 1bbaa0f92b..7863167c44 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -2985,6 +2985,17 @@ pages: targets: - ja + - md: references/rippled-api/api-conventions/rate-limiting.md + html: rate-limiting.html + funnel: Docs + doc_type: References + supercategory: rippled API + category: API Conventions + blurb: Information on how public APIs limit clients from making too many requests. + targets: + - en + - ja + - md: references/rippled-api/api-conventions/request-formatting.md html: request-formatting.html funnel: Docs From 0a763faad7c0632227d7730b568725170ccfb4fb Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 25 Feb 2020 09:51:40 -0800 Subject: [PATCH 2/4] Rate limiting: JSON-RPC info, log message --- .../rippled-api/api-conventions/rate-limiting.md | 8 +++++--- .../troubleshooting/understanding-log-messages.md | 14 ++++++++++++++ 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/content/references/rippled-api/api-conventions/rate-limiting.md b/content/references/rippled-api/api-conventions/rate-limiting.md index d5052b9c54..ae4e43eab3 100644 --- a/content/references/rippled-api/api-conventions/rate-limiting.md +++ b/content/references/rippled-api/api-conventions/rate-limiting.md @@ -29,10 +29,12 @@ If the connection is closed because of rate limiting, the close code is `1008` a ## JSON-RPC Rate Limited Error -For a JSON-RPC API request, the server rejects a request with a ***TODO*** error when the client is over the rate limit. For example: +For a JSON-RPC API request, the server responds with the HTTP status code **503 Service Unavailable** when the client is over the rate limit. This response has a text (not JSON) body stating that the server is overloaded: -```json -TODO +```text +503 Service Unavailable + +Server is overloaded ``` ## Rate Per Request diff --git a/content/tutorials/manage-the-rippled-server/troubleshooting/understanding-log-messages.md b/content/tutorials/manage-the-rippled-server/troubleshooting/understanding-log-messages.md index 2f6d8e958f..f0dc7bded2 100644 --- a/content/tutorials/manage-the-rippled-server/troubleshooting/understanding-log-messages.md +++ b/content/tutorials/manage-the-rippled-server/troubleshooting/understanding-log-messages.md @@ -58,6 +58,20 @@ A large number of these messages around the same time may indicate a problem, su - Your server may have been overloading the peer with requests, causing it to drop your server. +## Consumer entry dropped with balance at or above drop threshold + +The following log message indicates that a client to the server's public API has been dropped as a result of [rate limiting](rate-limiting.html): + +```text +2020-Feb-24 23:05:35.566312806 Resource:WRN Consumer entry 169.55.164.21 dropped with balance 15970 at or above drop threshold 15000 +``` + +The entry contains the IP address of the client that exceeded its rate limit, and the client's "balance", which is a score estimating the rate at which the client has been using the API. The threshold for dropping a client [hardcoded to a score of 15000](https://github.com/ripple/rippled/blob/06c371544acc3b488b9d9c057cee4e51f6bef7a2/src/ripple/resource/impl/Tuning.h#L34-L35). + +If you see frequent messages from the same IP address, you may want to block those IP addresses from your network to reduce the load on your server's public API. (For example, you may be able configure your firewall to block those IP addresses.) + +To avoid being dropped by rate limiting on your own server, [connect as an admin](get-started-with-the-rippled-api.html#admin-access). + ## InboundLedger 11 timeouts for ledger ```text From 8669ccb52ababd7bb7cb72ae095646d48af3325e Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Fri, 28 Feb 2020 16:33:17 -0800 Subject: [PATCH 3/4] Rate limiting: see also; copyediting --- .../api-conventions/rate-limiting.md | 19 ++++++++++++++++++- dactyl-config.yml | 3 ++- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/content/references/rippled-api/api-conventions/rate-limiting.md b/content/references/rippled-api/api-conventions/rate-limiting.md index ae4e43eab3..cf382b6f69 100644 --- a/content/references/rippled-api/api-conventions/rate-limiting.md +++ b/content/references/rippled-api/api-conventions/rate-limiting.md @@ -40,6 +40,23 @@ Server is overloaded ## Rate Per Request [[Source]](https://github.com/ripple/rippled/blob/master/src/ripple/resource/Fees.h "Source") -The server calculates a client's usage rate based on the number of requests made over time, and weighs different types of requests differently based on approximately how much work the server must do to serve them. Follow-up messages from the server for the [subscribe method][] and [path_find method][] also count towards a client's usage rate. +The server calculates a client's usage rate based on the number of requests made over time, and weighs different types of requests based on approximately how much work the server must do to serve them. Follow-up messages from the server for the [subscribe method][] and [path_find method][] also count towards a client's usage rate. The usage rate drops off exponentially over time, so a client that does not make requests automatically has its access restored after a period of seconds to minutes. + +## See Also + +- **Concepts:** + - [The `rippled` Server](the-rippled-server.html) + - [Software Ecosystem](software-ecosystem.html) +- **Tutorials:** + - [Getting Started with XRP Ledger APIs](get-started-with-the-rippled-api.html) + - [Troubleshooting rippled](troubleshoot-the-rippled-server.html) +- **References:** + - [rippled API Reference](rippled-api.html) + - [Error Formatting](error-formatting.html) + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/dactyl-config.yml b/dactyl-config.yml index 7863167c44..89c3bd90e0 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -285,7 +285,8 @@ targets: # Fix links from untranslated disable-master-key-pair.html. "transaction-basics.html#authorizing-transactions": "transaction-basics.html#トランザクションの承認" "transaction-common-fields.html#auto-fillable-fields": "transaction-common-fields.html#自動入力可能なフィールド" - + # Fix link from untranslated rate-limiting.html + "get-started-with-the-rippled-api.html#admin-access": "get-started-with-the-rippled-api.html" # The translated doc hasn't been updated to add the "Admin Access" section, but the details are mentioned near the top of the page - name: xrp-api-only lang: en From 1944beb32eb980fc999426161bf6f469dc83bcd5 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 3 Mar 2020 14:11:23 -0800 Subject: [PATCH 4/4] Rate limiting: copy-editing --- .../references/rippled-api/api-conventions/rate-limiting.md | 6 +++--- .../troubleshooting/understanding-log-messages.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/content/references/rippled-api/api-conventions/rate-limiting.md b/content/references/rippled-api/api-conventions/rate-limiting.md index cf382b6f69..d3aa0e6d5a 100644 --- a/content/references/rippled-api/api-conventions/rate-limiting.md +++ b/content/references/rippled-api/api-conventions/rate-limiting.md @@ -6,11 +6,11 @@ The `rippled` server limits the rate at which API clients can make requests on p When a client is approaching the rate limit, the server adds the field `"warning": "load"` at the top level of an [API response](response-formatting.html). This warning is not added to every response, but the server may send several such warnings before it disconnects a client. -If a client goes past the rate limit, the server disconnects that client and does not serve further requests from the client's API address for a while. The disconnect message is different for the WebSocket and JSON-RPC APIs. +If a client goes past the rate limit, the server disconnects that client and does not serve further requests from the client's API address for a while. The WebSocket and JSON-RPC APIs use different disconnect messages. ## WebSocket API Disconnect Message -For a WebSocket API connection, the server closes the connection and provides a close message and code with the close message. The way you access these messages depends on your WebSocket client implementation. For example, using the [Node.js ws library](https://github.com/websockets/ws), the following code prints the close reason when disconnected: +For the WebSocket API, the server closes the connection and provides a close message and code. The way you access these messages depends on your WebSocket client implementation. For example, using the [Node.js ws library](https://github.com/websockets/ws), the following code prints the close reason when disconnected: ```js const WebSocket = require('ws') @@ -25,7 +25,7 @@ ws.on('close', (code,reason) => { // reason: threshold exceeded ``` -If the connection is closed because of rate limiting, the close code is `1008` and the reason is the string `threshold exceeded`. +If the connection is closed because of rate limiting, the close code is `1008` and the close message is the string `threshold exceeded`. ## JSON-RPC Rate Limited Error diff --git a/content/tutorials/manage-the-rippled-server/troubleshooting/understanding-log-messages.md b/content/tutorials/manage-the-rippled-server/troubleshooting/understanding-log-messages.md index f0dc7bded2..cc230c2a78 100644 --- a/content/tutorials/manage-the-rippled-server/troubleshooting/understanding-log-messages.md +++ b/content/tutorials/manage-the-rippled-server/troubleshooting/understanding-log-messages.md @@ -55,7 +55,7 @@ Losing connections from time to time is normal for any peer-to-peer network. **O A large number of these messages around the same time may indicate a problem, such as: - Your internet connection to one or more specific peers was cut off. -- Your server may have been overloading the peer with requests, causing it to drop your server. +- Your server may have been overloading the peer with requests, causing the peer to disconnect your server. ## Consumer entry dropped with balance at or above drop threshold @@ -66,9 +66,9 @@ The following log message indicates that a client to the server's public API has 2020-Feb-24 23:05:35.566312806 Resource:WRN Consumer entry 169.55.164.21 dropped with balance 15970 at or above drop threshold 15000 ``` -The entry contains the IP address of the client that exceeded its rate limit, and the client's "balance", which is a score estimating the rate at which the client has been using the API. The threshold for dropping a client [hardcoded to a score of 15000](https://github.com/ripple/rippled/blob/06c371544acc3b488b9d9c057cee4e51f6bef7a2/src/ripple/resource/impl/Tuning.h#L34-L35). +The entry contains the IP address of the client that exceeded its rate limit, and the client's "balance", which is a score estimating the rate at which the client has been using the API. The threshold for dropping a client is [hardcoded to a score of 15000](https://github.com/ripple/rippled/blob/06c371544acc3b488b9d9c057cee4e51f6bef7a2/src/ripple/resource/impl/Tuning.h#L34-L35). -If you see frequent messages from the same IP address, you may want to block those IP addresses from your network to reduce the load on your server's public API. (For example, you may be able configure your firewall to block those IP addresses.) +If you see frequent messages from the same IP address, you may want to block those IP addresses from your network to reduce the load on your server's public API. (For example, you may be able to configure your firewall to block those IP addresses.) To avoid being dropped by rate limiting on your own server, [connect as an admin](get-started-with-the-rippled-api.html#admin-access).