Language support updates

- Unicode header IDs filter
- Change some links to use the new ID formula
  (It matches the GitHub-Flavored Markdown standard closely.)
- Build script changes

content/_snippets/tx-metadata-field-table.md

@@ -4,4 +4,4 @@
 | `DeliveredAmount`                       | [Currency Amount][] | _(May be omitted)_ For a [partial payment](partial-payments.html), this field records the amount of currency actually delivered to the destination. To avoid errors when reading transactions, instead use the `delivered_amount` field, which is provided for all Payment transactions, partial or not. |
 | `TransactionIndex`                      | Unsigned Integer    | The transaction's position within the ledger that included it. This is zero-indexed. (For example, the value `2` means it was the 3rd transaction in that ledger.) |
 | `TransactionResult`                     | String              | A [result code](transaction-results.html) indicating whether the transaction succeeded or how it failed. |
-| [`delivered_amount`](transaction-metadata.html#delivered-amount) | [Currency Amount][] | _(Omitted for non-Payment transactions)_ The [Currency Amount][] actually received by the `Destination` account. Use this field to determine how much was delivered, regardless of whether the transaction is a [partial payment](partial-payments.html). See [this description](transaction-metadata.html#delivered-amount) for details. [New in: rippled 0.27.0][] |
+| [`delivered_amount`](transaction-metadata.html#delivered_amount) | [Currency Amount][] | _(Omitted for non-Payment transactions)_ The [Currency Amount][] actually received by the `Destination` account. Use this field to determine how much was delivered, regardless of whether the transaction is a [partial payment](partial-payments.html). See [this description](transaction-metadata.html#delivered_amount) for details. [New in: rippled 0.27.0][] |

content/concepts/consensus-network/amendments/known-amendments.md

@@ -313,7 +313,7 @@ Changes the result codes returned by two transaction types:
 
 Adds delivered amount to metadata for CheckCash transactions cashed for a flexible amount. (Has no effect unless the [Checks](#checks) amendment is enabled.)
 
-With this amendment enabled, transaction processing adds a `DeliveredAmount` field to the metadata of [CheckCash transactions][] for a variable amount (using the `DeliverMin` field). This change is written to the ledger data, resulting in a different ledger hash than would result from processing the transaction without this amendment. It does not affect the actual amounts delivered. Additionally, with this amendment enabled, the [tx method][] and [account_tx method][] return a [`delivered_amount` field](transaction-metadata.html#delivered-amount) for CheckCash transactions. (The `delivered_amount` field is calculated when you look up a transaction, and is not part of the data that is written to the ledger.)
+With this amendment enabled, transaction processing adds a `DeliveredAmount` field to the metadata of [CheckCash transactions][] for a variable amount (using the `DeliverMin` field). This change is written to the ledger data, resulting in a different ledger hash than would result from processing the transaction without this amendment. It does not affect the actual amounts delivered. Additionally, with this amendment enabled, the [tx method][] and [account_tx method][] return a [`delivered_amount` field](transaction-metadata.html#delivered_amount) for CheckCash transactions. (The `delivered_amount` field is calculated when you look up a transaction, and is not part of the data that is written to the ledger.)
 
 The fix1623 amendment has no effect on [CheckCash transactions][] for a fixed amount (using the `Amount` field) or any other transaction types.
 

content/concepts/payment-types/partial-payments.md

@@ -73,7 +73,7 @@ You can find the `delivered_amount` field in the following places:
 
 If a financial institution's integration with the XRP Ledger assumes that the `Amount` field of a Payment is always the full amount delivered, malicious actors may be able to exploit that assumption to steal money from the institution. This exploit can be used against gateways, exchanges, or merchants as long as those institutions' software does not process partial payments correctly.
 
-**The correct way to process incoming Payment transactions is to use [the `delivered_amount` metadata field](#the-delivered-amount-field),** not the `Amount` field. This way, an institution is never mistaken about how much it _actually_ received.
+**The correct way to process incoming Payment transactions is to use [the `delivered_amount` metadata field](#the-delivered_amount-field),** not the `Amount` field. This way, an institution is never mistaken about how much it _actually_ received.
 
 
 ### Exploit Scenario Steps
@@ -100,7 +100,7 @@ In the case of a merchant, the order of operations is slightly different, but th
 
 ### Further Mitigations
 
-Using [the `delivered_amount` field](#the-delivered-amount-field) when processing incoming transactions is enough to avoid this exploit. Still, additional proactive business practices can also avoid or mitigate the likelihood of this and similar exploits. For example:
+Using [the `delivered_amount` field](#the-delivered_amount-field) when processing incoming transactions is enough to avoid this exploit. Still, additional proactive business practices can also avoid or mitigate the likelihood of this and similar exploits. For example:
 
 - Add additional sanity checks to your business logic for processing withdrawals. Never process a withdrawal if the total balance you hold in the XRP Ledger does not match your expected assets and obligations.
 - Follow "Know Your Customer" guidelines and strictly verify your customers' identities. You may be able to recognize and block malicious users in advance, or pursue legal action against a malicious actor who exploits your system.

content/references/rippled-api/public-rippled-methods/path-and-order-book-methods/path_find.md

@@ -518,7 +518,7 @@ The request includes the following parameters:
 
 ### Response Format
 
-If a pathfinding request was successfully closed, the response follows the same format as the initial response to [`path_find create`](#path-find-create), plus the following field:
+If a pathfinding request was successfully closed, the response follows the same format as the initial response to [`path_find create`](#path_find-create), plus the following field:
 
 | `Field`  | Type    | Description                                             |
 |:---------|:--------|:--------------------------------------------------------|
@@ -563,7 +563,7 @@ The request includes the following parameters:
 
 ### Response Format
 
-If a pathfinding request is open, the response follows the same format as the initial response to [`path_find create`](#path-find-create), plus the following field:
+If a pathfinding request is open, the response follows the same format as the initial response to [`path_find create`](#path_find-create), plus the following field:
 
 | `Field`  | Type    | Description                                             |
 |:---------|:--------|:--------------------------------------------------------|

content/references/rippled-api/transaction-formats/transaction-types/payment.md

@@ -100,7 +100,7 @@ A partial payment allows a payment to succeed by reducing the amount received. P
 
 A partial payment is any [Payment transaction][] with the **tfPartialPayment** flag enabled. A partial payment can be successful if it delivers any positive amount greater than or equal to its `DeliverMin` field (or any positive amount at all if `DeliverMin` is not specified) without sending more than the `SendMax` value.
 
-The [`delivered_amount`](transaction-metadata.html#delivered-amount) field of a payment's metadata indicates the amount of currency actually received by the destination account.
+The [`delivered_amount`](transaction-metadata.html#delivered_amount) field of a payment's metadata indicates the amount of currency actually received by the destination account.
 
 For more information, see the full article on [Partial Payments](partial-payments.html).
 

content/tutorials/get-started/look-up-transaction-results.md

@@ -208,7 +208,7 @@ A [Payment transaction][] can represent a direct XRP-to-XRP transaction, a [cros
 
 XRP amounts are tracked in the `Balance` field of `AccountRoot` objects. (XRP can also exist in [Escrow objects](escrow-object.html) and [PayChannel objects](paychannel.html), but Payment transactions cannot affect those.)
 
-You should always use [the delivered_amount field](partial-payments.html#the-delivered-amount-field) to see how much a payment delivered.
+You should always use [the delivered_amount field](partial-payments.html#the-delivered_amount-field) to see how much a payment delivered.
 
 If the payment contains a `CreatedNode` of LedgerEntryType `AccountRoot`, that means the payment [funded a new account](accounts.html#creating-accounts) in the ledger.
 

content/tutorials/manage-the-rippled-server/configuration/run-rippled-as-a-validator.md

@@ -264,7 +264,7 @@ Here are some methods you can use to verify that your validator has a healthy co
 
 - Use the [`server_info`](server_info.html) command to return some basic information about your validator. The `server_state` should be set to `proposing`. It may also be set to `full` or `validating`, but only for a few minutes before moving into `proposing`.
 
-    If the `server_state` does not spend the majority of its time set to `proposing`, it may be a sign that your validator is unable to fully participate in the XRP Ledger network. For more information about server states and using the `server_info` endpoint to diagnose issues with your validator, see [`rippled` Server States](rippled-server-states.html) and [Get the `server_info`](diagnosing-problems.html#get-the-server-info).
+    If the `server_state` does not spend the majority of its time set to `proposing`, it may be a sign that your validator is unable to fully participate in the XRP Ledger network. For more information about server states and using the `server_info` endpoint to diagnose issues with your validator, see [`rippled` Server States](rippled-server-states.html) and [Get the `server_info`](diagnosing-problems.html#get-the-server_info).
 
 - Use the [`validators`](validators.html) command to return the current list of published and trusted validators used by the validator. Ensure that the `validator_list_expires` value is either `never` or not expired or about to expire.
 

content/tutorials/manage-the-rippled-server/installation/capacity-planning.ja.md

@@ -32,7 +32,7 @@ Rippleでは、お使いのRAMでサポートできる最大のノードサイ
 
 `large`も`[node_size]`の値として有効ですが、実際に使用するとほとんどの環境で`huge`よりもパフォーマンスが悪くなります。Rippleでは、`large`の代わりに、常に`huge`を使用することを推奨します。
 
-`node_size`パラメーターを無効な値に設定すると、[サーバーは起動しません](server-wont-start.html#bad-node-size-value)。
+`node_size`パラメーターを無効な値に設定すると、[サーバーは起動しません](server-wont-start.html#bad-node_size-value)。
 
 
 ### ノードDBタイプ

content/tutorials/manage-the-rippled-server/installation/capacity-planning.md

@@ -32,7 +32,7 @@ To tune your server, it may be useful to start with `tiny` and increase the size
 
 Although `large` is also a legal value for `[node_size]`, in practice it performs worse than `huge` in most circumstances. Ripple recommends always using `huge` instead of `large`.
 
-If you set the `node_size` parameter to an invalid value, the [server fails to start](server-wont-start.html#bad-node-size-value).
+If you set the `node_size` parameter to an invalid value, the [server fails to start](server-wont-start.html#bad-node_size-value).
 
 
 ### Node DB Type

dactyl-config.yml

@@ -26,7 +26,7 @@ filter_paths:
 
 default_filters:
     - multicode_tabs
-    - standardize_header_ids
+    - unicode_header_ids
     - buttonize
     - callouts
     - badges

tool/build_all_langs.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+## Run from the root directory of the repo to build all languages
+## TODO: read the `-o` / `--out` argument and adapt it for non-en languages
+
+dactyl_build --vars "$dactyl_vars"
+dactyl_build -t ja -o out/ja --vars "$dactyl_vars"

tool/filter_unicode_header_ids.py

@@ -0,0 +1,62 @@
+################################################################################
+## Unicode-friendly header IDs                                                ##
+## Author: Rome Reginelli                                                     ##
+## Copyright: Ripple Labs, Inc. 2016-2019                                     ##
+## License: MIT https://github.com/ripple/xrpl-dev-portal/blob/master/LICENSE ##
+##                                                                            ##
+## Changes the header ID formula to accept more unicode characters, so        ##
+## non-English, non-Latin-script headers can have nice IDs. Closely matches   ##
+## the behavior of GitHub-Flavored Markdown.                                  ##
+################################################################################
+import re
+
+## HTML5's 'id' attribute requirements are:
+## - Must be at least one character in length.
+## - Must not contain ASCII whitespace.
+## - Must be unique within the document.
+## - No other requirements. (HTML4 had stricter requirements.)
+
+## Python-Markdown's default ID formula (as of v3.1.1) is:
+## 1. normalize unicode to NFKD form (so it can strip accent marks)
+## 2. remove all non-ASCII characters (this is the problem for us)
+## 3. remove leading/trailing whitespace
+## 4. convert to lowercase
+## 5. remove everything except space, hyphen, alphanumerics, and underscores
+## 6. replace each block of consecutive whitespace/hyphens with a single hyphen
+## 7. add _1, _2 etc. suffixes to non-unique IDs
+## Later, the dev portal uses a filter to replace underscores with dashes.
+
+## The new formula is similar, but has some important differences:
+## 1. No Unicode normalization. We'd rather leave things as-is.
+## 2. Leave the string as unicode throughout.
+## 5. Keep hyphens, unicode "space" and unicode "word" characters (includes
+##    underscores). Replace everything else with spaces, then strip trailing
+##    spaces.
+## 7. Add -1, -2 suffixes to non-unique IDs.
+## The end result is that this standard closely matches GitHub-flavored
+##     Markdown in almost all cases. (The exceptions are cases where GFM makes
+##     invalid empty IDs, for example with emoji headers...)
+
+def idify(utext):
+    """Make a string ID-friendly (but more unicode-friendly)"""
+    utext = re.sub(r'[^\w\s-]', '', utext).strip().lower()
+    utext = re.sub(r'[\s-]+', '-', utext)
+    if not len(utext):
+        # Headers must be non-empty
+        return '_'
+    return utext
+
+def filter_soup(soup, **kwargs):
+    """Generate new IDs for all headers"""
+    uniqIDs = {}
+    headers = soup.find_all(name=re.compile("h[0-9]"))
+    for h in headers:
+        new_id = idify(h.get_text())
+        if new_id not in uniqIDs.keys():
+            uniqIDs[new_id] = 0
+        else:
+            # not unique, append -1, -2, etc. to this instance
+            uniqIDs[new_id] += 1
+            new_id = "{id}-{n}".format(id=new_id, n=uniqIDs[new_id])
+
+        h["id"] = new_id

tool/jenkins.sh

@@ -20,4 +20,7 @@ fi
 set -e
 
 tool/conflictmarkers.sh
-tool/all-target-link-checker.sh --vars "$dactyl_vars"
+
+## Build all languages, then run the link checker once
+tool/build_all_langs.sh
+dactyl_link_checker -q