parse_pages - merge githubify; update readme

README.md

@@ -1,23 +1,28 @@
 ripple-dev-portal
 =================
 
-The [Ripple Developer Portal](https://dev.ripple.com) is the authoritative source for Ripple documentation, including the `rippled` server, Ripple-REST API, Gatewayd, and other Ripple software.
+The [Ripple Developer Portal](https://dev.ripple.com) is the authoritative source for Ripple documentation, including the `rippled` server, RippleAPI, the Ripple Data API, and other Ripple software.
 
 
 Repository Layout
 -----------------
 
-The pages in this portal are generated from the markdown files in the [content/](content/) folder.
+The HTML pages in this portal are generated from the markdown files in the [content/](content/) folder.
 
-You can run your own local copy of the Dev Portal by hosting it in a web server, such as Apache. By default, the Dev Portal uses static HTML pages and client-side JavaScript (specifically, [Flatdoc](http://ricostacruz.com/flatdoc/)) to parse and display the Markdown content on the fly.
+The [tool/](tool/) folder contains tools and templates for generating the HTML files in the top level. The `parse_pages.py` script (requires Python 3, [Jinja 2](http://jinja.pocoo.org/), and various pip modules) uses the templates and the [pages.json](tools/pages.json) file to generate all the HTML files on the top level. Files with "md" properties are considered documentation pages and appear in the appropriate lists. In general, `parse_pages.py` assumes you are running it with `tool/` as the current working directory.
 
-The [tool/](tool/) folder contains tools and templates for generating the HTML files in the top level. The `parse_pages.py` script (Python 3, [Jinja 2](http://jinja.pocoo.org/), and various pip modules required) uses the templates and the [pages.json](tools/pages.json) file to generate all the HTML files on the top level. Files with "md" properties are considered documentation pages and appear in the appropriate lists. There are a few modes for generating pages:
+There are a few modes for generating pages:
 
-* The default, dynamic pages. Generates thin HTML wrappers for each page and uses Flatdoc to load and parse the contents. Great for development since a refresh automatically picks up changes to the md files.
-* Pre-compiled pages. Parses the Markdown all at once and puts the contents directly into the HTML body of each file. Still experimental.
-* PDF output using [Prince](http://www.princexml.com/). Still experimental.
+* By default, parses the Markdown and generates all the HTML files for a locally-hosted environment.
+    * Local markdown files are pre-processed using Jinja so you they can have conditional logic
+    * You can also remotely reference a markdown file via URL. Such files are not pre-processed, though.
+* Build for a specific target (e.g. `./parse_pages.py -t ripple.com`). Replaces local links with alternate versions for the target, uncomments multi-code divs (if target needs them for multi-code tabs), and performs various other cleanup.
+* Watch for changes and re-build the HTML files automatically when changes occur. Example: `./parse_pages.py -w`
+* Output empty HTML files and use [Flatdoc](ricostacruz.com/flatdoc/) to parse the markdown in-browser. Requires that you host the dev portal in a web server (e.g. Apache), not just open the HTML files in a browser. Example: `./parse_pages.py --flatdoc`
+* Output a modified markdown file for GitHub (e.g. for sending changes back upstream). (Builds for a target, but stops at markdown instead of HTML.) e.g. `./parse_pages.py -t ripple.com -g data_v2.md > /tmp/data_v2_githubified.md`
+* PDF output using [Prince](http://www.princexml.com/). Use `./parse_pages.py --pdf some_out_file.pdf`. Experimental.
 
-To add more pages to the Dev Portal, simply modify `pages.json` and then run `parse_pages.py` again.
+To add more pages to the Dev Portal, simply modify `pages.json` and then run `parse_pages.py` again. For full usage, run `parse_pages.py --help`.
 
 
 Contributing
@@ -25,5 +30,5 @@ Contributing
 
 The Developer Portal welcomes outside contributions, especially to the documentation contents. If you have any corrections, improvements, or expansions of the portal, please contribute pull requests to the **gh-pages** branch.
 
-Contributions become copyright Ripple Labs and are provided under the MIT [LICENSE](LICENSE).
+Contributions become copyright Ripple, Inc. and are provided under the MIT [LICENSE](LICENSE).
 

tool/githubify.py

@@ -1,49 +0,0 @@
-#!/bin/env python3
-"""
-Convert between Dev-Portal-ready markdown and Github-ready markdown. Has two modes:
-Normal - Convert from Dev Portal format to Github:
-    * Comments out divs that the dev portal uses to generate tabs, so that Github parses the markdown inside
-    * Replaces local relative links with absolute links to ripple.com
-Reverse - Convert from Github format to Dev Portal:
-    * Uncomments multicode divs
-    * Replaces absolute links with local ones that will work even offline.
-    
-Usage: githubify.py ripplerest_api.md > readme.md
-
-You may still want to do some manual editing (for example, to add Travis status icons to your readme)
-"""
-
-import sys
-
-def convert_page(text, reverse):
-    replacements = {
-        "<div class='multicode'>": "<!-- <div class='multicode'> -->",
-        "</div>": "<!-- </div> -->",
-        "(rest-api-tool.html": "(https://ripple.com/build/rest-tool",
-        "(transactions.html": "(https://ripple.com/build/transactions",
-        "(rippled-apis.html": "(https://ripple.com/build/rippled-apis",
-        "(charts-api-tool.html": "(https://ripple.com/build/charts-api-tool/",
-        }
-    
-    for (k,v) in replacements.items():
-        if reverse:
-            text = text.replace(v,k)
-        else:
-            text = text.replace(k,v)
-        
-    return text
-    
-if __name__ == "__main__":
-    if len(sys.argv) < 2 or len(sys.argv) > 3:
-        exit("usage: %s infile [reverse]" % sys.argv[0])
-    
-    if len(sys.argv) == 3 and sys.argv[2].lower() == "reverse":
-        reverse = True
-    else:
-        reverse = False
-    
-    with open(sys.argv[1]) as f:
-        text = f.read()
-        text = convert_page(text, reverse)
-        print(text)
-

tool/parse_pages.py

@@ -121,9 +121,52 @@ def parse_markdown(md, target=DEFAULT_TARGET, pages=None):
     html2 = str(soup)
     print("done")
     return html2
+
+MARKDOWN_LINK_REGEX = re.compile(r"(\[([^\]]+)\]\(([^:)]+)\)|\[([^\]]+)\]:\s*(\S+)$)", re.MULTILINE)
+def githubify_markdown(md, target=DEFAULT_TARGET, pages=None):
+    if not pages:
+        pages = get_pages()
+
+    class MDLink:
+        def __init__(self, fullmatch, label, url, label2, url2):
+            self.fullmatch = fullmatch
+            if label:
+                self.label = label
+                self.url = url
+                self.is_reflink = False
+            elif label2:
+                self.label = label2
+                self.url = url2
+                self.is_reflink = True
+        
+        def to_markdown(self):
+            s = "["
+            s += self.label
+            s += "]"
+            if self.is_reflink:
+                s += ": "
+                s += self.url
+            else:
+                s += "("
+                s += self.url
+                s += ")"
+            return s
+
+    links = [MDLink(*m) for m in MARKDOWN_LINK_REGEX.findall(md)]
     
-def get_pages(target=None):
-    print("reading page manifest...")
+    for link in links:
+        for page in pages:
+            if target in page:
+                #There's a replacement link for this
+                local_url = page["html"]
+                target_url = page[target]
+                if link.url[:len(local_url)] == local_url:
+                    link.url = link.url.replace(local_url, target_url)
+                    md = md.replace(link.fullmatch, link.to_markdown())
+    
+    return md
+    
+def get_pages(target=None,verbose=True):
     with open(PAGE_MANIFEST_FILE) as f:
         pages = json.load(f)
     
@@ -132,7 +175,6 @@ def get_pages(target=None):
         pages = [page for page in pages
                  if "targets" not in page or target in page["targets"]
                 ]
-    print("done")
     return pages
 
 def render_pages(precompiled, target=DEFAULT_TARGET):
@@ -271,6 +313,15 @@ def make_pdf(outfile):
     print("generating PDF: running ", " ".join(args),"...")
     prince_resp = subprocess.check_output(args, universal_newlines=True)
 
+
+def githubify(md_file_name, target=DEFAULT_TARGET):
+    filein = os.path.join(CONTENT_PATH, md_file_name)
+    with open(filein, "r") as f:
+        md = f.read()
+    pages = get_pages()
+    print(githubify_markdown(md, target=target, pages=pages))
+
+
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(
             description='Generate static site from markdown and templates.')
@@ -280,10 +331,15 @@ if __name__ == "__main__":
                        help="Watch for changes and re-generate the files. This runs until force-quit.")
     parser.add_argument("--pdf", type=str, 
             help="Generate a PDF, too. Requires Prince.")
+    parser.add_argument("-g","--githubify", type=str, help="Output md prepared for GitHub")
     parser.add_argument("--target", "-t", type=str, default=DEFAULT_TARGET)
     args = parser.parse_args()
     pre_parse = not args.flatdoc
     
+    if args.githubify:
+        githubify(args.githubify, args.target)
+        exit(0)
+    
     if args.pdf:
         if args.pdf[-4:] != ".pdf":
             exit("PDF filename must end in .pdf")