Install Doxygen
Download the following zip files from xsltproc (Alternate download: ftp://ftp.zlatkovic.com/libxml/), and extract the bin\ folder contents into any folder in your path.

Download Boost
Extract the compressed file contents to your (new) $BOOST_ROOT location.
Open a command prompt or shell in the $BOOST_ROOT.
./bootstrap.bat
(Optional, if you also plan to build rippled) ./bjam.exe --toolset=msvc-14.0 --build-type=complete variant=debug,release link=static runtime-link=static address-model=64 stage
If it is not already there, add your $BOOST_ROOT to your environment $PATH.

Use homebrew to install: brew install doxygen. The executable will be installed in /usr/local/bin which is already in your path.
Alternatively, install from here: doxygen. You'll then need to make doxygen available to your command line. You can do this by adding a symbolic link from /usr/local/bin to the doxygen executable. For example, $ ln -s /Applications/Doxygen.app/Contents/Resources/doxygen /usr/local/bin/doxygen

Install Boost
Extract the compressed file contents to your (new) $BOOST_ROOT location.
Open a command prompt or shell in the $BOOST_ROOT.
$ ./bootstrap.bat
(Optional, if you also plan to build rippled) $ ./b2 toolset=clang threading=multi runtime-link=static link=static cxxflags="-stdlib=libc++" linkflags="-stdlib=libc++" adress-model=64
If it is not already there, add your $BOOST_ROOT to your environment $PATH. This makes the b2 command available to the command line.
That should be all that's required. In OS X 10.11, at least, libxml2 and libxslt come pre-installed.

Under construction

Setup project submodules

Do it

From the rippled root folder:

cd docs
./makeqbk.sh && b2

The output will be in docs/html.