Documentation of Code - Documentation as Code

Goals

1. Documentation-Of-Code: documentation that helps the user to do what they need i.e. explanation of the APIs supplemented with examples, howtos with associated diagrams.

2. Documentation-As-Code: documentation that is current i.e. all documentation and diagram source maintained in GIT/Bitbucket with the source code. In addition the toolset is maintained as code.

Sub-goals

1. Bring documentation closer to the code so that it is in sync with the code:

   1. Code is developed consistent with the documentation

   2. Documentation diagrams and description is current and correct

2. Remove the initial friction to create diagrams - by auto-generating them

3. Diagram source text files are version-controlled in GIT in relevant source code repo

4. Consistent diagrams - that can be easily interlinked

   1. Use Themed Templates

   2. Use a common icon-set

   3. Hyper-Linkable diagrams

5. Different types of documentation integrated i.e. make it easy to combine documentation into one repo - exportable to different views.

Use-Case

Given a code base consisting of:

1. Code written in C, C++

2. Diagrams in Plantuml, Gliffy, various image formats (png, jpeg,…)

3. Documentation written in a Wiki or Markup language or Office document

We want to

1. To create documentation for users of the software - developers, and end-users.

2. For the documention to be pretty and modern looking

3. To support documentation extraction from C, C++ code and also standalone descriptive text files that include PlantUML diagrams.

What’s Out There

Github

Github supports several markup languages for documentation; the most popular ones are Markdown (.md) and reStructuredText (.rst).

Markup supported by GitHub

Markup	Associated with / Origins	Example Projects Using
Markdown	Web content in general	Most Github projects. Static Site Generators with content in GitHub e.g. Gatsby
RST reStructured Text	Python	Most Python projects e.g. Scrapy. The Linux kernel as described here
POD Plain Old Documentation	Perl	Perl, OpenSSL
ASCIIDoc	General documentation
Doxygen	C, C++.	Most C, C++ projects
Rdoc	Ruby

Bitbucket

Bitbucket supports Markdown, reStructured Text, Textile and Creole.

However a plugin may be required e.g. ReStructuredText Viewer for Bitbucket plugin to render an rst file when viewed in Bitbucket.

Confluence

Today we use Confluence which makes it easy for everyone to add content or provide comments on it. However, the content becomes stale and is disconnected from the code and software teams.

Many community projects use Confluence for user documentation e.g. Apache https://cwiki.apache.org/confluence/display/HTTPD/ which is a wiki containing user-contributed recipes, tips, and tricks for the Apache HTTP Server.

Important

1. We work with many teams that don’t have Confluence access (VFC, PCI and EMV labs, other subsiduaries/contractors). The information about the s/w should travel with the s/w.

Why Sphinx and reStructuredText

See these 2 good articles that answer this question:

1. https://varnish-cache.org/docs/trunk/phk/sphinx.html:

2. https://devblogs.microsoft.com/cppblog/clear-functional-c-documentation-with-sphinx-breathe-doxygen-cmake/

Note

The Linux kernel uses Sphinx and ReStructuredText as described here.

Overview

Overview of the Documentation System

Note

This diagram was created from PlantUML C4.

Tools

Tools Used

Tool	Details
Doxygen	Doxygen is the de facto standard tool for generating documentation from annotated C++ sources, but it also supports other popular programming languages. Runs native.
Sphinx	Sphinx is a tool that makes it easy to create intelligent and beautiful documentation. Sphinx uses reStructuredText as its markup language, and many of its strengths come from the power and straightforwardness of reStructuredText and its parsing and translating suite, the Docutils. Runs on Python.
Breathe	Breathe provides a bridge between the Sphinx and Doxygen documentation systems. It is an easy way to include Doxygen information in a set of documentation generated by Sphinx. Runs on Python.
Exhale	Automatic C++ library API documentation generator using Doxygen, Sphinx, and Breathe. Exhale revives Doxygen’s class / file hierarchies using reStructuredText for superior markup syntax / websites. Exhale provides a layer of automation, enabling launching Doxygen and generating the full website all from your conf.py. Runs on Python.
PlantUML	PlantUML creates UML and other diagrams from a simple text description. Runs on Java.

Setup Toolset

1. Linux host is used.

2. Source code will reside on host - but be shared into the Docker container to process with the tools.

3. Toolset will be installed in Docker container i.e. no tools installed/needed on host (except Docker).

4. Docker Command will be run as current user. (Docker Daemon always runs as the root user)

5. Docker Container will be run as current user i.e. not root which is the default.

6. Docker Container runs stateless and on demand.

Layout

The files layout is as per https://github.com/svenevs/exhale:

MyProject/
├── documentation # Doxygen stuff - optional - for using doxygen manually
│   ├── html/
│   ├── latex/
│   └── Doxyfile
│
├── docs/ #Sphinx stuff
│   ├── api/
│   │   └── library_root.rst
│   │
│   ├── conf.py # Sphinx configuration file
│   ├── index.rst # Sphinx index file
│   │
│   └── doxyoutput/
│       └── xml/ # Doxygen xml output - used as input to Sphinx
│           └── index.xml
│
├── include/ # source headers
│   └── h,hpp files
│
└── src/ # source C, C++ files
    └── c,cpp files

Pre-Requisites

Docker is installed.

Make Docker Accessible by Current User

Note

1. There’s 2 common ways to make Docker accessible by current user - see https://docs.docker.com/install/linux/linux-postinstall/.

2. Setfacl is used as an alternative to give more fine grain control.

Allow user to read/write the Unix socket used by docker.

sudo setfacl -m user:`whoami`:rw /var/run/docker.sock

Get Toolset

git clone ssh://git@bitbucket.MyCompany.com:7999/~chris_m11/docdac.git
cd ./docdac

Build Toolset from Dockerfile

./docker_build.sh

This takes several minutes e.g. ~4 minutes in timed example below (most of which is updating the apt repos)

time ./docker_build.sh
...
Successfully tagged docdac:latest
Successfully tagged docdac:ubuntu-1.0

real    4m12.616s
user    0m0.350s
sys     0m0.236s

docker_build.sh source

#!/bin/bash

docker build -t docdac:latest -t docdac:ubuntu-1.0 .

# This fixes ReadTheDocs Theme Bug where table text does not auto wrap for long lines.
# https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html 
mkdir ./docs/build/_static/
cp ./theme_overrides.css ./docs/build/_static/

Dockerfile source

FROM ubuntu:latest


# MAINTAINER="Crashedmind"


RUN mkdir -p /usr/share/man/man1 \ 
  && apt-get update \
  && apt-get install -y \
    wget \
    python3-pip \
    python3-dev \
    default-jre \
    graphviz \
    doxygen \
  && cd /usr/local/bin \
  && ln -s /usr/bin/python3 python \
  && pip3 install --no-cache --upgrade pip && pip install \
     sphinx_rtd_theme \ 
     sphinxcontrib-plantuml \
     exhale \ 
  && mkdir /usr/share/plantuml \
  && wget https://sourceforge.net/projects/plantuml/files/latest/download -O ./plantuml.jar \
  && mv ./plantuml.jar /usr/share/plantuml/ \
  && rm -rf /var/lib/apt/lists/* 

  
  RUN pip3 install --no-cache --upgrade  sphinxcontrib.yt  sphinxcontrib-confluencebuilder

  # since plantuml url is not the final file location, we need to explicitly save the file as "plantuml.jar" 
  # it would save as file "download" if we didn't         
  
  # /usr/share/man/man1 required so default-jre (Java Runtime required for Plantuml) installs

  # https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#run recommends to RUN 'rm -rf /var/lib/apt/lists/*' at end to reduce image size

  # Plantuml + Java content adds ~300MB to image size

#env setup
ENV GRAPHVIZ_DOT=/usr/bin/dot

WORKDIR /home/documentation

Ubuntu vs Alpine base

Ubuntu:latest is used as the base (64MB). While Alpine base is smaller (5MB) it is not used (even as a second Dockerfile option) for these reasons:

1. Consistency with Ubuntu development environment

2. Alpine packaging and packages not compatible with Ubuntu (apk vs deb)

3. Poor CVE tracking

Sanity Test Toolset

./docker_test.sh

Example Output

doxygen
1.8.13

sphinx-build 2.2.0

(0.000 - 502 Mo) 495 Mo - PlantUML Version 1.2019.09
(0.004 - 502 Mo) 495 Mo - GraphicsEnvironment.isHeadless() true
(0.004 - 502 Mo) 495 Mo - Forcing -Djava.awt.headless=true
(0.004 - 502 Mo) 495 Mo - java.awt.headless set as true
(0.004 - 502 Mo) 495 Mo - Forcing resource load on OpenJdk
(0.125 - 502 Mo) 490 Mo - Found 0 files
No diagram found

docker_test.sh source

#!/bin/bash

#test installed executables by running to show version of executable

echo "doxygen"
docker run -u `id -u $USER`:`id -g $USER` -i --rm -v $(pwd):/home/documentation -t docdac:ubuntu-1.0 doxygen -v

echo ""
docker run -u `id -u $USER`:`id -g $USER` -i --rm -v $(pwd):/home/documentation -t  docdac:ubuntu-1.0 sphinx-build --version

echo ""
docker run -u `id -u $USER`:`id -g $USER` -i --rm -v $(pwd):/home/documentation -t  docdac:ubuntu-1.0 java -jar /usr/share/plantuml/plantuml.jar -v

Transpile DoxyGen Output to Sphinx and add other Documentation

Configure Sphinx conf.py

Note

PlantUML better supports PNG than SVG (as in some of my SVG diagrams turned out weird whereas in PNG they were fine). I was originally thinking to default to SVG because it supports embedding hyperlinks.

conf.py source

# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# http://www.sphinx-doc.org/en/master/config

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#





# -- Project information -----------------------------------------------------

project = 'DOCDAC'
copyright = 'Crashedmind'


# Setup the breathe extension
breathe_projects = {
    "docdac": "./doxyoutput/xml"
}

# Breathe Configuration
breathe_default_project = "docdac"

# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
# https://www.sphinx-doc.org/en/master/usage/extensions/index.html
# sphinx.ext.autosectionlabel – Allow reference sections using its title

extensions = [ "breathe",
              "exhale",
              "sphinxcontrib.plantuml",
              "sphinx.ext.graphviz",
              "sphinx.ext.autosectionlabel",
              "sphinxcontrib.yt",
              "sphinx.ext.todo",
              "sphinxcontrib.confluencebuilder"
             
]


# Setup the exhale extension
exhale_args = {
    # These arguments are required
    "containmentFolder":     "./api",
    "rootFileName":          "library_root.rst",
    "rootFileTitle":         "Library API",
    "doxygenStripFromPath":  "..",
    # Suggested optional arguments
    "createTreeView":        True,
    # TIP: if using the sphinx-bootstrap-theme, you need
    # "treeViewIsBootstrap": True,
    "exhaleExecutesDoxygen": True,
    "exhaleDoxygenStdin":    "INPUT = ../include"
}

plantuml = 'java -jar /usr/share/plantuml/plantuml.jar -config /home/documentation/docs/plantuml/plantuml_cfg.puml' #location as per Dockerfile
#plantuml_output_format = 'svg' # svg format is used for images because it supports embedding hyperlinks. This causes some funnies so only enable per file basis as required

# Configure Graphviz
graphviz_output_format = 'svg'
#...


# 
confluence_publish = True
confluence_space_name = 'docdac'
# (for confluence cloud)
#confluence_server_url = 'https://confluence.verifone.com'
#confluence_server_url = 'https://confluence.verifone.com:8443'
#confluence_server_url = 'https://confluence.verifone.com:8443/'
#confluence_server_url = 'http://confluence.verifone.com'
# confluence_server_url = 'https://confluence.company.com:8443/display'
confluence_server_url = 'https://confluence.verifone.com:8443/display/docdac'

#  curl -u user:password https://confluence.verifone.com:8443/display/docdac > ~/tmp.html works to read/browse page

# confluence_proxy = 'https://confluence.verifone.com:8443/'


confluence_server_user = 'chris_m11'
# confluence_server_pass = 'password'
confluence_ask_password = True



# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']


# This fixes ReadTheDocs Theme Bug where table text does not auto wrap for long lines.
# https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html 
html_context = {
    'css_files': [
        '_static/theme_overrides.css',  # override wide tables in RTD theme
        ],
     }

# Tell sphinx what the primary language being documented is.
primary_domain = 'cpp'

# Tell sphinx what the pygments highlight language should be.
highlight_language = 'cpp'


#List any ToDos https://www.sphinx-doc.org/en/master/usage/extensions/todo.html
todo_include_todos = True

Add Source Code

include/hello-world.h source

/** 
 *  @file hello-world.h
 */


#pragma once

/**
 * A dummy struct
 */
struct Hellomessage {
  /**
  * message to use to say hello
  */
	char message [20];
} hellomessage;

/**
 * @brief	brief explanation of hello-world.c
 * @return 1
 */
int HelloWorld(void);

Add index.rst

index.rst source

Welcome to DOCDAC's documentation!
=======================================

    *Documentation-Of-Code Documentation-As-Code*

NOTE: See https://crashedmind.github.io/docdac-site/ for the rendered version.

.. tip ::
    Within 5 minutes, you can build this guide and all the diagrams in it, and the toolset, from source text files in GIT.

    .. code-block:: bash

        git clone git@github.com:Crashedmind/docdac.git #get the documentation and toolset (Dockerfile) source text files
        cd ./docdac
        ./docker_build.sh # build toolset form Docker file 
        ./sphinx_build.sh # build this html guide using the toolset 

    You can then use this as the basis to integrate your own DoxyGen content into a modern, powerful, good-looking documentation that includes easily generated diagrams. 
    Click "View page source" on pages to see how they were written.

    #. Create your own content 
    #. Build it using ./sphinx_build.sh
    #. Give users useful documentation they will enjoy reading




.. comment
    Sphinx is expecting 3 spaces under toctree i.e. it does not work with 4

.. toctree::
   :maxdepth: 2
   :caption: Document and Reference C/C++ Code

   readme
   api/library_root
   xref.rst
   
.. toctree::
   :maxdepth: 2   
   :caption: Create Diagrams with Plantuml

   plantuml
   plantuml_sprites
   plantuml_server
   c4
   plantuml_c4
   attack_trees

.. toctree::
   :maxdepth: 2
   :caption: Confluence
   
   confluence
   confluence_plantuml

.. toctree::
   :maxdepth: 2
   :caption: Github

   github

.. toctree::
   :maxdepth: 2
   :caption: Other Features
      
   todo
   
Index
==================

* :ref:`genindex`

Run Sphinx

./sphinx_build.sh

sphinx_build.sh source

#!/bin/bash

# By default we use png image format for plantuml diagrams because it is the most supported. 
# However, for diagrams with hyperlinks we use svg since it supports hyperlinks (png does not)
docker run -u `id -u $USER`:`id -g $USER` -i  --rm -v $(pwd):/home/documentation -t docdac:ubuntu-1.0 java -jar /usr/share/plantuml/plantuml.jar -config /home/documentation/docs/plantuml/plantuml_cfg.puml -tsvg ./docs/plantuml/examples/links.puml

docker run -u `id -u $USER`:`id -g $USER` -i  --rm -v $(pwd):/home/documentation -t docdac:ubuntu-1.0 bash ./_sphinx_build.sh

Annex: Build Source code Doxygen

The toolset above (Exhale) builds Doxygen for us i.e. does all of below automatically. This section is shown here if someone wants to build DoxyGen only using toolset.

Get Source Code

git clone git@github.com:Crashedmind/docdac.git #get the documentation and toolset (Dockerfile) source text files
cd ./docdac

Create Doxygen Config File

Run doxygen from container to create config file. Run container exes as me (current user:group) - not root

mkdir documentation #We do this from host so dir owner:group is host user (not root)
docker run --user $(id -u):$(id -g) -i  -v $(pwd):/home/documentation/ -t docdac:ubuntu-1.0 doxygen -g

Files created from container as local host user

ls -la documentation/
total 116
drwxrwxr-x 2 chris_m11 chris_m11   4096 Sep  3 15:02 .
drwxrwxr-x 8 chris_m11 chris_m11   4096 Sep  3 15:02 ..
-rw-r--r-- 1 chris_m11 chris_m11 108516 Sep  3 15:02 Doxyfile

Configure Doxyfile. Only minimal changes shown for illustration purposes.

PROJECT_NAME           = "MyProject"
...
INPUT = ../include

Build Source Code Doxygen Output

docker run -u `id -u $USER`:`id -g $USER` -i  -v $(pwd):/home/documentation/ -t docdac:ubuntu-1.0 doxygen

#view pages with browser e.g.
google-chrome documentation/html/index.html
google-chrome documentation/html/structvfc__keystore__key__info__t.html