Documentation Production using Open Source Tools Elena Zannoni, - PowerPoint PPT Presentation

<Insert Picture Here> Documentation Production using Open Source Tools Elena Zannoni, elena.zannoni@oracle.com Stefan Hinz, stefan.hinz@oracle.com Paul Du Bois, paul.dubois@oracle.com CONFIDENTIAL – ORACLE HIGHLY RESTRICTED 1

Introduction • Documentation processes for Oracle Linux, MySQL, Oracle VM, and other virtualization products • Other Oracle documentation teams use different technology • All our documentation is public on the Oracle Technology Network (OTN) • MySQL documentation is also on dev.mysql.com • Formats published: XHTML, WebHelp, PDF, ePUB, man pages • Toolchain in use for many years (with many improvements along the way) LinuxCon Japan 2015

Infrastructure Overview • SVN (Subversion) for all documentation and all tools • Store product documentation sources, images • Plus, common repository for all our tools, called mysqldoc-toolset • Build scripts also stored, mysqldoc-tools • Planning to move to GIT • DocBook XML as the base documentation format • Currently DocBook XML 4.5 • DocBook XSL style sheets for processing • Currently DocBook XSL 1.78.1 • Customized XSL style sheets for output customization • To meet Oracle publication style requirements, accessibility, legal notices, etc. Using XSL, Perl, and Python • Set of Makefiles and make targets for building • Fully automated build & publishing system written in Ruby LinuxCon Japan 2015 3

Writer's Workflow: Overview • Checkout documentation repositories • One repository per product • Some common repositories needed for stylesheets and for local building of the output formats • Edit XML files • Validate resulting XML files • Locally build at least one output format • Check modified XML files into repository • Monitor the automated build to make sure all is well • Retrieve output and publish externally LinuxCon Japan 2015 4

Writer's Workflow: Authoring • DocBook XML is a markup language • We use DocBook XML DTD 4.5 • For editing: use emacs, VI or Oxygen XML Editor • Most of our writers prefer Oxygen because of its builtin features such as: • Completion of XML tags • Integrated SVN client • Integrated XML processing (previews for different formats) • Multi platform (runs on MacOS, Linux, Windows..) • Inexpensive LinuxCon Japan 2015 5

XML File in Oxygen: English LinuxCon Japan 2015 6

XML File in Oxygen: Japanese LinuxCon Japan 2015 7

Writer's Workflow: Validation & Local Testing • This sequence of operations is done locally on author's machine • Author validates the XML file – making sure the XML file is valid DocBook XML 4.5 • This is done using a program called xmllint • Invoked via “make <filename>.valid” • Author builds one or more output formats locally (for instance) • “make <filename>.html” • “make <filename>.pdf” LinuxCon Japan 2015 8

Overview of Make Targets • Several Makefile targets to facilitate the workflow • Make <filename>.valid – verify xml is valid • Make <filename>.prepped – preprocessing step (include files, xref..) • Make <filename>.format – format file to standard xml (indentation,...) • Make <filename>.html • Make <filename>.xhtml • Make <filename>.webhelp • Make <filename>.pdf • Make depend: generate list of doc dependencies • Make collected: build xhtml file with ID references resolved to URLs • Make clean • […] plus many flavors of each format LinuxCon Japan 2015 9

Resolving References We call this step ID mapping • References to sections in same document or in other documents must be resolved to correct URLs before building the output targets • References look like: • <xref linkend="option-file-options"/> (section ID) • <xref linkend="refman:replication"/> (file name and section ID) • Build the “prepped” file: intermediate XML file with the references resolved to correct URLs. • Build the output formats from the prepped file. • Topic for another talk....it involves building more intermediate files and dependencies LinuxCon Japan 2015 10

Building Output Formats • Process book.xml (which is a DocBook XML 4.5 file) to an output format • This uses a program called xsltproc • That program uses the DocBook XSL Style Sheets to find out how to do the transformation to another format • We use the DocBook XSL Style Sheets 1.78.1 LinuxCon Japan 2015 11

Basic Output Formatting Workflow XHTML XML man XSLTPROC ePub your-html.xsl .fo PDF FOP your-man.xsl XHTML/ Lucene webhelp webhelp your-epub.xsl jQuery your-fo.xsl your-webhelp.xsl LinuxCon Japan 2015

Writer's Workflow: Final Steps • If all looks good, format the XML file according to standards: • “make <filename>.format” • Check the modified XML file into the repository • This action will trigger an official build in the central build system • Check for build exit status • Publish the resulting books if build successful LinuxCon Japan 2015 13

Build System • Once the file has been checked in to the central SVN repository various output formats are built from it – this is fully automated • Output formats include HTML, XHTML, WebHelp, EPUB, and PDF • The build system is written in Ruby. More precisely dRuby with Rinda. • For some things that are hard to do in XSL, it uses Perl • For the transformation of huge XML documents to HTML, it uses Python • To build the PDF output format, it uses FOP – we use FOP 1.1 LinuxCon Japan 2015 14

Build System Architecture Highlights Async User/Web UI Commands Build Control Built Targets (Status & Output) Commit Notification Build Master Repo to Update Targets to Build Commit Monitor Build Slave Checkout Revision # changed Writer Commit Central Local Copy Doc repo of Doc repo LinuxCon Japan 2015 15

Build System in a Nutshell • Communication between master, slave, commit monitor happens via messages • Messages are tuples • Commit monitor notices source repo changes and informs build master • Build master dispatches a request for repo update to a slave • Build master dispatches a build to a slave • Slave does the build and returns built files and status to master LinuxCon Japan 2015 16

Implementation: Rinda Tuple Space • Communication server operates on the TupleSpace (in Rinda) • Every client interacts with the TupleSpace via the server. • Clients in the build system: Build master, Build slaves, Build Control, Commit Monitor • Communication done by sending and receiving tuples to/from tuplespace • Tuples have destination/sender plus message body LinuxCon Japan 2015 17

Configuration Files • Build slaves conf file: list of repositories to checkout/update • Targets conf file: list of build targets • Commit monitor conf file: list of repositories to monitor for commits LinuxCon Japan 2015 18

Commit-monitor and Slave Conf Fragment from commit-monitor configuration file: [...] "repositories" => { "linux" => { "check-type" => "svn info", Command to check revision numbers "repo-url" => "#{repo_url_prefix}/linux/trunk" }, [...] } Fragment from build slaves configuration file: "repositories" => { "linux" => { Path to the repo "basedir" => "#{repo_parent}/linux", "update-cmd" => update_cmd, Command to get local copy of repo "documents" => { "uek3-qu4-relnotes" => "uek3/qu4", List of all possible documents to be built "uek3-qu5-relnotes" => "uek3/qu5", […] }}} LinuxCon Japan 2015 19

Target Configuration Example "linux" => { Multiple Languages Possible "lang-code" => "en", "documents" => { "uek3-qu5-relnotes" => { Output directory in the output tree "output-directory" => "linux/uek3-qu5-relnotes", <tree-root>/<output-directory>/<lang-code>/ "target-base" => "relnotes", Standard prerequisites used "document-barriers" => std_document_barriers, "goal-targets" => [ List of targets [ "%T.html-embed", 900 ], “make <target-name>” will be used to build [ "%T.webhelp", 900 ], Each target has a priority Lower number == higher priority [ "%T.xhtml-dir", 900 ], [ "%T.xhtml-otn", 900 ], [ "%T.xhtml-review", 900 ], [ "%T.pdf", 910 ], [ "%T-review.pdf", 910 ], [ "%T.epub", 920 ], [ "%T.txt", 920 ], ], } } } LinuxCon Japan 2015 20

Build Targets For each document there is a series of targets to be built • Goal Targets: targets that correspond to desired output formats (HTML, XHTML, PDF, etc). Output is saved and returned. • Document Barriers: targets that must be built successfully before goal targets can be built. For instance making sure that the source document (XML file) is valid. No output is produced. • docs_valid LinuxCon Japan 2015 21