I got inspired to write Digital Superpowers after realizing how straightforward it is to publish books these days. Anyone who can put together a manuscript can head on over to a direct publishing service and have it up and for sale. E-books are particularly straightforward to self-publish, but print-on-demand paperbacks are almost as easy. If you have dreams of getting a book out the door, now is a great time to do so.
Writing and promoting any book is the bulk of the work, but this post is about the toolset I chose to use in preparing my manuscript.
Many authors will find that writing manuscripts in word processors like LibreOffice or Microsoft Word will be a better choice for them. This toolset is fairly obscure and labor-intensive. This post is in general more advanced than most of the content in Digital Superpowers itself.
I wrote Digital Superpowers as a series of text files in reStructuredText. I used Sphinx to process these into the ePub format, then used Calibre for final touches, and finally KindleGen to convert the manuscript into the Amazon MOBI format for upload to the Kindle store.
Why reStructuredText?
I chose to use reStructuredText (RST) for a few reasons:
- It’s text-based, so I can do change management with
git - It has many useful capabilities designed to assist with technical documentation, like math environments, automatic table of contents, technical figures, glossary, footnotes, etc.
- The same manuscript can be used to automatically generate PDF, HTML, or ePub final products
RST is a lightweight markup language. That is, it’s intended to be relatively easy for a
human to read and write while still being expressive enough to tell a computerized
processor exactly how to structure the document. Sphinx is the program that interprets RST
into various other format suitable for publishing, including HTML, PDF, ePub, and
even plain text. Together, Sphinx and RST are a formidable publishing system.
Revision control with git
I wanted to keep a running history of writing the book so I could easily try big changes without worrying about being able to go back. I also knew I’d want to keep track of the various revisions I publish as I found typos and made corrections later. The git version control system is perfect for this, and it works best with text files like RST.
Index
I was particularly impressed with the RST/Sphinx index capability. You can just list various key words or phrases before each paragraph in your text and a rich index will be created. It has all the features you might expect, like Tor, see Onion Routing and nesting.
Adding index terms to a paragraph in Sphinx looks like:
.. index::
single: Programming languages
single: Compiling
single: Source code
Programming languages
=====================
Hundreds of *programming languages* exist. Each of them has the
goal of translating (or *compiling*) human input (*source code*)
into electrical operations the computer can perform. And the resulting index ends up looking like:
For more details, see the Sphinx docs on index directives.
Glossary
RST features a basic glossary system that will hyperlink your usages of terms to their definition. It can even automatically alphabetize the entries. See the Sphinx docs for details.
Footnotes
Every book needs footnotes, and RST handles them quite nicely. They show up as clickable pop-ups in the e-book version and as normal footnotes in the PDF.
Then in the ``Y`` column enter our line as ``=3.5*A1 + 2.3 +
(RAND()*8-4)`` [#noise]_. Double-click its little black box to
fill down. Save the file as ``data.xlsx``.
[...]
.. rubric:: Footnotes
.. [#noise] To get our noise from -4 to 4 we do a
random number between 0 and 8 and subtract 4
from it.⋅The PDF result looks like:
Special environments
RST was designed for technical documentation, and some features that have been built as extensions by the technical documentation community are very useful for writing books like mine.
For example, TikZ is a pretty useful way to make technical figures, and RST allows me to embed its source right in my documents using the Tikz extension
.. only:: not text
.. tikz:: The dartboard
\filldraw[fill=blue!40!white, draw=black] (-2,-2) rectangle (2,2);
\filldraw[fill=red!40!white, draw=black] (0,0) circle (2cm);
\draw[thick,->] (0,0) -- (4.5,0);
\draw[thick,->] (0,0) -- (0,4.5);
\draw[step=1cm,gray,very thin] (-3,-3) grid (3,3);
\foreach \x in {0,1,2}
\draw (\x*2 cm,1pt) -- (\x*2 cm,-1pt) node[anchor=north] {$\x$};
\foreach \y in {0,1,2}
\draw (1pt,\y*2 cm) -- (-1pt,\y*2 cm) node[anchor=east] {$\y$};
Which leads to:
Similarly, you can create Graphviz flow charts in text:
.. graphviz::
:caption: Browsing the web through a VPN hides what you're
up to from almost everyone. After the VPN service, metadata is
available but it's anonymized to everyone except the VPN operator
themselves. So you only have to trust the VPN operator in this
scenario.
:align: center
digraph fig {
#graph[margin=0.2, nodesep=0.3, ranksep=0.4];
node [shape=record];
laptop [label="Your laptop"];
guests [label="Other guests"];
hotel [label="Hotel Wi-Fi"];
local_operator [label="Local Wi-Fi contractor"];
laptop -> guests -> hotel -> local_operator [dir="both",color="green"];
local_government [label="Local government"];
routers [label="Nefarious routers"];
VPN [label="VPN operator"];
website [label="Website"];
routers2 [label="Oblivious routers"];
local_government->routers->VPN [dir="both", color="green"];
VPN -> routers2 -> website [dir="both", color="red",style="dashed"];
{rank=same;laptop, local_government}
{rank=same;guests, routers}
{rank=same;hotel, VPN}
local_operator -> local_government [dir="both",color="green"];
}This produces the following flow-chart:
Spelling and Grammar
One big disadvantage of not using a normal office suite for writing a book is the relative shortage of spelling and especially grammar checkers for pure text. I did find a few options that worked out though. Spelling is actually fairly easy in most text editors. In vim, it worked fine, except I couldn’t figure out how to keep RST syntax highlighting and still get indications of misspelled words in blocks that were already highlighted (like in admonitions). I ended up turning syntax highlighting off to check everything.
Grammar is a bit tougher. I tried a few things out, including:
- LanguageTool vim plugin — this found a lot of errors and integrated nicely with Vim.
- Diction and style commands — these actually were better than I expected at first. They had pretty meaningful suggestions and improved the writing. I used the Sphinx pure-text builder to render the book to text files before running these so they wouldn’t get messed up by syntax things.
- Grammarly — I tried it out (again using pure-text). It was pretty useful as well.
Customizing the theme
The ePub theme in Sphinx needed a bit of customization to suit my needs. I ended up
copying the basic theme from the /sphinx/themes/basic folder (in site-packages),
activate it in the config, and made a few tweaks to it. I also used some of the hints and
suggestions from Friends of ePub in my css
files. The Calibre Tools → Check book feature was quite useful to find any mistakes I made
in customizing the theme.
It took me a while to figure out that the theme was putting extraneous <p> tags inside
my table tags causing significant extra vertical padding in my tables. By taking those out
in the custom theme, I was able to get much tighter, good-looking tables.
Frontmatter and copyright pages
I struggled for a while to figure out how to get a cover page, a title page, and a copyright page added to the beginning of my book in a way that Kindle would handle it appropriately. I’m not entirely happy with the solutions I settled on because they’re a bit too manual , but that just means there’s room to improve.
For the PDF, I had to override the \sphinxmaketitle and \sphinxbackoftitlepage
commands in the LaTeX preamble in the conf.py of my Sphinx project.
Here’s my copyright page in the PDF:
\newcommand{\sphinxbackoftitlepage}{%
\vspace*{\fill}
\begingroup
\centering
© 2019 Nicholas Touran, All Rights Reserved
\\~\\
\url{https://digitalsuperpowers.com}
\\~\\
Icons on cover are from FontAwesome, used under a CC BY 4.0 License available at
\url{https://fontawesome.com/license/free}. Modifications to the laptop icon were
made.
\\~\\
Revision: $release
\\~\\
Published on: $date
\endgroup
\vspace*{\fill}
\clearpage
}
I used the sphinxmanual.cls file included with Sphnix as a starting point for my title page and tweaked it to my liking.
Then, I just added 'maketitle': COPYRIGHT + TITLE_PAGE, to the latex_elements section
in the config (where COPYRIGHT is the variable containing my custom LaTeX code) to apply it.
For the ePub version, I actually just hand-made a titlepage.xhtml and copyright.xhtml
page and got them included before the table of contents with the following configuration:
epub_pre_files = [('_static/titlepage.xhtml','Title'),
('_static/copyright.xhtml','Copyright'),
]
epub_guide = (
('toc', 'index.xhtml', u'Table of contents'),
('cover', 'epub-cover.xhtml', u'Cover'),
('title-page', '_static/titlepage.xhtml', u'Title page'),
('copyright-page','_static/copyright.xhtml',u'Copyright'),
)For some other unknown reason, the PDFs generated by the latexpdf Sphinx builder was
corrupt every other time I ran it. I found (by accident) that slapping the single-page
cover on after generating it with a pdftk cat command (as discussed in the book) always
cleaned up the PDF and made it open just fine.
Editing text with vim
I’ve finally gotten good enough at Vim to be productive in editing text. It’s a threshold
situation, where once you put in the effort to learn enough about vim, other text editors
quickly feel lacking. I think the book, Practical Vim by Drew
Neil, really brought me
over the edge. It’s certainly possible to use any other text editor to write your RST, but
using vim just made a few things extra easy. For example, in pure-text, you don’t want to
have really long lines, so auto-wrapping is useful. One neat thing vim can do is reflow
paragraphs of text. So when you edit a nicely formatted text block, you can just reflow it
really easily with a visual selection followed by the gw normal command, like this:
There are thousands of useful features like this in vim. Another nice one is
digraphs, where I can just type
Ctrl-k -> and it inserts a →. Or Ctrl-k M- for an em-dash, or Ctrl-k Co for a ©. I
really can edit text at the speed of thought!
Empty footnotes sections on pages without footnotes
For some reason, the ePub builder in Sphinx put empty Footnotes sections at the end of the table of contents and any other chapter that didn’t have any footnotes. I tried to figure this out but ended up opening the file with Calibre and removed the manually before converting to MOBI format.
Double-copies of images
The file size was larger than I expected, and through inspection with Calibre, I found
that there were two copies of all of my images being included in the ePub file. Since book
distributors charge extra for larger books, I was motivated to reduce this. Again, I ended
up using Calibre to manually remove the files in the _static folder that were duplicated
in the _images folder. I think this is a Sphinx ePub builder bug but I haven’t quite isolated
it.
Final step
Once the ePub version was all set and cleaned up by Calibre, I just ran Kindlegen to get the final file for uploading to Amazon:
../kindlegen/kindlegen -c2 "Digital Superpowers - Nick Touran.epub"
And voilà, there it was.
Path forward: the paperback!
I have not yet pulled the trigger on the paperback print-on-demand version of the book. I still need to polish up the formatting of the PDF and make sure I get the dimensions right and whatnot.
Overall, this process wasn’t easy. I need to follow up on some of the Sphinx ePub quirks I ran into and maybe open up a ticket or two. But in the end I was quite happy with the workflow and impressed with the capabilities of the toolset to make a nicely-formatted book.