Literate programming in Go | InfoWorld

All through a latest enterprise hackathon, my team’s constitution was to make improvements to the documentation for the Steampipe plugin SDK. Like other components of the Steampipe system, the plugin SDK is composed in Go and released to pkg.go.dev. The variation that existed when we started off is below. As is typical, the documentation was an autogenerated catalog of capabilities and forms. To explain how to use all those capabilities and styles, we delivered guidance on the steampipe.io internet site.

Our hackathon obstacle was to weave such guidance into the generated documentation. If you might be creating a Steampipe plugin, your list of tasks looks like this:

1. Determine the plugin
2. Generate the plugin entry stage
3. Outline your first table
4. …

These duties involve that you use functions and kinds, but though remarks attached to those functions and sorts can improve the generated documentation, they are far too granular for the large-level exposition we aimed for. Hunting for inspiration, Steampipe guide developer Kai Daguerre found that our top rated-stage site lacked the overview section he saw in the documentation for pgx, a Go driver for Postgres.

That overview arrives from https://github.com/jackc/pgx/blob/master/doc.go, which is a person extended remark (that utilizes Go remark syntax) adopted by a deal declaration.

So we added a doc.go at the prime degree of our repo to make this overview.

IDG

Creating a wiki in Go feedback

We utilised the name doc.go since that appears to be to be standard, but it could have been termed foo.go. What is salient is that it is a legitimate Go file that belongs to a deal. The package deal contains no code, only documentation. We wrote headers to produce the sections of the overview, and in each individual section we described a plugin writer’s activity employing narrative, inline code illustrations, inner one-way links to capabilities and varieties, and exterior back links to examples elsewhere.

The means to url inside the code’s namespace was a revelation. To explain the endeavor known as Insert hydrate capabilities, for case in point, we wrote this:

# Incorporate hydrate capabilities

A column may possibly be populated by a Checklist or Get connect with. If a column calls for facts not provide by Checklist 
or Get, it could outline a [plugin.HydrateFunc] that makes an added API contact for every row.
Insert a hydrate function for a column by setting [plugin.Column.Hydrate].

The segment outlined by the Add hydrate functions header is a backlink goal: Insert hydrate functions. And the bracketed products render as hyperlinks to a kind, plugin.HydrateFunc, and to a house, plugin.Column.Hydrate. This was beginning to really feel like wiki writing!

What we still lacked, though, was the capacity to generate new wiki webpages where by we could make clear increased-level ideas. The overview was a person area to do that, but we needed to reserve that for narration of the plugin writer’s journey. In which could we focus on a strategy like dynamic tables, an highly developed characteristic that allows plugins like CSV which have no set schema and must determine columns on the fly?

Kai understood that not only could we produce new documentation-only deals for these kinds of matters, we could also import them so that their names were being offered for the same form of shorthand linking we could do with, e.g., [plugin.HydrateFunc]. In the leading-stage doc.go he did this:

deal steampipe_plugin_sdk

import (
"github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables"
)

var forceImportDynamicPlugin dynamic_tables.ForceImport

And in /docs/dynamic_tables/doc.go he did this:

offer dynamic_tables

sort ForceImport string

Now we could write a sentence in a remark like this:

/*
Use [dynamic_tables] when you are not able to know a table's schema in advance.
*/
package deal steampipe-plugin-sdk

And the bracketed time period autolinks just like any other name in the code’s namespace.

If that would seem like a lot more difficulties than it’s worth, you can skip the import gymnastics and just use an external connection:

/*
Use [dynamic_tables] when you are not able to know a table's schema in advance.


[dynamic_tables]: https://pkg.go.dev/github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables
*/
offer steampipe-plugin-sdk

Either way, the authoring knowledge now feels much more fully wiki-like. You can produce new internet pages as necessary, and weave them into the hypertextual documentation that life inside the code foundation.

It was, admittedly, a bit of a battle to use the Go method in the strategies explained listed here. The remark syntax is Markdown-like but frustratingly not Markdown it relies upon on quite a few implicit formatting conventions. If you go this route you’ll have to have a neighborhood previewing instrument, and it’s not tremendous-noticeable that the just one you want is not godoc but somewhat pkgsite. Had we uncovered the command go set up golang.org/x/pkgsite/cmd/pkgsite@most recent we would have saved ourselves a good deal of grief.

How is this literate programming?

It isn’t! In his eponymous paper Knuth wrote:

The practitioner of literate programming can be regarded as an essayist, whose main concern is with exposition and excellence of type. These types of an writer, with thesaurus in hand, chooses the names of variables meticulously and clarifies what each individual variable indicates. He or she strives for a plan that is comprehensible for the reason that its principles have been launched in an purchase that is greatest for human being familiar with, employing a mixture of official and informal strategies that enhance every single other.

To assist this practice he invented a program termed world-wide-web whose parts, tangle and weave, enabled the creator of a plan to inform its tale in a language that mixed code (at first, Pascal) and documentation (initially, TeX) in a narrative-initially way. Our contemporary techniques of mixing code and documentation, and generating docs from embedded feedback, only superficially resemble Knuth’s follow, as Mark Jason Dominus pointed out in his essay POD is not Literate Programming (2000).

And but, now that our hackathon exercising has given me a flavor of what code-as-wiki can be, it does come to feel like a stage toward the storytelling approach that Knuth advocates. Recall that he named his procedure world wide web just before there was the net we now inhabit, in no way thoughts wikis! I can’t assert that we’re now literate programmers in the Knuthian sense, and I’ve usually questioned if only a Knuthian brain could employ that first eyesight. But this way of enhancing the narrative top quality of autogenerated docs does really feel like a phase in the suitable direction.