PAX manual

Table of Contents

[in package MGL-PAX]

1 mgl-pax ASDF System Details

2 Background

As a user, I frequently run into documentation that's incomplete and out of date, so I tend to stay in the editor and explore the code by jumping around with SLIME's M-.. As a library author, I spend a great deal of time polishing code, but precious little writing documentation.

In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.

That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programming weenie turned inside out. The original prototype which did almost everything I wanted was this:

(defmacro defsection (name docstring)
  `(defun ,name () ,docstring))

Armed with DEFSECTION, I soon found myself organizing code following the flow of user level documentation and relegated comments to implementational details entirely. However, some portions of DEFSECTION docstrings were just listings of all the functions, macros and variables related to the narrative, and this list was effectively repeated in the DEFPACKAGE form complete with little comments that were like section names. A clear violation of OAOO, one of them had to go, so DEFSECTION got a list of symbols to export.

That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class are named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type which was later generalized into the concept of locatives:

(defsection @pax-introduction
  "A single line for one man ..."
  (foo class)
  (bar function))

After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol and everything was good for a while.

Then I realized that sections could refer to other sections if there were a SECTION locative. Going down that path, I soon began to feel the urge to generate pretty documentation as all the necessary information was manifest in the DEFSECTION forms. The design constraint imposed on documentation generation was that following the typical style of upcasing symbols in docstrings there should be no need to explicitly mark up links: if M-. works, then the documentation generator shall also be able find out what's being referred to.

I settled on Markdown as a reasonably non-intrusive format, and a few thousand lines later PAX was born.

3 Tutorial

PAX provides an extremely poor man's Explorable Programming environment. Narrative primarily lives in so called sections that mix markdown docstrings with references to functions, variables, etc, all of which should probably have their own docstrings.

The primary focus is on making code easily explorable by using SLIME's M-. (slime-edit-definition). See how to enable some fanciness in Emacs Integration. Generating documentation from sections and all the referenced items in Markdown or HTML format is also implemented.

With the simplistic tools provided, one may accomplish similar effects as with Literate Programming, but documentation is generated from code, not vice versa and there is no support for chunking yet. Code is first, code must look pretty, documentation is code.

In typical use, PAX packages have no :EXPORT's defined. Instead the DEFINE-PACKAGE form gets a docstring which may mention section names (defined with DEFSECTION). When the code is loaded into the lisp, pressing M-. in SLIME on the name of the section will take you there. Sections can also refer to other sections, packages, functions, etc and you can keep exploring.

Here is an example of how it all works together:

(mgl-pax:define-package :foo-random
  (:documentation "This package provides various utilities for
  random. See FOO-RANDOM:@FOO-RANDOM-MANUAL.")
  (:use #:common-lisp #:mgl-pax))

(in-package :foo-random)

(defsection @foo-random-manual (:title "Foo Random manual")
  "Here you describe what's common to all the referenced (and
  exported) functions that follow. They work with *FOO-STATE*,
  and have a :RANDOM-STATE keyword arg. Also explain when to 
  choose which."
  (foo-random-state class)
  (state (reader foo-random-state))
  "Hey we can also print states!"
  (print-object (method () (foo-random-state t)))
  (*foo-state* variable)
  (gaussian-random function)
  (uniform-random function)
  ;; this is a subsection
  (@foo-random-examples section))

(defclass foo-random-state ()
  ((state :reader state)))

(defmethod print-object ((object foo-random-state) stream)
  (print-unreadable-object (object stream :type t)))

(defvar *foo-state* (make-instance 'foo-random-state)
  "Much like *RANDOM-STATE* but uses the FOO algorithm.")

(defun uniform-random (limit &key (random-state *foo-state*))
  "Return a random number from the [0, LIMIT) uniform distribution."
  nil)

(defun gaussian-random (stddev &key (random-state *foo-state*))
  "Return a random number from a zero mean normal distribution with
  STDDEV."
  nil)

(defsection @foo-random-examples (:title "Examples"))

Generating documentation in a very stripped down markdown format is easy:

(describe @foo-random-manual)

For this example, the generated markdown would look like this:

# Foo Random manual

Here you describe what's common to all the referenced (and
exported) functions that follow. They work with *FOO-STATE*,
and have :RANDOM-STATE keyword arg. Also explain when to choose
which.

- [class] FOO-RANDOM-STATE

- [reader] STATE FOO-RANDOM-STATE

Hey we can also print states!

- [method] PRINT-OBJECT (OBJECT FOO-RANDOM-STATE) STREAM

- [variable] *FOO-STATE* #<FOO-RANDOM-STATE >

    Much like *RANDOM-STATE* but uses the FOO algorithm.

- [function] GAUSSIAN-RANDOM STDDEV &KEY RANDOM-STATE

    Return a random number from a zero mean normal distribution with
    STDDEV.

- [function] UNIFORM-RANDOM LIMIT &KEY RANDOM-STATE

    Return a random number from the [0, LIMIT) uniform distribution.

## Examples

More fancy markdown or html output with automatic markup and linking of uppercase symbol names found in docstrings, section numbering, table of contents, etc is possible by calling the DOCUMENT function.

One can even generate documentation for different, but related libraries at the same time with the output going to different files, but with cross-page links being automatically added for symbols mentioned in docstrings. For a complete example of how to generate HTML with multiple pages, see src/doc.lisp.

Note how (VARIABLE *FOO-STATE*) in the DEFSECTION form both exports *FOO-STATE* and includes its documentation in @FOO-RANDOM-MANUAL. The symbols VARIABLE and FUNCTION are just two instances of 'locatives' which are used in DEFSECTION to refer to definitions tied to symbols. See Locative Types.

4 Emacs Integration

Integration into SLIME's M-. (slime-edit-definition) allows one to visit the source location of the thing that's identified by a symbol and the locative before or after the symbol in a buffer. With this extension, if a locative is the previous or the next expression around the symbol of interest, then M-. will go straight to the definition which corresponds to the locative. If that fails, M-. will try to find the definitions in the normal way which may involve popping up an xref buffer and letting the user interactively select one of possible definitions.

Note that the this feature is implemented in terms of SWANK-BACKEND:FIND-SOURCE-LOCATION and SWANK-BACKEND:FIND-DEFINITIONS whose support varies across the Lisp implementations.

In the following examples, pressing M-. when the cursor is on one of the characters of FOO or just after FOO, will visit the definition of function FOO:

function foo
foo function
(function foo)
(foo function)

In particular, references in a DEFSECTION form are in (SYMBOL LOCATIVE) format so M-. will work just fine there.

Just like vanilla M-., this works in comments and docstrings. In this example pressing M-. on FOO will visit FOO's default method:

;;;; See FOO `(method () (t t t))` for how this all works.
;;;; But if the locative has semicolons inside: FOO `(method
;;;; () (t t t))`, then it won't, so be wary of line breaks
;;;; in comments.

With a prefix argument (C-u M-.), one can enter a symbol plus a locative separated by whitespace to preselect one of the possibilities.

The M-. extensions can be enabled by done adding this to your Emacs initialization file:

(defun slime-edit-locative-definition (name &optional where)
  (or (slime-locate-definition name (slime-locative-before))
      (slime-locate-definition name (slime-locative-after))
      (slime-locate-definition name (slime-locative-after-in-brackets))
      ;; support "foo function" and "function foo" syntax in
      ;; interactive use
      (let ((pos (cl-position ?s name)))
        (when pos
          (or (slime-locate-definition (cl-subseq name 0 pos)
                                       (cl-subseq name (1+ pos)))
              (slime-locate-definition (cl-subseq name (1+ pos))
                                       (cl-subseq name 0 pos)))))))

(defun slime-locative-before ()
  (ignore-errors (save-excursion
                   (slime-beginning-of-symbol)
                   (slime-last-expression))))

(defun slime-locative-after ()
  (ignore-errors (save-excursion
                   (slime-end-of-symbol)
                   (slime-forward-sexp)
                   (slime-last-expression))))

(defun slime-locative-after-in-brackets ()
  (ignore-errors (save-excursion
                   (slime-end-of-symbol)
                   (skip-chars-forward "`" (+ (point) 1))
                   (when (and (= 1 (skip-chars-forward "\]"
                                                       (+ (point) 1)))
                              (= 1 (skip-chars-forward "\["
                                                       (+ (point) 1))))
                     (buffer-substring-no-properties
                      (point)
                      (progn (search-forward "]" nil (+ (point) 1000))
                             (1- (point))))))))

(defun slime-locate-definition (name locative)
  (when locative
    (let ((location
           (slime-eval
            ;; Silently fail if mgl-pax is not loaded.
            `(cl:when (cl:find-package :mgl-pax)
                      (cl:funcall
                       (cl:find-symbol 
                        (cl:symbol-name :locate-definition-for-emacs)
                        :mgl-pax)
                       ,name ,locative)))))
      (when (and (consp location)
                 (not (eq (car location) :error)))
        (slime-edit-definition-cont
         (list (make-slime-xref :dspec `(,name)
                                :location location))
         "dummy name"
         where)))))

(add-hook 'slime-edit-definition-hooks 'slime-edit-locative-definition)

5 Basics

Now let's examine the most important pieces in detail.

6 Markdown Support

The Markdown in docstrings is processed with the 3BMD library.

6.1 Indentation

Docstrings can be indented in any of the usual styles. PAX normalizes indentation by converting:

(defun foo ()
  "This is
  indented
  differently")

to

(defun foo ()
  "This is
indented
differently")

See DOCUMENT-OBJECT for the details.

6.2 Syntax highlighting

For syntax highlighting, github's fenced code blocks markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:

```elisp
(defun foo ())
```

to get syntactically marked up HTML output. Copy doc/style.css from PAX and you are set. The language tag, elisp in this example, is optional and defaults to common-lisp.

See the documentation of 3BMD and colorize for the details.

7 Documentation Printer Variables

Docstrings are assumed to be in markdown format and they are pretty much copied verbatim to the documentation subject to a few knobs described below.

8 Locative Types

These are the locatives type supported out of the box. As all locative types, they are symbols and their names should make it obvious what kind of things they refer to. Unless otherwise noted, locatives take no arguments.

9 Extension API

9.1 Locatives and References

While Common Lisp has rather good introspective abilities, not everything is first class. For example, there is no object representing the variable defined with (DEFVAR FOO). (MAKE-REFERENCE 'FOO 'VARIABLE) constructs a REFERENCE that captures the path to take from an object (the symbol FOO) to an entity of interest (for example, the documentation of the variable). The path is called the locative. A locative can be applied to an object like this:

(locate 'foo 'variable)

which will return the same reference as (MAKE-REFERENCE 'FOO 'VARIABLE). Operations need to know how to deal with references which we will see in LOCATE-AND-COLLECT-REACHABLE-OBJECTS, LOCATE-AND-DOCUMENT and LOCATE-AND-FIND-SOURCE.

Naturally, (LOCATE 'FOO 'FUNCTION) will simply return #'FOO, no need to muck with references when there is a perfectly good object.

9.2 Adding New Object Types

One may wish to make the DOCUMENT function and M-. navigation work with new object types. Extending DOCUMENT can be done by defining a DOCUMENT-OBJECT method. To allow these objects to be referenced from DEFSECTION a LOCATE-OBJECT method is to be defined. Finally, for M-. FIND-SOURCE can be specialized. Finally, EXPORTABLE-LOCATIVE-TYPE-P may be overridden if exporting does not makes sense. Here is a stripped down example of how all this is done for ASDF:SYSTEM:

(define-locative-type asdf:system ()
  "Refers to an asdf system.")

(defmethod exportable-locative-type-p ((locative-type (eql 'asdf:system)))
  nil)

(defmethod locate-object (symbol (locative-type (eql 'asdf:system))
                          locative-args)
  (assert (endp locative-args))
  (or (asdf:find-system symbol nil)
      (locate-error)))

(defmethod canonical-reference ((system asdf:system))
  (make-reference (asdf/find-system:primary-system-name system)
                  'asdf:system))

(defmethod document-object ((system asdf:system) stream)
  (with-heading (stream system
                        (format nil "~A ASDF System Details"
                                (asdf/find-system:primary-system-name
                                 system)))
    (format stream "Some content~%")))

(defmethod find-source ((system asdf:system))
  `(:location
    (:file ,(namestring (asdf/system:system-source-file system)))
    (:position 1)
    (:snippet "")))

9.3 Reference Based Extensions

Let's see how to extend DOCUMENT and M-. navigation if there is no first class object to represent the thing of interest. Recall that LOCATE returns a REFERENCE object in this case. DOCUMENT-OBJECT and FIND-SOURCE defer to LOCATE-AND-DOCUMENT and LOCATE-AND-FIND-SOURCE which have LOCATIVE-TYPE in their argument list for EQL specializing pleasure. Here is a stripped down example of how the VARIABLE locative is defined:

(define-locative-type variable (&optional initform)
  "Refers to a global special variable. INITFORM, or if not specified,
  the global value of the variable is included in the documentation.")

(defmethod locate-object (symbol (locative-type (eql 'variable))
                          locative-args)
  (assert (<= (length locative-args) 1))
  (make-reference symbol (cons locative-type locative-args)))

(defmethod locate-and-document (symbol (locative-type (eql 'variable))
                                locative-args stream)
  (destructuring-bind (&optional (initform nil initformp)) locative-args
    (format stream "- [~A] " (string-downcase locative-type))
    (print-name (prin1-to-string symbol) stream)
    (write-char #Space stream)
    (multiple-value-bind (value unboundp) (symbol-global-value symbol)
      (print-arglist (prin1-to-string (cond (initformp initform)
                                            (unboundp "-unbound-")
                                            (t value)))
                     stream))
    (terpri stream)
    (maybe-print-docstring symbol locative-type stream)))

(defmethod locate-and-find-source (symbol (locative-type (eql 'variable))
                                   locative-args)
  (declare (ignore locative-args))
  (find-one-location (swank-backend:find-definitions symbol)
                     '("variable" "defvar" "defparameter"
                       "special-declaration")))

We have covered the basic building blocks of reference based extensions. Now let's see how the obscure DEFINE-SYMBOL-LOCATIVE-TYPE and DEFINE-DEFINER-FOR-SYMBOL-LOCATIVE-TYPE macros work together to simplify the common task of associating definition and documentation with symbols in a certain context.

9.4 Sections

Section objects rarely need to be dissected since DEFSECTION and DOCUMENT cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.


[generated by MGL-PAX]