As a user, I frequently run into documentation that's incomplete
and out of date, so I tend to stay in the editor and explore the
code by jumping around with SLIME's M-.
. As a library
author, I spend a great deal of time polishing code, but precious
little writing documentation.
In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.
That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programming weenie turned inside out. The original prototype which did almost everything I wanted was this:
(defmacro defsection (name docstring)
`(defun ,name () ,docstring))
Armed with DEFSECTION
, I soon found myself organizing code following
the flow of user level documentation and relegated comments to
implementational details entirely. However, some portions of
DEFSECTION
docstrings were just listings of all the functions,
macros and variables related to the narrative, and this list was
effectively repeated in the DEFPACKAGE
form complete with little
comments that were like section names. A clear violation of
OAOO, one of them had to go, so DEFSECTION
got a list of
symbols to export.
That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class are named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type which was later generalized into the concept of locatives:
(defsection @pax-introduction
"A single line for one man ..."
(foo class)
(bar function))
After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol and everything was good for a while.
Then I realized that sections could refer to other sections if there
were a SECTION
locative. Going down that path, I soon began to feel
the urge to generate pretty documentation as all the necessary
information was manifest in the DEFSECTION
forms. The design
constraint imposed on documentation generation was that following
the typical style of upcasing symbols in docstrings there should be
no need to explicitly mark up links: if M-. works, then the
documentation generator shall also be able find out what's being
referred to.
I settled on Markdown as a reasonably non-intrusive format, and a few thousand lines later PAX was born.
PAX provides an extremely poor man's Explorable Programming environment. Narrative primarily lives in so called sections that mix markdown docstrings with references to functions, variables, etc, all of which should probably have their own docstrings.
The primary focus is on making code easily explorable by using
SLIME's M-. (slime-edit-definition
). See how to enable some
fanciness in Emacs Integration. Generating documentation from
sections and all the referenced items in Markdown or HTML format is
also implemented.
With the simplistic tools provided, one may accomplish similar effects as with Literate Programming, but documentation is generated from code, not vice versa and there is no support for chunking yet. Code is first, code must look pretty, documentation is code.
In typical use, PAX packages have no :EXPORT
's defined. Instead the
DEFINE-PACKAGE
form gets a docstring which may mention section
names (defined with DEFSECTION
). When the code is loaded into the
lisp, pressing M-. in SLIME on the name of the section will take you
there. Sections can also refer to other sections, packages,
functions, etc and you can keep exploring.
Here is an example of how it all works together:
(mgl-pax:define-package :foo-random
(:documentation "This package provides various utilities for
random. See FOO-RANDOM:@FOO-RANDOM-MANUAL.")
(:use #:common-lisp #:mgl-pax))
(in-package :foo-random)
(defsection @foo-random-manual (:title "Foo Random manual")
"Here you describe what's common to all the referenced (and
exported) functions that follow. They work with *FOO-STATE*,
and have a :RANDOM-STATE keyword arg. Also explain when to
choose which."
(foo-random-state class)
(state (reader foo-random-state))
"Hey we can also print states!"
(print-object (method () (foo-random-state t)))
(*foo-state* variable)
(gaussian-random function)
(uniform-random function)
;; this is a subsection
(@foo-random-examples section))
(defclass foo-random-state ()
((state :reader state)))
(defmethod print-object ((object foo-random-state) stream)
(print-unreadable-object (object stream :type t)))
(defvar *foo-state* (make-instance 'foo-random-state)
"Much like *RANDOM-STATE* but uses the FOO algorithm.")
(defun uniform-random (limit &key (random-state *foo-state*))
"Return a random number from the [0, LIMIT) uniform distribution."
nil)
(defun gaussian-random (stddev &key (random-state *foo-state*))
"Return a random number from a zero mean normal distribution with
STDDEV."
nil)
(defsection @foo-random-examples (:title "Examples"))
Generating documentation in a very stripped down markdown format is easy:
(describe @foo-random-manual)
For this example, the generated markdown would look like this:
# Foo Random manual
Here you describe what's common to all the referenced (and
exported) functions that follow. They work with *FOO-STATE*,
and have :RANDOM-STATE keyword arg. Also explain when to choose
which.
- [class] FOO-RANDOM-STATE
- [reader] STATE FOO-RANDOM-STATE
Hey we can also print states!
- [method] PRINT-OBJECT (OBJECT FOO-RANDOM-STATE) STREAM
- [variable] *FOO-STATE* #<FOO-RANDOM-STATE >
Much like *RANDOM-STATE* but uses the FOO algorithm.
- [function] GAUSSIAN-RANDOM STDDEV &KEY RANDOM-STATE
Return a random number from a zero mean normal distribution with
STDDEV.
- [function] UNIFORM-RANDOM LIMIT &KEY RANDOM-STATE
Return a random number from the [0, LIMIT) uniform distribution.
## Examples
More fancy markdown or html output with automatic markup and linking
of uppercase symbol names found in docstrings, section numbering,
table of contents, etc is possible by calling the DOCUMENT
function.
One can even generate documentation for different, but related
libraries at the same time with the output going to different files,
but with cross-page links being automatically added for symbols
mentioned in docstrings. For a complete example of how to generate
HTML with multiple pages, see src/doc.lisp
.
Note how (VARIABLE *FOO-STATE*)
in the DEFSECTION
form both
exports *FOO-STATE*
and includes its documentation in
@FOO-RANDOM-MANUAL
. The symbols VARIABLE
and FUNCTION
are just two
instances of 'locatives' which are used in DEFSECTION
to refer to
definitions tied to symbols. See Locative Types.
Integration into SLIME's M-. (slime-edit-definition
) allows one
to visit the source location of the thing that's identified by a
symbol and the locative before or after the symbol in a buffer. With
this extension, if a locative is the previous or the next expression
around the symbol of interest, then M-. will go straight to the
definition which corresponds to the locative. If that fails, M-.
will try to find the definitions in the normal way which may involve
popping up an xref buffer and letting the user interactively select
one of possible definitions.
Note that the this feature is implemented in terms of
SWANK-BACKEND:FIND-SOURCE-LOCATION
and
SWANK-BACKEND:FIND-DEFINITIONS
whose support varies across the Lisp
implementations.
In the following examples, pressing M-. when the cursor is on one of
the characters of FOO
or just after FOO
, will visit the
definition of function FOO
:
function foo
foo function
(function foo)
(foo function)
In particular, references in a DEFSECTION
form are in (SYMBOL
LOCATIVE
) format so M-. will work just fine there.
Just like vanilla M-., this works in comments and docstrings. In
this example pressing M-. on FOO
will visit FOO
's default
method:
;;;; See FOO `(method () (t t t))` for how this all works.
;;;; But if the locative has semicolons inside: FOO `(method
;;;; () (t t t))`, then it won't, so be wary of line breaks
;;;; in comments.
With a prefix argument (C-u M-.
), one can enter a symbol plus a
locative separated by whitespace to preselect one of the
possibilities.
The M-. extensions can be enabled by done adding this to your Emacs initialization file:
(defun slime-edit-locative-definition (name &optional where)
(or (slime-locate-definition name (slime-locative-before))
(slime-locate-definition name (slime-locative-after))
(slime-locate-definition name (slime-locative-after-in-brackets))
;; support "foo function" and "function foo" syntax in
;; interactive use
(let ((pos (cl-position ?s name)))
(when pos
(or (slime-locate-definition (cl-subseq name 0 pos)
(cl-subseq name (1+ pos)))
(slime-locate-definition (cl-subseq name (1+ pos))
(cl-subseq name 0 pos)))))))
(defun slime-locative-before ()
(ignore-errors (save-excursion
(slime-beginning-of-symbol)
(slime-last-expression))))
(defun slime-locative-after ()
(ignore-errors (save-excursion
(slime-end-of-symbol)
(slime-forward-sexp)
(slime-last-expression))))
(defun slime-locative-after-in-brackets ()
(ignore-errors (save-excursion
(slime-end-of-symbol)
(skip-chars-forward "`" (+ (point) 1))
(when (and (= 1 (skip-chars-forward "\]"
(+ (point) 1)))
(= 1 (skip-chars-forward "\["
(+ (point) 1))))
(buffer-substring-no-properties
(point)
(progn (search-forward "]" nil (+ (point) 1000))
(1- (point))))))))
(defun slime-locate-definition (name locative)
(when locative
(let ((location
(slime-eval
;; Silently fail if mgl-pax is not loaded.
`(cl:when (cl:find-package :mgl-pax)
(cl:funcall
(cl:find-symbol
(cl:symbol-name :locate-definition-for-emacs)
:mgl-pax)
,name ,locative)))))
(when (and (consp location)
(not (eq (car location) :error)))
(slime-edit-definition-cont
(list (make-slime-xref :dspec `(,name)
:location location))
"dummy name"
where)))))
(add-hook 'slime-edit-definition-hooks 'slime-edit-locative-definition)
Now let's examine the most important pieces in detail.
[macro] DEFSECTION NAME (&KEY (PACKAGE *PACKAGE*) (EXPORT T) TITLE (DISCARD-DOCUMENTATION-P *DISCARD-DOCUMENTATION-P*)) &BODY ENTRIES
Define a documentation section and maybe export referenced symbols.
A bit behind the scenes, a global variable with NAME
is defined and is
bound to a SECTION
object. By convention, section names start
with the character @. See Tutorial for an example.
ENTRIES
consists of docstrings and references. Docstrings are
arbitrary strings in markdown format, references are defined in the
form:
(symbol locative)
For example, (FOO FUNCTION)
refers to the function FOO
, (@BAR
SECTION)
says that @BAR
is a subsection of this
one. (BAZ (METHOD () (T T T)))
refers to the default method of the
three argument generic function BAZ
. (FOO FUNCTION)
is equivalent
to (FOO (FUNCTION))
.
A locative in a reference can either be a symbol or it can be a list
whose CAR
is a symbol. In either case, the symbol is the called the
type of the locative while the rest of the elements are the locative
arguments. See Locative Types for the list of locative types
available out of the box.
The same symbol can occur multiple times in a reference, typically with different locatives, but this is not required.
The references are not looked up (see RESOLVE
in the
Extension API) until documentation is generated, so it is allowed
to refer to things yet to be defined.
If EXPORT
is true (the default), the referenced symbols and NAME
are
candidates for exporting. A candidate symbol is exported if
it is accessible in PACKAGE
(it's not OTHER-PACKAGE:SOMETHING
) and
there is a reference to it in the section being defined with a
locative whose type is approved by EXPORTABLE-LOCATIVE-TYPE-P
.
See DEFINE-PACKAGE
if you use the export feature. The idea with
confounding documentation and exporting is to force documentation of
all exported symbols.
When DISCARD-DOCUMENTATION-P
(defaults to *DISCARD-DOCUMENTATION-P*
)
is true, ENTRIES
will not be recorded to save memory.
[variable] *DISCARD-DOCUMENTATION-P* NIL
The default value of DEFSECTION
's DISCARD-DOCUMENTATION-P
argument.
One may want to set *DISCARD-DOCUMENTATION-P*
to true before
building a binary application.
[macro] DEFINE-PACKAGE PACKAGE &REST OPTIONS
This is like CL:DEFPACKAGE
but silences warnings and errors
signaled when the redefined package is at variance with the current
state of the package. Typically this situation occurs when symbols
are exported by calling EXPORT
(as is the case with DEFSECTION
) as
opposed to adding :EXPORT
forms to the DEFPACKAGE
form and the
package definition is reevaluated. See the section on package
variance in the SBCL
manual.
The bottom line is that if you rely on DEFSECTION
to do the
exporting, then you'd better use DEFINE-PACKAGE
.
[function] DOCUMENT OBJECT &KEY STREAM PAGES (FORMAT :MARKDOWN)
Write OBJECT
in FORMAT
to STREAM
diverting some output to PAGES
.
FORMAT
can be anything 3BMD supports which is
currently :MARKDOWN
, :HTML
and :PLAIN
. STREAM
may be a stream
object, T
or NIL
as with CL:FORMAT
.
Most often, this function is called on section objects
like (DOCUMENT @PAX-MANUAL)
, but it supports all kinds of objects
for which DOCUMENT-OBJECT
is defined. To look up the documentation
of function DOCUMENT
:
(document #'document)
To generate the documentation for separate libraries with automatic cross-links:
(document (list @cube-manual @mat-manual))
Note that not only first class objects can have documentation. For
instance, variables and deftypes are not represented by objects.
That's why CL:DOCUMENTATION
has a DOC-TYPE
argument. DOCUMENT
does
not have anything like that, instead it relies on REFERENCE
objects
to carry the extra information. We are going to see later how
references and locatives work. Until then, here is an example on how
to look up the documentation of type FOO
:
(document (locate 'foo 'type))
One can call DESCRIBE
on SECTION
objects to get
documentation in markdown format with less markup than the default.
See DESCRIBE-OBJECT
(METHOD () (SECTION T))
.
There are quite a few special variables that affect how output is generated, see Documentation Printer Variables.
The rest of this description deals with how to generate multiple pages.
The PAGES
argument is to create multi-page documents by routing some
of the generated output to files, strings or streams. PAGES
is a
list of page specification elements. A page spec is a plist with
keys :OBJECTS
, :OUTPUT
, :URI-FRAGMENT
, :HEADER-FN
and :FOOTER-FN
.
OBJECTS
is a list of objects (references are allowed but not
required) whose documentation is to be sent to OUTPUT
. OUTPUT
can be a number things:
If it's a list whose first element is a string or a pathname, then
output will be sent to the file denoted by that and the rest of
the elements of the list are passed on as arguments to CL:OPEN
.
One extra keyword argument is :ENSURE-DIRECTORIES-EXIST
. If it's
true, ENSURE-DIRECTORIES-EXIST
will be called on the pathname
before it's opened.
If it's NIL
, then output will be collected in a string.
If it's T
, then output will be sent to *STANDARD-OUTPUT*
.
If it's a stream, then output will be sent to that stream.
If some pages are specified, DOCUMENT
returns a list of designators
for generated output. If a page whose OUTPUT
refers to a file that
was created (which doesn't happen if nothing would be written to
it), then the corresponding pathname is included in the list. For
strings the string itself, while for streams the stream object is
included in the list. This way it's possible to write some pages to
files and some to strings and have the return value indicate what
was created. The output designators in the returned list are ordered
by creation time.
If no PAGES
are specified, DOCUMENT
returns a single pathname,
string or stream object according to the value of the STREAM
argument.
Note that even if PAGES
is specified, STREAM
acts as a catch all
taking the generated documentation for references not claimed by any
pages. Also, the filename, string or stream corresponding to STREAM
is always the first element in list of generated things that is the
return value.
HEADER-FN
, if not NIL
, is a function of a single stream argument
which is called just before the first write to the page.
Since :FORMAT
:HTML
only generates HTML fragments, this makes it
possible to print arbitrary headers, typically setting the title,
css stylesheet, or charset.
FOOTER-FN
is similar to HEADER-FN
, but it's called after the last
write to the page. For HTML, it typically just closes the body.
Finally, URI-FRAGMENT
is a string such as "doc/manual.html"
that
specifies where the page will be deployed on a webserver. It defines
how links between pages will look. If it's not specified and OUTPUT
refers to a file, then it defaults to the name of the file. If
URI-FRAGMENT
is NIL
, then no links will be made to or from that
page.
It may look something like this:
`((;; The section about SECTIONs and everything below it ...
:objects (,@pax-sections)
;; ... is so boring that it's not worth the disk space, so
;; send it to a string.
:output (nil)
;; Explicitly tell other pages not to link to these guys.
:uri-fragment nil)
;; Send the @PAX-EXTENSIONS section and everything reachable
;; from it ...
(:objects (,@pax-extension-api)
;; ... to build/tmp/pax-extension-api.html.
:output ("build/tmp/pax-extension-api.html")
;; However, on the web server html files will be at this
;; location relative to some common root, so override the
;; default:
:uri-fragment "doc/dev/pax-extension-api.html"
;; Set html page title, stylesheet, charset.
:header-fn 'write-html-header
;; Just close the body.
:footer-fn 'write-html-footer)
(:objects (,@pax-manual)
:output ("build/tmp/manual.html")
;; Links from the extension api page to the manual page will
;; be to ../user/pax-manual#<anchor>, while links going to
;; the opposite direction will be to
;; ../dev/pax-extension-api.html#<anchor>.
:uri-fragment "doc/user/pax-manual.html"
:header-fn 'write-html-header
:footer-fn 'write-html-footer))
The Markdown in docstrings is processed with the 3BMD library.
Docstrings can be indented in any of the usual styles. PAX normalizes indentation by converting:
(defun foo ()
"This is
indented
differently")
to
(defun foo ()
"This is
indented
differently")
See DOCUMENT-OBJECT
for the details.
For syntax highlighting, github's fenced code blocks markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:
```elisp
(defun foo ())
```
to get syntactically marked up HTML output. Copy doc/style.css
from PAX and you are set. The language tag, elisp
in this example,
is optional and defaults to common-lisp
.
See the documentation of 3BMD and colorize for the details.
Docstrings are assumed to be in markdown format and they are pretty much copied verbatim to the documentation subject to a few knobs described below.
[variable] *DOCUMENT-UPPERCASE-IS-CODE* T
When true, words with at least three characters and no lowercase
characters naming an interned symbol are assumed to be code as if
they were marked up with backticks which is especially useful when
combined with *DOCUMENT-LINK-CODE*
. For example, this docstring:
"`FOO` and FOO."
is equivalent to this:
"`FOO` and `FOO`."
iff FOO
is an interned symbol. To suppress this behavior, add a
backslash to the beginning of the symbol or right after the leading
* if it would otherwise be parsed as markdown emphasis:
"\\MGL-PAX *\\DOCUMENT-NORMALIZE-PACKAGES*"
The number of backslashes is doubled above because that's how the
example looks in a docstring. Note that the backslash is discarded
even if *DOCUMENT-UPPERCASE-IS-CODE*
is false.
[variable] *DOCUMENT-LINK-CODE* T
When true, during the process of generating documentation for a
SECTION
, html anchors are added before the documentation of
every reference that's not to a section. Also, markdown style
reference links are added when a piece of inline code found in a
docstring refers to a symbol that's referenced by one of the
sections being documented. Assuming BAR
is defined, the
documentation for:
(defsection @foo
(foo function)
(bar function))
(defun foo (x)
"Calls `BAR` on `X`."
(bar x))
would look like this:
- [function] FOO X
Calls [`BAR`][1] on `X`.
Instead of BAR
, one can write [bar][]
or [`bar`][]
as well.
Since symbol names are parsed according to READTABLE-CASE
, character
case rarely matters.
Now, if BAR
has references with different locatives:
(defsection @foo
(foo function)
(bar function)
(bar type))
(defun foo (x)
"Calls `BAR` on `X`."
(bar x))
then documentation would link to all interpretations:
- [function] FOO X
Calls `BAR`([`1`][link-id-1] [`2`][link-id-2]) on `X`.
This situation occurs in PAX with SECTION
(0
1
) which is both a class (see
SECTION
) and a locative type denoted by a symbol (see
SECTION
). Back in the example above, clearly,
there is no reason to link to type BAR
, so one may wish to select
the function locative. There are two ways to do that. One is to
specify the locative explicitly as the id of a reference link:
"Calls [BAR][function] on X."
However, if in the text there is a locative immediately before or after the symbol, then that locative is used to narrow down the range of possibilities. This is similar to what the M-. extension does. In a nutshell, if M-. works without questions then the documentation will contain a single link. So this also works without any markup:
"Calls function `BAR` on X."
This last option needs backticks around the locative if it's not a single symbol.
Note that *DOCUMENT-LINK-CODE*
can be combined with
*DOCUMENT-UPPERCASE-IS-CODE*
to have links generated for
uppercase names with no quoting required.
[variable] *DOCUMENT-LINK-SECTIONS* T
When true, html anchors are generated before the heading of
sections which allows the table of contents to contain links and
also code-like references to sections (like @FOO-MANUAL
) to be
translated to links with the section title being the name of the
link.
[variable] *DOCUMENT-MIN-LINK-HASH-LENGTH* 4
Recall that markdown reference style links (like [label][id]
) are
used for linking to sections and code. It is desirable to have ids
that are short to maintain legibility of the generated markdown, but
also stable to reduce the spurious diffs in the generated
documentation which can be a pain in a version control system.
Clearly, there is a tradeoff here. This variable controls how many characters of the md5 sum of the full link id (the reference as a string) are retained. If collisions are found due to the low number of characters, then the length of the hash of the colliding reference is increased.
This variable has no effect on the html generated from markdown, but it can make markdown output more readable.
[variable] *DOCUMENT-MARK-UP-SIGNATURES* T
When true, some things such as function names and arglists are rendered as bold and italic.
[variable] *DOCUMENT-MAX-NUMBERING-LEVEL* 3
A non-negative integer. In their hierarchy, sections on levels less
than this value get numbered in the format of 3.1.2
. Setting it to
0 turns numbering off.
[variable] *DOCUMENT-MAX-TABLE-OF-CONTENTS-LEVEL* 3
A non-negative integer. Top-level sections are given a table of
contents which includes a nested tree of section titles whose depth
is limited by this value. Setting it to 0 turns generation of the
table of contents off. If *DOCUMENT-LINK-SECTIONS*
is true, then the
table of contents will link to the sections.
[variable] *DOCUMENT-TEXT-NAVIGATION* NIL
If true, then before each heading a line is printed with links to
the previous, parent and next section. Needs
*DOCUMENT-LINK-SECTIONS*
to be on to work.
[variable] *DOCUMENT-FANCY-HTML-NAVIGATION* T
If true and the output format is HTML, then headings get a
navigation component that consists of links to the previous, parent,
next section and a permalink. This component is normally hidden, it
is visible only when the mouse is over the heading. Needs
*DOCUMENT-LINK-SECTIONS*
to be on to work.
[variable] *DOCUMENT-NORMALIZE-PACKAGES* T
If true, symbols are printed relative to SECTION-PACKAGE
of the
innermost containing section or with full package names if there is
no containing section. To eliminate ambiguity [in package ...]
messages are printed right after the section heading if necessary.
If false, symbols are always printed relative to the current
package.
These are the locatives type supported out of the box. As all locative types, they are symbols and their names should make it obvious what kind of things they refer to. Unless otherwise noted, locatives take no arguments.
[locative] ASDF/SYSTEM:SYSTEM
Refers to an asdf system. The generated documentation will include meta information extracted from the system definition. This also serves as an example of a symbol that's not accessible in the current package and consequently is not exported.
[locative] SECTION
Refers to a section defined by DEFSECTION
.
[locative] VARIABLE &OPTIONAL INITFORM
Refers to a global special variable. INITFORM
, or if not specified,
the global value of the variable is included in the documentation.
[locative] CONSTANT &OPTIONAL INITFORM
Refers to a DEFCONSTANT
. INITFORM
, or if not specified,
the value of the constant is included in the documentation.
[locative] FUNCTION
Note that the arglist in the generated documentation depends on
the quality of SWANK-BACKEND:ARGLIST
. It may be that default
values of optional and keyword arguments are missing.
[locative] METHOD METHOD-QUALIFIERS METHOD-SPECIALIZERS
See CL:FIND-METHOD
for the description of the arguments.
To refer to the default method of the three argument generic
function FOO
:
(foo (method () (t t t)))
[locative] ACCESSOR CLASS-NAME
To refer to an accessor named FOO-SLOT
of class
FOO
:
(foo-slot (accessor foo))
[locative] READER CLASS-NAME
To refer to a reader named FOO-SLOT
of class
FOO
:
(foo-slot (reader foo))
[locative] WRITER CLASS-NAME
To refer to a writer named FOO-SLOT
of class
FOO
:
(foo-slot (writer foo))
[locative] STRUCTURE-ACCESSOR
This is a synonym of FUNCTION
with the difference that the often
ugly and certainly uninformative lambda list will not be printed.
[locative] TYPE
TYPE
can refer to classes as well, but it's better style to use the
more specific CLASS
locative type for that. Another difference to
CLASS
is that an attempt is made at printing the arguments of type
specifiers.
[locative] LOCATIVE LAMBDA-LIST
This is the locative for locatives. When M-. is pressed on
VARIABLE
in (VARIABLE LOCATIVE)
, this is what makes it possible
to land at the (DEFINE-LOCATIVE-TYPE VARIABLE ...)
form.
Similarly, (LOCATIVE LOCATIVE)
leads to this very definition.
While Common Lisp has rather good introspective abilities, not
everything is first class. For example, there is no object
representing the variable defined with (DEFVAR
FOO)
. (MAKE-REFERENCE 'FOO 'VARIABLE)
constructs a REFERENCE
that
captures the path to take from an object (the symbol FOO
) to an
entity of interest (for example, the documentation of the variable).
The path is called the locative. A locative can be applied to an
object like this:
(locate 'foo 'variable)
which will return the same reference as (MAKE-REFERENCE 'FOO
'VARIABLE)
. Operations need to know how to deal with references
which we will see in LOCATE-AND-COLLECT-REACHABLE-OBJECTS
,
LOCATE-AND-DOCUMENT
and LOCATE-AND-FIND-SOURCE
.
Naturally, (LOCATE 'FOO 'FUNCTION)
will simply return #'FOO
, no
need to muck with references when there is a perfectly good object.
[function] LOCATE OBJECT LOCATIVE &KEY (ERRORP T)
Follow LOCATIVE
from OBJECT
and return the object it leads to or a
REFERENCE
if there is no first class object corresponding to the
location. If ERRORP
, then a LOCATE-ERROR
condition is signaled when
lookup fails.
[condition] LOCATE-ERROR ERROR
Signaled by LOCATE
when lookup fails and ERRORP
is
true.
[function] RESOLVE REFERENCE &KEY (ERRORP T)
A convenience function to LOCATE
REFERENCE
's object with its
locative.
[class] REFERENCE
A REFERENCE
represents a path (REFERENCE-LOCATIVE
)
to take from an object (REFERENCE-OBJECT
).
[function] LOCATIVE-TYPE LOCATIVE
The first element of LOCATIVE
if it's a list. If it's a symbol then
it's that symbol itself. Typically, methods of generic functions
working with locatives take locative type and locative args as
separate arguments to allow methods have eql specializers on the type
symbol.
[function] LOCATIVE-ARGS LOCATIVE
The REST
of LOCATIVE
if it's a list. If it's a symbol then
it's ().
One may wish to make the DOCUMENT
function and M-. navigation work
with new object types. Extending DOCUMENT
can be done by defining a
DOCUMENT-OBJECT
method. To allow these objects to be referenced from
DEFSECTION
a LOCATE-OBJECT
method is to be defined. Finally, for M-.
FIND-SOURCE
can be specialized. Finally, EXPORTABLE-LOCATIVE-TYPE-P
may be overridden if exporting does not makes sense. Here is a
stripped down example of how all this is done for ASDF:SYSTEM:
(define-locative-type asdf:system ()
"Refers to an asdf system.")
(defmethod exportable-locative-type-p ((locative-type (eql 'asdf:system)))
nil)
(defmethod locate-object (symbol (locative-type (eql 'asdf:system))
locative-args)
(assert (endp locative-args))
(or (asdf:find-system symbol nil)
(locate-error)))
(defmethod canonical-reference ((system asdf:system))
(make-reference (asdf/find-system:primary-system-name system)
'asdf:system))
(defmethod document-object ((system asdf:system) stream)
(with-heading (stream system
(format nil "~A ASDF System Details"
(asdf/find-system:primary-system-name
system)))
(format stream "Some content~%")))
(defmethod find-source ((system asdf:system))
`(:location
(:file ,(namestring (asdf/system:system-source-file system)))
(:position 1)
(:snippet "")))
[macro] DEFINE-LOCATIVE-TYPE LOCATIVE-TYPE LAMBDA-LIST &BODY DOCSTRING
Declare LOCATIVE-TYPE
as a LOCATIVE
. One gets two
things in return: first, a place to document the format and
semantics of LOCATIVE-TYPE
(in LAMBDA-LIST
and DOCSTRING
); second,
being able to reference (LOCATIVE-TYPE LOCATIVE)
. For example, if
you have:
(define-locative-type variable (&optional initform)
"Dummy docstring.")
then (VARIABLE LOCATIVE)
refers to this form.
[generic-function] EXPORTABLE-LOCATIVE-TYPE-P LOCATIVE-TYPE
Return true iff symbols in references with
LOCATIVE-TYPE
are to be exported by default when they occur in a
DEFSECTION
. The default method returns T
, while the methods for
PACKAGE
, ASDF:SYSTEM
and METHOD
return NIL
.
DEFSECTION
calls this function to decide what symbols to export when
its EXPORT
argument is true.
[generic-function] LOCATE-OBJECT OBJECT LOCATIVE-TYPE LOCATIVE-ARGS
Return the object OBJECT
+ locative refers to. For
example, if LOCATIVE-TYPE
is the symbol PACKAGE
, this
returns (FIND-PACKAGE SYMBOL)
. Signal a LOCATE-ERROR
condition by
calling the LOCATE-ERROR
function if lookup fails. Signal other
errors if the types of the argument are bad, for instance
LOCATIVE-ARGS
is not the empty list in the package example. If a
REFERENCE
is returned then it must be canonical in the sense that
calling CANONICAL-REFERENCE
on it will return the same reference.
For extension only, don't call this directly.
[function] LOCATE-ERROR &REST FORMAT-AND-ARGS
Call this function to signal a LOCATE-ERROR
condition from a
LOCATE-OBJECT
method. FORMAT-AND-ARGS
contains a format string and
args suitable for FORMAT
from which the LOCATE-ERROR-MESSAGE
is
constructed. If FORMAT-AND-ARGS
is NIL
, then the message will be NIL
too.
The object and the locative are not specified, they are added by
LOCATE
when it resignals the condition.
[generic-function] CANONICAL-REFERENCE OBJECT
Return a REFERENCE
that resolves to OBJECT
.
[generic-function] COLLECT-REACHABLE-OBJECTS OBJECT
Return a list of objects representing all things
that would be documented in a (DOCUMENT
OBJECT
) call. For sections
this is simply the union of references reachable from references in
SECTION-ENTRIES
. The returned objects can be anything provided that
CANONICAL-REFERENCE
works on them. The list need not include OBJECT
itself.
One only has to specialize this for new container-like objects.
[method] COLLECT-REACHABLE-OBJECTS OBJECT
This default implementation returns the empty list. This means that
nothing is reachable from OBJECT
.
[generic-function] DOCUMENT-OBJECT OBJECT STREAM
Write OBJECT
(and its references recursively) in
FORMAT
to STREAM
.
The DOCUMENT
function calls this generic function with LEVEL
0,
passing FORMAT
on. Add methods specializing on OBJECT
to customize
how objects of that type are presented in the documentation.
[method] DOCUMENT-OBJECT (STRING STRING) STREAM
Print STRING
verbatim to STREAM
after cleaning up indentation.
Docstrings in sources are indented in various ways which can easily mess up markdown. To handle the most common cases leave the first line alone, but from the rest of the lines strip the longest run of leading spaces that is common to all non-blank lines.
[generic-function] FIND-SOURCE OBJECT
Like SWANK:FIND-DEFINITION-FOR-THING
, but this one
is a generic function to be extensible. In fact, the default
implementation simply defers to SWANK:FIND-DEFINITION-FOR-THING
.
This function is called by LOCATE-DEFINITION-FOR-EMACS
which lies
behind the M-. extension (see Emacs Integration).
If successful, the return value looks like this:
(:location (:file "/home/mega/own/mgl/pax/test/test.lisp")
(:position 24) nil)
The NIL
is the source snippet which is optional. Note that position
1 is the first character. If unsuccessful, the return values is
like:
(:error "Unknown source location for SOMETHING")
Let's see how to extend DOCUMENT
and M-. navigation if there is no
first class object to represent the thing of interest. Recall that
LOCATE
returns a REFERENCE
object in this case. DOCUMENT-OBJECT
and
FIND-SOURCE
defer to LOCATE-AND-DOCUMENT
and LOCATE-AND-FIND-SOURCE
which have LOCATIVE-TYPE
in their argument list for EQL
specializing
pleasure. Here is a stripped down example of how the VARIABLE
locative is defined:
(define-locative-type variable (&optional initform)
"Refers to a global special variable. INITFORM, or if not specified,
the global value of the variable is included in the documentation.")
(defmethod locate-object (symbol (locative-type (eql 'variable))
locative-args)
(assert (<= (length locative-args) 1))
(make-reference symbol (cons locative-type locative-args)))
(defmethod locate-and-document (symbol (locative-type (eql 'variable))
locative-args stream)
(destructuring-bind (&optional (initform nil initformp)) locative-args
(format stream "- [~A] " (string-downcase locative-type))
(print-name (prin1-to-string symbol) stream)
(write-char #Space stream)
(multiple-value-bind (value unboundp) (symbol-global-value symbol)
(print-arglist (prin1-to-string (cond (initformp initform)
(unboundp "-unbound-")
(t value)))
stream))
(terpri stream)
(maybe-print-docstring symbol locative-type stream)))
(defmethod locate-and-find-source (symbol (locative-type (eql 'variable))
locative-args)
(declare (ignore locative-args))
(find-one-location (swank-backend:find-definitions symbol)
'("variable" "defvar" "defparameter"
"special-declaration")))
[method] COLLECT-REACHABLE-OBJECTS (REFERENCE REFERENCE)
If REFERENCE
can be resolved to a non-reference, call
COLLECT-REACHABLE-OBJECTS
with it, else call
LOCATE-AND-COLLECT-REACHABLE-OBJECTS
on the object, locative-type,
locative-args of REFERENCE
[generic-function] LOCATE-AND-COLLECT-REACHABLE-OBJECTS OBJECT LOCATIVE-TYPE LOCATIVE-ARGS
Called by COLLECT-REACHABLE-OBJECTS
on REFERENCE
objects, this function has essentially the same purpose as its
caller but it has different arguments to allow specializing on
LOCATIVE-TYPE
.
[method] LOCATE-AND-COLLECT-REACHABLE-OBJECTS OBJECT LOCATIVE-TYPE LOCATIVE-ARGS
This default implementation returns the empty list. This means that nothing is reachable from the reference.
[method] DOCUMENT-OBJECT (REFERENCE REFERENCE) STREAM
If REFERENCE
can be resolved to a non-reference, call
DOCUMENT-OBJECT
with it, else call LOCATE-AND-DOCUMENT-OBJECT on the
object, locative-type, locative-args of REFERENCE
[generic-function] LOCATE-AND-DOCUMENT OBJECT LOCATIVE-TYPE LOCATIVE-ARGS STREAM
Called by DOCUMENT-OBJECT
on REFERENCE
objects,
this function has essentially the same purpose as DOCUMENT-OBJECT
but it has different arguments to allow specializing on
LOCATIVE-TYPE
.
[method] FIND-SOURCE (REFERENCE REFERENCE)
If REFERENCE
can be resolved to a non-reference, call FIND-SOURCE
with it, else call LOCATE-AND-FIND-SOURCE
on the object,
locative-type, locative-args of REFERENCE
[generic-function] LOCATE-AND-FIND-SOURCE OBJECT LOCATIVE-TYPE LOCATIVE-ARGS
Called by FIND-SOURCE
on REFERENCE
objects, this
function has essentially the same purpose as FIND-SOURCE
but it has
different arguments to allow specializing on LOCATIVE-TYPE
.
[method] LOCATE-AND-FIND-SOURCE OBJECT LOCATIVE-TYPE LOCATIVE-ARGS
This default implementation simply calls FIND-SOURCE
with OBJECT
which should cover the common case of a macro expanding to, for
instance, a defun but having its own locative type.
We have covered the basic building blocks of reference based
extensions. Now let's see how the obscure
DEFINE-SYMBOL-LOCATIVE-TYPE
and
DEFINE-DEFINER-FOR-SYMBOL-LOCATIVE-TYPE
macros work together to
simplify the common task of associating definition and documentation
with symbols in a certain context.
[macro] DEFINE-SYMBOL-LOCATIVE-TYPE LOCATIVE-TYPE LAMBDA-LIST &BODY DOCSTRING
Similar to DEFINE-LOCATIVE-TYPE
but it assumes that all things
locatable with LOCATIVE-TYPE
are going to be just symbols defined
with a definer defined with DEFINE-DEFINER-FOR-SYMBOL-LOCATIVE-TYPE
.
It is useful to attach documentation and source location to symbols
in a particular context. An example will make everything clear:
(define-symbol-locative-type direction ()
"A direction is a symbol. (After this M-. on `DIRECTION LOCATIVE`
works and it can also be included in DEFSECTION forms.)")
(define-definer-for-symbol-locative-type define-direction direction ()
"With DEFINE-DIRECTION one document how what a symbol means when
interpreted as a direction.")
(define-direction up ()
"UP is equivalent to a coordinate delta of (0, -1).")
After all this, (UP DIRECTION)
refers to the DEFINE-DIRECTION
form above.
[macro] DEFINE-DEFINER-FOR-SYMBOL-LOCATIVE-TYPE NAME LOCATIVE-TYPE &BODY DOCSTRING
Define a macro with NAME
which can be used to attach documentation,
a lambda-list and source location to a symbol in the context of
LOCATIVE-TYPE
. The defined macro's arglist is (SYMBOL
LAMBDA-LIST
&OPTIONAL DOCSTRING
). LOCATIVE-TYPE
is assumed to have been defined
with DEFINE-SYMBOL-LOCATIVE-TYPE
.
Section
objects rarely need to be dissected since
DEFSECTION
and DOCUMENT
cover most needs. However, it is plausible
that one wants to subclass them and maybe redefine how they are
presented.
[class] SECTION
DEFSECTION
stores its NAME
, TITLE
and ENTRIES
in
SECTION
objects.
[reader] SECTION-NAME SECTION
The name of the global variable whose value is this section object.
[reader] SECTION-TITLE SECTION
Used in generated documentation.
[reader] SECTION-ENTRIES SECTION
A list of strings and REFERENCE
objects in the
order they occurred in DEFSECTION
.
[method] DESCRIBE-OBJECT (SECTION SECTION) STREAM
SECTION
objects are printed by calling DOCUMENT
on them
with all Documentation Printer Variables, except for
*DOCUMENT-NORMALIZE-PACKAGES*
, turned off to reduce clutter.