How templates work

Behind the scenes

In the previous chapter, we had fun creating some custom templates for the Mau source code, but there was no explanation of what actually happens and how Mau selects and uses templates.

In this chapter, you will learn what happens behind the scenes, how to decide which template file to create, and what variables are available to the template itself.

The Abstract Syntax Tree

To start off on our journey with templates let's have a look at how Mau parses text. Let's assume the input is

Mau source
Stars identify *important* text.

This piece of text is processed by Mau and an internal representation called Abstract Syntax Tree (AST) is created. A compact version of the AST for this input is

Abstract Syntax Tree
_type: document  1
content:

- _type: paragraph  2
  content:
  
  - _type: paragraph-line  3
    content:
    
    - _type: text  4
      value: 'Stars identify '
      
    - _type: style  5
      content:
      
      - _type: text  7
        value: important
      style: star
      
    - _type: text  6
      value: ' text.'

Here, you can see that Mau creates a document node 1 for the whole input. The node contains exactly one paragraph node 2 made of one line 3.

The first and only paragraph line contains 3 nodes: a text node 4 with the value Stars identify , a style node 5, and another text node 6 with the text text..

Finally, the style node 5 contains a specific style name star and a text node 7 with the value important.

You can always see the AST created from your document using the visitor core:YamlVisitor. However, nodes contain much more information than you see above, and the AST is pretty difficult to read, so this option is recommended only if you are debugging Mau's source code.

How Mau renders nodes

Once the AST has been created, the visitor processes each node in a depth-first fashion. Each visited node is transformed into text according to the nature of the visitor, and the result is saved into the output file.

For example, the AST above results in the following processing steps:

  1. Process the first text node 4.
  2. Process the text node inside the style node 7.
  3. Process the style node 5.
  4. Process the second text node 6.
  5. Process the paragraph line node 3.
  6. Process the paragraph node 2.
  7. Process the document node 1.

The main type of visitor in Mau is base on Jinja, even though Mau is flexible enough to allow for other template engines to be used. The Jinja visitor processes each node passing it to a Jinja template, which is used to render the resulting text.

An example of what happens using the HTML visitor (based on the Jinja visitor) is the following.

  1. First text node 4. Load the template text.html, that renders {{ value }}. The result is the string |Stars identify | (the vertical lines are used here to highlight the presence of spaces and are not part of the output).
  2. Text node inside the style node 7. Once again, the template is text.html and the output is |important|.
  3. Style node 5. The template in this case is style.style__star.html that renders <strong>{{ content }}</strong>. The variable content mentioned here is the rendering of the contained node, which is the string |important|. The result is then |<strong>important</strong>|.
  4. Second text node 6. The process is the same as for the first text node and the result is the string | text.|.
  5. The result of the three nodes contained in the paragraph line is then |Stars identify <strong>important</strong> text.|.
  6. Paragraph line node 3. The template paragraph-line.html renders {{ content }}, which is the string seen in the previous step without additional text or tags.
  7. Paragraph node 2. The template paragraph.html renders <p>{{ content }}</p>, where content is the list of all paragraph lines, joined with a space. The output is the string |<p>Stars identify <strong>important</strong> text.</p>|
  8. Process the document node 1. The template document.html renders <html><head></head><body>{{ content }}</body></html> and the result is the final output, the string <html><head></head><body><p>Stars identify <strong>important</strong> text.</p></body></html>

How nodes are structured

Each node in Mau has a standard base structure and custom fields that depend on its type. The structure is

  • _type
  • parent
  • args
  • kwargs
  • tags
  • subtype
  • Custom fields

Consider for example the style node 5 corresponding to *important*. The text node 7 contained into it is

_type: text
parent: <the style node>
args: []
internal_tags: []
kwargs: {}
tags: []
subtype: null
value: important

While custom fields vary among node types, two fields are present in many nodes: content and labels. The first is a list of child nodes that are "contained" inside the node (e.g. paragraph lines for paragraph, text nodes inside a header), while the second is a dictionary of all the labels attached to the node.

For example, consider again the style node 5 corresponding to *important*. The node itself is

_type: style
parent: <the paragraph node>
args: []
internal_tags: []
kwargs: {}
tags: []
subtype: null
style: star
content:
  - <the text node>

where <the text node> is the text node we discussed previously.

Template naming and selection

Mau select templates in a way that is similar to how CSS rules are selected for HTML pages.

The process happens in three steps:

  1. Once all templates have been loaded, they are sorted in decreasing specificity order.
  2. The sorted templates are tested against the node being processed to see if they match it or not.
  3. The most specific matching template is selected.

Templates declare their specificity through their name. This can be the file name (for templates provided through the file system), or the name of the environment variable (for templates provided through the configuration).

The schema used to name a template is:

[NODE TYPE][OPTIONAL FIELDS][.EXTENSION]
  • The node type is provided in the documentation of every node, e.g. header.
  • The extension is enforced by the visitor and will be documented there, e.g. .html.
  • The optional fields are all in the form .NAME and can appear in any order. They are:
    • .SUBTYPE, where SUBTYPE is the value of the subtype, e.g. warning --> .warning.
    • .KEY__VALUE, where KEY and VALUE identify a custom template field of the node, e.g. style.style__star.html matches a style node with custom field style equal to star.
    • .tg_TAG, where TAG is the value of the tag, e.g. important --> .tg_important.
    • .pt_PARENT, where PARENT is the node type of the parent, e.g. document --> .pt_document.
    • .pts_SUBTYPE, where SUBTYPE is the value of the parent's subtype, e.g. quote --> .pts_quote.
    • .pts_KEY__VALUE, where KEY and VALUE identify a custom template field of the parent node, e.g. .pts_role__title matches a node whose parent has custom field role equal to title.
    • .pf_PREFIX, where PREFIX is a custom template prefix, e.g. page --> .pf_page.

Please note that prefixes haven't been discussed yet, so we will ignore them for the time being.

Example 1

The template paragraph.warning.html is decomposed into

  • Node type: paragraph
  • Extension: .html
  • Subtype: warning
  • Tags: []
  • Parent: None
  • Prefix: None

This will match any node that has:

  • The node type paragraph.
  • The subtype warning.
  • Any tag.
  • Any parent.
  • No prefix.

So, that template will match the first two document nodes in the following Mau document

[*warning]
This paragraph will use the template because the subtype is `warning`.

[*warning, #tag1]
This paragraph will use the template because the subtype is `warning`. The template doesn't care about the tags.

This paragraph will NOT use the template because the subtype is not `warning`.

// This header will NOT use the template because it is not a paragraph.
== Header 1

Example 2

The template block.tip.tg_important.html is decomposed into

  • Node type: block
  • Extension: .html
  • Subtype: tip
  • Tags: ["important"]
  • Parent: None
  • Prefix: None

Please note that the template block.tg_important.tip.html is going to be decomposed into the same object. The order of the optional fields is not important.

This will match any node that has:

  • The node type block.
  • The subtype tip.
  • At least the tag important
  • Any parent.
  • No prefix.

So, that template will match only two of the following blocks

[*tip, #important]
----
This block will use the template because the subtype is `tip` and it has the tag `important`.
----

[*tip, #important, #big]
----
This block will use the template because the subtype is `tip` and it has the tag `important`.
The additional tag `big` is not required by the template but also not excluded.
----

[*tip]
----
This block will NOT use the template because it doesn't have the tag `important`.
----

[#important]
----
This block will NOT use the template because it doesn't have the subtype `tip`.
----

Example 3

The template macro.name__kbd.html that we used in the previous chapter is decomposed into

  • Node type: macro
  • Extension: .html
  • Subtype: None
  • Tags: []
  • Parent: None
  • Prefix: None
  • Custom fields: {"name":"kbd"}

This will therefor match any node of type macro with the name kbd.

How Mau finds templates

Mau collects templates from several sources always in the same order. If two sources provide two templates with the same name, the one loaded later overrides the other. In the following list, sources are in loading order and thus in increasing order of importance.

  1. Templates from the current visitor.
  2. Templates from template plugins.
  3. Templates from the local filesystem.
  4. Templates from the configuration.

Templates from the current visitor

Each visitor can define a set of templates that are loaded directly into the Python class. The HTML visitor, for example, comes with a set of more than 30 templates for document, headers, lists, and so on.

Generally, visitor provide a template for every type of node that Mau supports, with a basic implementation.

Templates from template plugins

Mau plugins can provide new visitors and their templates or just templates. Such plugins are called template providers. A good example is the plugin Mau Docs Templates.

Template plugins must be explicitly loaded in the configuration file with the configuration value visitor.templates.providers. For example

visitor:
  templates:
    providers:
      - mau-docs-templates

Templates from the local filesystem

Mau scans the paths listed under the configuration value visitor.templates.paths in order. This is the method we used in the previous chapter to load the templates we stored in the local directory templates.

Paths are visited recursively, but there is no guarantee about the order of nested directories. Therefore, it's important not to have conflicting templates nested inside the same path. For example, if visitor.templates.paths contains the directory custom, the following case would result in unpredictable behaviour.

custom/
  oldtemplates/
    header.html
  templates/
    header.html

Mau might load oldtemplates/header.html first and templates/header.html second (and use the last one) or the opposite.

Templates from the configuration

It is possible to define templates directly inside the Mau configuration file under the configuration value visitor.templates.custom. This method is useful for small templates or for test customisations, but it might result in templates that are difficult to read or that require a huge amount of escape characters.

visitor:
  templates:
    custom:
      paragraph.html: "<p>{{ content }}</p>"

Template prefixes

Prefixes are a way to bulk select a subset of templates.

When the visitor checks which templates match the current node, it can be optionally given a list of strings called prefixes. When a prefix NAME is given to a visitor, templates must have the part pf_NAME in their name, and the visitor will try all listed prefixes in order.

For example, consider the following templates:

  • block.html
  • block.tip.html
  • block.pf_page.html
  • block.pf_page.tip.html

If the visitor is run without any prefix, the first two templates match blocks with the subtype tip (the second being more specific than the first). The third and fourth templates do not match, as the prefix is mandatory, and is currently None.

Conversely, if the visitor is run with the prefix page, the third and fourth templates will match (the fourth being more specific than the third), and the first two will not.

You can pass a list of prefixes through the environment variable mau.visitor.templates.prefixes. The visitor will try all of them in order.

Prefixes are a very convenient way to alter the rendering of a document according to the context. For example, a blog might use the mechanism to render standard posts and featured posts in a different way.

How to debug nodes

Sometimes, it's difficult to understand what information about a node is being passed to the visitor. One way to debug Mau's output is to use the visitor core:YamlVisitor, but the YAML output is often difficult to read, and contains all nodes in the document.

A more targeted way is to use a special debug tag #mau:debug. This tag is detected by the visitor and triggers a dump of the information available of the tagged node. This method can be used only for document nodes, which are however often the most problematic ones.

Example

Add the following node to your test files:

[#mau:debug]
----
This is just a normal block.
----

Then run Mau with the following command line

mau -i FILENAME.mau --verbose -o FILENAME.html -t 'mau_html_visitor:HtmlVisitor'

Please note the option --verbose that will actually print the information we need. The standard output of the command above will be:

INFO:main:[MAU] Visitor message
INFO:main:[MAU] Message: This is a debug message activated by a Mau internal tag.
INFO:main:[MAU] Context: test.mau:2,0-4,4
INFO:main:[MAU] Node type: block
INFO:main:[MAU] Template data - _type: block
INFO:main:[MAU] Template data - args: []
INFO:main:[MAU] Template data - kwargs: {}
INFO:main:[MAU] Template data - tags: []
INFO:main:[MAU] Template data - internal_tags: ['debug']
INFO:main:[MAU] Template data - subtype: None
INFO:main:[MAU] Template data - _context: {'start_line': 1, 'start_column': 0, 'end_line': 3, 'end_column': 4, 'source': 'test.mau'}
INFO:main:[MAU] Template data - parent: {'_type': 'document', 'args': [], 'kwargs': {}, 'tags': [], 'internal_tags': [], 'subtype': None, '_context': {'start_line': 1, 'start_column': 0, 'end_line': 3, 'end_column': 4, 'source': 'test.mau'}}
INFO:main:[MAU] Template data - classes: []
INFO:main:[MAU] Template data - content: <p>This is just a normal block.</p>
INFO:main:[MAU] Template data - labels: {}
INFO:main:[MAU] Available templates: ['block']
INFO:main:[MAU] Matching templates: ['block']
INFO:main:[MAU] Prefixes: []

Let's have an in-depth look at that. The header is guidance text, but it contains the context: test.mau:2,0-4,4 (in the format {source_file}:{line_start},{column_start}-{line_end},{column_end}).

INFO:main:[MAU] Visitor message
INFO:main:[MAU] Message: This is a debug message activated by a Mau internal tag.
INFO:main:[MAU] Context: test.mau:2,0-4,4

Indeed, you can notice that the block fences begin at line 2 and terminate at line 4. The opening fence starts at column 0 and the closing fence stops at column 4.

After that we have the type of block, which determines the main part of the template name.

INFO:main:[MAU] Node type: block

The next 11 lines are a dump of the data passed to the template. In this case, if the template contains the text {{ content }}, Jinja will replace it with the string <p>This is just a normal block.</p>. Please note that Jinja templates receive Python objects, so the text {{ parent._type }} is perfectly valid and would be translated into the string document.

INFO:main:[MAU] Template data - _type: block
INFO:main:[MAU] Template data - args: []
INFO:main:[MAU] Template data - kwargs: {}
INFO:main:[MAU] Template data - tags: []
INFO:main:[MAU] Template data - internal_tags: ['debug']
INFO:main:[MAU] Template data - subtype: None
INFO:main:[MAU] Template data - _context: {'start_line': 1, 'start_column': 0, 'end_line': 3, 'end_column': 4, 'source': 'test.mau'}
INFO:main:[MAU] Template data - parent: {'_type': 'document', 'args': [], 'kwargs': {}, 'tags': [], 'internal_tags': [], 'subtype': None, '_context': {'start_line': 1, 'start_column': 0, 'end_line': 3, 'end_column': 4, 'source': 'test.mau'}}
INFO:main:[MAU] Template data - classes: []
INFO:main:[MAU] Template data - content: <p>This is just a normal block.</p>
INFO:main:[MAU] Template data - labels: {}

Finally, we see a list of templates for this node type that Mau found in the system, a list of the templates that match the current node, and a list of prefixes loaded into the system.

INFO:main:[MAU] Available templates: ['block']
INFO:main:[MAU] Matching templates: ['block']
INFO:main:[MAU] Prefixes: []