Yodl 4.05.01

- Table of Contents
- Previous Chapter

# Chapter 6: Technical information

This chapter consists of various sections. The first section describes `Yodl` from the point of view of the systems administrator. Issues such as the installation of the package are addressed here. The second section describes `Yodl`'s technical implementation in some detail. Apart from the documentation about `Yodl` given here, much can be found in the individual source files. However, section 6.2 describes `the broad picture'. Having read section 6.2, it should be relatively easy to determine what happens where inside the `Yodl` program and the `yodl-post` post processor.

## 6.1: Obtaining Yodl

`Yodl` and the distributed macro package can be obtained from https://fbb-git.gitlab.io/yodl/. The associated git-repository page is located at https://gitlab.com/fbb-git/yodl.

Yodl can also be retrieved as a binary Debian package: to find it at http://www.debian.org/distrib/packages by entering `yodl` (in any distribution) in the form-fields you'll find on that page.

### 6.1.1: Installing Yodl

The binary package, distributed in `yodl-X.Y.Z_a.b.c.deb` can be installed using `dpkg -install yodl-X.Y.Z`. It installs:

- `Yodl`'s binaries in `/usr/bin`;
- `Yodl`'s macros in `/usr/share/yodl`
- `Yodl`'s documentation in `/usr/share/doc/yodl`;
- `Yodl`'s manpages in `/usr/share/man/man{1,7}`;

Local installations, not using the Debian installation process, can be obtained using the provided `icmake` build-script see below. An alternative (cf. `contrib/build.pl`) is to use the contributed `build.pl` script. Note that `build.pl` is not maintained by me. If you bump against problems when using it, I'll probably not be able to help you out.

If a local installation is preferred or required, unpack the file `yodl-X.Y.Z.tar.gz`. Next, chdir to the directory `yodl-X.Y.Z`, and optionally tweak the file `config` to your needs. Next, issue the command:

        build programs    (or use `build programs strip' to strip the binaries)
        build macros
        build man
        build manual

followed by

       install programs WHERE        (install the programs under WHERE,
                                      e.g. install programs /usr/bin)

       install macros WHERE          (install the macros under WHERE
                                      e.g., install macros /usr/share/yodl)

       install man WHERE             (install the man-pages under WHERE
                                      e.g., install man /usr/share/man)

       install manual WHERE          (install the manual under WHERE
                                      e.g., install manual /usr/share/doc/yodl)

       install docs WHERE            (install additional docs under WHERE
                                      e.g., install docs /usr/share/doc/yodl)

The installation process installs the binaries, manual pages, other documentation and macro files under the indicated directories.

However, by far the easiest way to install a binary distribution is to use the Debian `dpkg --install yodl*.deb` command. `Dpkg` installs the various parts according to Debian's conventions under `usr/`.

Installation from source requires you to have the following programs installed on your system:

- A recent C compiler and run-time environment. A POSIX-compliant compiler, libraries and set of header files should work without problems.
- A recent installation of `Icmake`: `Icmake` is part of the standard Debian distribution, and can also be obtained from https://fbb-git.gitlab.io/icmake/
- Standard tools, like `sed`, `grep`, `perl`, etc..
- `/bin/sh`: a POSIX-compliant shell interpreter. The GNU shell interpreter `bash` can be used instead.

## 6.2: Organization of the software

This section describes the organization of the source files. Its contents are not necessarily relevant for the binary distribution. The section is probably most useful to those readers who want to be able to extend or who want to do maintenance on `Yodl`'s sources, or who want simply to understand what's happening inside the `Yodl` program.

Much of the documentation is provided in the individual source files themselves. This section, however, should offer the `broad picture', allowing you to understand the logic behind `Yodl` relatively fast.

### 6.2.1: Subdirectories and their meanings

After unpacking `Yodl`'s source archive, the following directories are available:

- `yodl`: the root-directory of the `Yodl` tree. All sources and program maintenance scripts are found in or below this directory.
- `yodl/macros`: This directory contains all the macro definitions of the standard macro package. It contains the following subdirectories:
  - `yodl/macros/in`: This directory contains generic macro files. These macro files contain the word `@STD_INCLUDE@`, which is replaced by the standard include directory used in a particular distribution.
  - `yodl/macros/rawmacros`: This directory contains the raw macro definition files themselves. One file per raw macro. A raw macro contains the implementations of that macro for all supported conversion types, and has the extension `.raw`. Furthermore, this directory contains some support scripts: `create, separator.pl, startdoc.pl`.
  - `yodl/macros/yodl`: this is the directory to contain `Yodl`'s standard macros. The (recursive) contents of this directory is eventually copied by the installation procedure to the `.../share/yodl` directory, which then becomes `Yodl`'s standard include directory.
  - `yodl/macros/yodl/chartables`: This directory contains character-translation tables for various target languages.
  - `yodl/macros/yodl/xml`: This directory contains the XML frame files, used to convert `Yodl` documents to XML, as implemented by the `webplatform' of the University of Groningen. All these frame files have the extensions `.xml`.
- `yodl/man`: The raw source files of all man-pages: manpages of the `Yodl` program itself, of the yodl post-processor, of the conversion scripts, of the builtin-functions, of the standard macros and of `Yodl`'s `manpage` and `letter` document types. These raw source files have the extensions `.in`, indicating that they may contain `@STD_INCLUDE@` words, which is replaced by the eventually used standard include path.
  - `yodl/man/1`: The destination for `Yodl`'s manual pages in section 1 (programs).
  - `yodl/man/7`: The destination for `Yodl`'s manual pages in section 7 (macro packages and conventions).
- `yodl/manual`: The source files of the complete `Yodl` manual, as well as the directories for the various converted formats. The script `build`, found in this directory, constructs the manual in the subdirectories:
  - `yodl/manual/html`: the HTML-converted manual;
  - `yodl/manual/latex`: the LaTeX-version of the manual;
  - `yodl/manual/pdf`: the pdf-version of the manual;
  - `yodl/manual/ps`: the PostScript-version of the manual;
  - `yodl/manual/txt`: the plain text-version of the manual;
- `yodl/manual/`: The source files of the complete The `Yodl` document files themselves are located in subdirectories of this directory. They are:
  - `yodl/manual/converters`
  - `yodl/manual/intro`
  - `yodl/manual/macros`
  - `yodl/manual/technical`
  - `yodl/manual/userguide` (and various subdirectories)
- `yodl/scripts`: support scripts used by the building process: `configreplacements` replaces `@XXX@` words by their actual values as found in `yodl/src/config.h`; `yodl2whatever.in` is the generic yodl-converter, calling macros specific for a particular conversion type. This generic converter is installed in `.../bin/`, together with specific converters, installed as soft-links to this generic converter.
- `Yodl`'s base directory contains sub-directories holding the source files of the C programs `Yodl` and `yodl-post`, as well as all auxiliary directories containing sources of the (logical) components of these programs. Most of these components are like C++ classes in that they define a building block of the `Yodl` and/or `yodl-post` program. Their organization, interaction and relationship is described below. They are:
  - `yodl/args`: the component handling the command-line arguments;
  - `yodl/builtin`: the component handling `Yodl`'s builtin functions;
  - `yodl/chartab`: the component handling `Yodl`'s character table type;
  - `yodl/counter`: the component handling `Yodl`'s counter type;
  - `yodl/file`: the component handling all file operations (locating, opening, etc.);
  - `yodl/hashitem`: key/value combinations stored in `Yodl`'s hashtable;
  - `yodl/hashmap`: `Yodl`'s hashtable;
  - `yodl/lexer`: `Yodl`'s lexical scanner: this component consumes the `.yo` file, and produces a continuous stream of tokens to be handled by another component: the parser.
  - `yodl/lines`: the component storing lines of text, used by `yodl-post`.
  - `yodl/macro`: the component handling `Yodl`'s macro type;
  - `yodl/message`: the component handling all messages (warnings, errors, verbosity settings, etc.).
  - `yodl/new`: the component handling all memory allocations (except for duplicating strings, which is handled by the root-component).
  - `yodl/ostream`: the component handling all `Yodl`'s output to its output-file (`Yodl` may also output to strings, which is not handled by the ostream component).
  - `yodl/parser`: the component handling the tokens produced by the lexer-component. This component governs all actions to be taken during a conversion. Its actions all derive from its function `parser_process()`.
  - `yodl/postqueue`: the component handling the postprocessing required by most conversions.
  - `yodl/process`: the component handling the execution of child- or system-processes.
  - `yodl/queue`: the component allowing the lexical scanner to queue its input, awaiting further processing.
  - `yodl/root`: the component defining some basic typedefs and enumerations, as well as the `new_str()` function duplicating a string, and the `out_of_memory()` function handling memory allocation failures.
  - `yodl/stack`: the component implementing a stack data structure.
  - `yodl/string`: the component implementing a text-storage data structure and its functionality.
  - `yodl/strvector`: the component implementing a table of `SUBST` key-value pairs.
  - `yodl/subst`: the component handling `Yodl`'s SUBST definitions;
  - `yodl/symbol`: the component handling `Yodl`'s symbol type;
  - `yodl/yodl`: the sources of the `Yodl` program itself. This directory also contains the implementations of all builtin functions, whose filenames all start with `gram_` (E.g., `gramaddtocounter.c`).
  - `yodl/yodlpost`: the sources of the `yodl-post` program.
  The script `build`, found in this directory, compiles the programs and macros, storing them in:
  - `tmp/install/usr/bin` (the binary programs)
  - `tmp/install/usr/share/yodl` (the macros)
  - `tmp/install/usr/share/man` (the man-pages)
  - `tmp/install/usr/share/doc` (the manual and additional documentation)

## 6.3: Yodl's component interrelations and component setup

`Yodl`'s components show a strict hierarchical ordering. This allows the testing and development of components placed nearer to the component's tree without considering anything that's placed farther away.

The following piece of `ascii-art' shows the relationships for the `Yodl` program. The root of the tree starts at the top, at the `root` component. The tree can be read from the top to the bottom, where each horizontal line starts a level of components mentioned immediately below it, and each vertical route through the figure a series of components whose functioning depend on at least the components mentioned earlier.

However, a more natural way to look at it is to start somewhere in the tree, and see what's envountered going up. Doing so, all components that are required are visited. Once the figure shows a

            |
        --- | ---
            |

construction. This means that the horizontal line is not related to the vertical dependency crossing (but not touching) it.

                                    root
                                    |
                                    message
                                    |
                                    new
                                    |
                        +-------+---+-------+
                        |       |           |
                        string  queue       stack
                        |       |           |
        +--+--+---------+       |           hashitem
        |  |  |         |       |           |
        |  |  strvector |       |           |
        |  |  |         |       |           |
        |  |  +---------+       |           |
        |  |            |       |           |
        |  args         subst   |           hashmap
        |  |            |       |           |
        |  |            +-------+       +---+-------+
        |  |                    |       |           |
        |  |                    |       symbol  +---+----+-------+-------+
        |  |                    |       |       |        |       |       |
        |  +------------+------ | ------+       chartab  counter macro   builtin
        |               |       |               |        |       |       |
        |               file    |               +---+----+-------+-------+
        |               |       |                   |
        |               +---+---+                   |
        |                   |                       |
        |               +---+---+                   |
        |               |       |                   |
        process         lexer   ostream             |
        |               |       |                   |
        |               +-------+-------+-----------+
        |                               |
        |                               parser
        |                               |
        +-------------------------------+
                                        |
                                        (yodl)

A similar, albeit much simpler, tree can be drawn for `yodl-post`. Here is the organization of the components for the `yodl-post` program:

                                    root
                                    |
                                    message
                                    |
                                    new
                                    |
                          +-----+---+---+
                          |     |       |
                          |     |       |
                          lines string  hashitem
                          |     |       |
                          |     args    hashmap
                          |     |       |
                          |     +-------+
                          |     |
                          |     file
                          |     |
                          +-----+
                                |
                                postqueue
                                |
                                yodl2html-post

The source files of each component are organized as follows:

- All the files of a component are stored in a directory, named after the component. For example, the `counter` component is found in the directory

          yodl/counter

  containing all the (source) files that define that component.
- Each function is stored in a file of its own inside its component-directory. For example, the function `counter_value()` is defined in the source file `countervalue.c`.
- The file names are identical to the names of the functions, except for the fact that only lower case letters are used for the file names, and that the file names never use underscore characters.
- The `.h` header files declare the functions that can be used by other components. These functions are comparable to C++'s public members. Furthermore, these `.h` files define all structs and typedefs that are required for other components to use a particular component. For example, the `component.h` header file may contain

      #ifndef _INCLUDED_COUNTER_H_
      #define _INCLUDED_COUNTER_H_

      #include "../root/root.h"
      #include "../hashmap/hashmap.h"

      void        counter_add(HashItem *item, int add);   /* err  if no counter   */
      bool        counter_has_value(int *valuePtr, HashItem *item);
      Result      counter_insert(HashMap *symtab, char const *key, int value);
      void        counter_set(HashItem *item, int value); /* err  if no counter   */
      char const *counter_text(HashItem *item);       /* returns static buffer    */
      int         counter_value(HashItem *item);      /* err  if no stack/item    */

      #endif

- All functions declared in `.h` file start with the name of the component, and often contain an initial pointer to some `struct` containing the essential fields that are associated with that particular component. For example, most `counter_` functions have a `HashItem *` as their first argument, as a `HashItem` is normally used to store the details about a counter.
- The modifier `const` is used with pointers to indicate that the information pointed to by the pointer is `owned' by the provider of that information. With parameters it indicates that the caller owns the information, and the function will not modify the provided info; with return types it indicates that the function `owns' the returned information, which therefore may not be modified (or freed) by the caller of that function (e.g., `char const *counter_text`). The absence of `const` in combination with pointers indicates that the information pointed to by the pointer could, in principle, be modified by the code receiving the pointer value.
- Most components also show a `.ih` file, a so-called internal header file. The internal header declares `internal support functions', not to be used by other parts of the software, and defines internal typedefs. Since they are an essential ingredient of the component, all these internal headers start to include the component's `.h` file, followed by the declarations of the `private' functions. All these private functions start with abbreviated component names, like `co_` in the case of counters. Here is a possible implementation of the `counter.ih` internal header file:

      #include "counter.h"

      #include <stdio.h>

      #include "../stack/stack.h"
      #include "../message/message.h"
      #include "../new/new.h"

      Stack  *co_construct(int value);
      Stack  *co_sp(HashItem *item, bool errOnFailure);

- The combination of `.h` and `.ih` files define the dependencies of the component in the component hierarchy. As can be seen, `counter` depends on `stack, message, new, hashmap` and `root`. The actual dependency listing may be a bit more complex, as some `.h` files themselves depend on other `.h` files. This is clearly visible in the `counter.h` file. The class hierarchy given earlier shows the final component dependencies.
- A `.h` file of a component `X` never includes a `.ih` file of component `Y`, but only the `.h` files of other components.

## 6.4: The token-producer `lexer_lex()'

Tokens are produced by the lexical scanner. The function `lexer_lex()` produces the next token, which is always an element of the following set:

        TOKEN_UNKNOWN,          /* should never be returned */

        TOKEN_SYMBOL,
        TOKEN_TEXT,
        TOKEN_PLAINCHAR,        /* formerly: anychar */
        TOKEN_OPENPAR,
        TOKEN_CLOSEPAR,
        TOKEN_PLUS,             /* it's semantics what we do with a +, not      */
                                /* something for the lexer to worry about       */

        TOKEN_SPACE,            /* Blanks should be at the end                  */
        TOKEN_NEWLINE,

        TOKEN_EOR,              /* end of record: ends pushed strings           */
        TOKEN_EOF,              /* at the end of nested evaluations/eof         */

In particular note the existence of a `TOKEN_EOR` token: this token indicates the end of a piece of text, a string, inserted into the input stream by the parser's actions, when it calls `lexer_push_str()`. Such a situation occurs in particular when a macro is evaluated: having read a macro, and replacing its parameters `ARG1, ARG2, ... ARGn` by their respective argumentes, the resulting string is pushed back into the input stream by `lexer_push_str()`. This happens, e.g., inside the function `p_expand_macro()`. An excerpt from this function shows this call:

        void p_expand_macro(register Parser *pp, register HashItem *item)
        {
            ...
                if (argc)                           /* macro with arguments     */
                    p_macro_args(pp, &expansion, argc);
                ...
                lexer_push_str(&pp->d_lexer, string_str(&expansion));
            ...
        }

The parser repeatedly calls the lexer's function `lexer_lex()`. This happens most dramatically inside the function `p_parse()`, defined by a mere single statement:

        void p_parse(register Parser *pp)
        {
            while ((*pp->d_handler[lexer_lex(&pp->d_lexer)])(pp))
                ;
        }

Here, in a loop continuing until the handler indicates that the loop should terminate, `lexer_lex()` is called to produce the next token. The finite state automaton (FSA) implemented here is described in more detail in section 6.5.

Apart from here, `lexer_lex()` is called from four other locations inside the `parser` component:

- `parser_parlist()` repeatedly calls `lexer_lex()` to obtain all the tokens associated with a parameter list;
- `p_handle_default_newline()` repeatedly calls `lexer_lex()` to obtain all the tokens until all consecutive spaces and newlines are read. This is one of the handlers of the parser FSA 6.5;
- `p_no_user_macro()` calls `lexer_lex()` to determine whether a `no user macro' has been detected;
- `p_plus_series()` calls `lexer_lex()` to determine whether a `+symbol` has been encountered.

So, `lexer_lex()` is the parser's `window to the outside world'. The `lexer_lex()` function, however, is a fairly complex animal:

- `lexer_lex()`: returns next token. It calls `l_lex()` to retrieve the next character from the info waiting to be read;
- `l_lex()`: calls `l_nextchar()` to obtain the next token, and appends all char-tokens to the lexer's matched text buffer. Potential compound symbols (words, numbers) are combined by `l_compound()` and are then returned as `TOKEN_PLAINCHAR` or as a compound token like `TOKEN_IDENT`;
- `l_nextchar()`: calls `l_get()` to get the next character, and handles escape chars, including \ at eoln;
- `l_get()`: if there are no media left, `EOF` is returned. If there are media left, then `l_subst_get()` retrieves the next character, handling possible `SUBST` definitions. At the end of the current input buffer (memory buffer or file) `l_pop()` attempts to reactivate the previous buffer. If this succeeds, `EOR` is returned, otherwise `EOF` is returned. So, the lexer is not able to switch between truly nested media, as in `EVAL()` calls, but is able to switch between nested buffers resulting from replacing macro calls by their definitions;
- `l_subst_get()`: calls `l_media_get()` to get the next char from the media. The next char is passed to subst_find() which is a FSA trying to match the longest `SUBST`. This may be done repeatedly, and eventually `subst_text()` either returns a substitution text, or the next plain character. A substitution text is pushed onto the lexer's media buffer. The next character returned is then the next one to appear at the lexer's media buffer;
- `l_media_get()`: If the current active source of information is a file, it returns the next character from that file or `EOF` if no such char is available anymore. If the current active source is a memory buffer then the next char from the buffer is returned. If the buffer is empty `EOF` is returned. The media buffer is a circular, self-expanding Queue.

## 6.5: The Parser's Finite State Automaton

The input files are parsed by the function `parser_process()`, which is called by `Yodl`'s `main()` function.

This processor pushes all files that were specified on the input in reverse order on the input stack, and then calls the support function `p_parse()` to process each of them in turn.

`p_parse()` is an very short function: it contains one `while` statement, repeatedly calling a handler appropriate with the next token returned by the lexical scanner. Therefore, the parser can be considered as a table driven finite state automaton (FSA).

The table itself is initialized in `parser/psetuphandlerset.c`, by the function `p_setup_handlerSet()`. It fills the two dimensional array `ps_handlerSet` with the address of the function that must be called for each combination of parser-state (as defined in the `HANDLER_SET_ELEMENTS` enum) in `parser/parser.h` and token that may be produced by the lexical scanner (as defined in the `LEXER_TOKEN` enum in `lexer/lexer.h`). Depending on the situation the parser encounters, it may point its pointer `d_handler` to a particular row in this table. Since the rows represent the parser's states, states can be switched easily by reassigning this pointer. This happens all the time. For example, when in `parsernameparlist.c` a name must be retrieved from a parameter list, it calls `parser_parlist(pp, COLLECT_SET)`, which function temporarily switches the parser's state to `COLLECT_SET`, returning the parameter list's contents. to its caller.

The functions whose addresses are stored in the various column-elements of the array `ps_handlerSet` are called handler. Most handlers are named `p_handle_<state>_<lextoken>()`, where `<state>` is the name of the associated parser state, and `<lextoken>` is the name of the appropriate lexical scanner token. For example, `p_handle_default_symbol()` is the handler that was designed for the situation where the parser is in its initial, or default, state, and the lexical scanner returns a `TOKEN_SYMBOL` token. Some handlers have more generic names, like `p_handle_unknown()`, which is some sort of emergengy exit, called when the parser doesn't know what to do with the received lexical scanner token (a situation which should, of course, not happen).

In version 2.00, the following handler functions are available:

- `p_handle_insert(Parser *pp)`: insert matched text
- `p_handle_default_eof(Parser *pp)`: return false
- `p_handle_default_newline(Parser *pp)`: series of \n's
- `p_handle_default_plus(Parser *pp)`: handle + series
- `p_handle_default_symbol(Parser *pp)`: handle all symbols
- `p_handle_ignore(Parser *pp)`: ignores token
- `p_handle_ignore_closepar(Parser *pp)`: handle openpar
- `p_handle_ignore_openpar(Parser *pp)`: handle openpar
- `p_handle_noexpand_plus(Parser *pp)`: handle + series
- `p_handle_noexpand_symbol(Parser *pp)`: handle executed symbols in NOEXPAND
- `p_handle_parlist_closepar(Parser *pp)`: handle closepar
- `p_handle_parlist_openpar(Parser *pp)`: handle openpar
- `p_handle_skipws_unget(Parser *pp)`: unget received text
- `p_handle_unexpected_eof(Parser *pp)`: EMERG exit
- `p_handle_unknown(Parser *pp)`: emergency exit

The parser has the following states:

COLLECT_SET
retrieves parameter lists as they are encountered on the input. The parameter list is not processed in any way, and omits the surrounding parentheses. So, when entering this state (e.g., in the function `parser_parlist()`), a parameter list is completely consumed, but only its contents (and not its surrounding parentheses) become available. In fact, when entering a state, `p_parse()` can be called again to process the information in this state. Eventually a state will encounter some stopping signal (e.g., a non-nested close parenthesis in the collect-state results in `p_handle_parlist_closepar()` to return `false`, thus terminating `p_parse()`), terminating that particular state. The function `parser_parlist()` shows this process in further detail.
DEFAULT_SET
In this state macros, builtins etc. are processed. For most of the tokens that can be returned by the lexical scanner `p_handle_insert()` is called.

- When receiving EOF it tries to switch to the next file on the stack (or stop),
- When receiving a symbol, it either handles them as plain symbols or as macros,
- When receiving newlines they are handled (maybe merging them by calling a paragraph handler (if defined)),
- Series of + characters are handled
- All other tokens are inserted into the current output medium (which may be a file, but it may also be a memory buffer).

IGNORE_SET
In this state a parameter list is completely skipped. This state is used, for example, when processing `COMMENT()`.
NOEXPAND_SET
The contents of a parameter list is not expanded, but `CHAR` builtins are processed. In `Yodl` version 2.00 there is only one situation where this state (and its companion state NOTRANS_SET) is actively used: `Yodl`'s function `gram_NOEXPAND()` uses these states to retrieve the contents of a no-expanded or no-transed parameter list.
NOTRANS_SET
When the parser is in this state, a parameter list is inserted using the currently active insertion function (inserting to file or memory) It is identical to the NOEXPAND_SET state, but the character translation table is not used in the NOTRANS_STATE, whereas it is used in the NOEXPAND_STATE.
SKIPWS_SET
In this state all white-space characters are consumed. The lexical scanner merely returns the next non-whitespace character. This state is used, e.g., to skip the white space between multiple parameter lists when they are defined for macros.

## 6.6: Adding a new macro

With the advent of `Yodl` V 2.00, raw macros files are introduced. A raw macro file defines one macro, and all of its conversions. The raw macro files must be organized as follows:

        <STARTDOC>
        macro(name(arg1)(arg2)(etc))
        (

            Description of the macro `name', having arguments `arg1', `arg2',
            `etc', each argument is given its own parameter list. The names of the
            arguments in this description should be chosen in such a way that they
            suggest their function or purpose. All macro descriptions starting
            with tt(<STARTDOC>) are included in both the `man yodlmacros'
            manpage and the description of the macro in the user guide. If this is
            not considered appropriate (e.g., tt(XX...()) macros are not described
            in these documents) then use tt(<COMMENT>) rather than
            tt(<STARTDOC>).
        )
        <>
        DEFINEMACRO(name)(#)(
            statements of macro `name' expecting `#' arguments used by all
            conversions. This section is optional
        <html>
            statements that should be executed by the HTML converter
        <man ms>
            statements that should be executed by two converters. In this case,
            the `man' and `ms' converters
        <else>
            statements that should be executed by all converters not explicitly
            mentioned above
        <>
            statements of macro `name' expecting `#' arguments used by all
            conversions, having processed their specific statements.
            This section is also optional
        )

When setting up these macro definitions, the `<>` tags must appear with the initial documentation section. It must also appear when at least one specific converter tag is used. For a macro which is converter independent, the macro definition doesn't contain these pointed-arrow tags.

When writing standard `Yodl` macros, each macro should be stored in a file ``name'.raw`, where ``name'` is the lower-case name of the macro. This file should then be kept in the `macros/rawmacros` directory. The `macros/build std` call then adds the macro (filtering only the required statements per conversion) to each of the standard conversion formats.

If the macro requires a counter or symbol, consider defining the counter or symbol in, respectively, `@counters` and `@symbols`. Furthermore, consider pushing and popping these `variables', rather than plain assigning them, to allow other macros to use the variables as well. A case in point is the counter `XXone` which was added to the set of counters representing a local counter. Macros may always push `XXone` and pop `Xxone`, but should never reassign `XXone` before its value has been pushed. For `Yodl` version 2.00 only `XXone` was required, but other local counters might be considered useful in the future. In that case, `XXtwo`, `XXthree` etc. are used. For local symbold `XXs` prefixes are used: `XXsone`, `XXstwo`, etc.

## 6.7: The Yodl post-processor

With `Yodl` version 2.00 the old-style post-processor has ceased to exist. Also, the `.tt(Yodl)TAGSTART.` and `.tt(Yodl)TAGEND.` symbols no longer appear in `yodl`'s output.

Instead, a system using an index file was adopted. When converting information, `yodl` produces an output file and an associated index file. The index file defines offsets in the output file up to where certain actions are to be performed. Each line in the index file contains the required information of one directive for `yodlpost`. For example:

        0 set extension man
        53 ignorews
        2112 verb on
        2166 verb off
        80007 ignorews
        80065 copy
        80065 mandone

Entries can be written into the index file using the `INTERNALINDEX` builtin function. This function has one argument: the information following the offset where it is called. So, there will be a `INTERNALINDEX(set extension man)` in the macro definitions for this particular conversion (obviously it is a `man` conversion. The particular `INTERNALINDEX` call is found in the standard `man.yo` macro definition file).

When `yodlmacros` is called, it processes the directives on the `idx` file in two steps:

- First, it reads all directives, and constructs a queue of actions to perform. During this phase it solves all references to, e.g., labels defined in the s processed by `yodl`. This queue is constructed by a `PostQueue` object, during its construction phase.

  Postprocessing is realized by a template-method design pattern-like construction in C.

  The algorithm proceeds as follows:

  Each element of the index file is read, and its keyword (the word following the offset) is determined. Then the 'construct' function associated with that keyword is called. The `construct' functions return pointers to HashItem elements, which areprocessed by storing them either into the the symbol table or into the work-queue. The construct functions can use all `PostQueue, New, Message String Args` and `File` functions. Which function is actually called is determined in the file `yodlpost/data.c`, where the array `Task task[]` is initialized. `Task` structs have three elements:

  - `char const *d_key` points to the name of the keyword that triggers the corresponding `Task` struct;
  - `HashItem *(*d_constructor)(char const *key, char *rest)` points to the function that is called when the task struct is created.
  - `void (*d_handler)(long offset, HashItem *item)` points to the function that is called when the queue is processed.

- Then, when all commands are available, the queued commands are processed. For this, the appropriate 'handle' functions are called.

For example, when the `INTERNALINDEX(htmllabel ...)` is specified, the function `construct_label()` is called. This function receives a line line

        432 label Overview

meaning that this label has been defined in offset 432 in the file generated by `yodl`. The `construct_label()` function now:

- Stores the current section number, the filecount and the sectionnumber in a HashItem.
- Stores the hashitem inside its hash-table.

Then, when the queue is processed, a reference to this label may be encountered. This is signalled by an `INTERNALINDEX(ref Overview)` call. In this case the `construct_ref()` function doesn't have to do much. Here it is the handler that's doing all the work:

- First it looks up the label in the symbol table. The label should be there, as a result of the earlier construction of the symbol table during the `postqueue_construct()` call.
- Then it copies the file written by `yodl` up to the offset mentioned in the the `ref` command.
- Then (since we're talking about an html-specific reference) the appropriate `<a href=...` command is inserted into the current output file.

When references are solved in text-files, the `INTERNALINDEX(txtref ...)` command is used. Here, `construct_ref()` can still be used, but a specific `handle_txt_ref()` function is required.

New postprocessing labels can be constructed easily:

- Add an element to the array `Task task[]` in `src/yodlpost/data.c`. For example, add a line like:

          {"verb",            construct_verb,         handle_verb},

- Declare the functions in `yodlpost.h`:

          HashItem *construct_verb(char const *key, char *rest);
          void handle_verb(long offset, HashItem *item);

- The `construct_verb()` function receives the key (e.g., `verb`) and any information that may be available beyond the key as a trimmed line (not beginning or ending in white space). The construct function should return a pointer to a hashitem, which can be constructed by `hashitem_construct()`. This function should be called with the following arguments:
  - `VOIDPTR`;
  - a pointer to some text to be stored as the hashitem's key (use an empty string if nothing needs to be stored in a hashtable);
  - A pointer to the information associated with the key (use 0 if no information is used; use `(void *)intValue` to store an `int` value. Note that this is not `(void *)&intValue`: it is the value of the variable that is interpreted as a pointer here).
  - The function that handles the destruction of the value-information. Use `free` if some information was actually allocated and must be freed. E.g.,

          hashitem_construct(VOIDPTR, "", new_str(rest), free);

  Use `root_nop` if no allocation took place. E.g.,

          hashitem_construct(VOIDPTR, "", (void *)s_lastLabelNr, root_nop);

  Often the constructor doesn't have to do anything at all. In that case, initialize the `Task` element with the existing `construct_nop` function. E.g.,

          {"drainws",         construct_nop,          handle_drain_ws},

- The `handle_verb()` function is called when the file produced by `yodl` is processed by `postqueue_process()`. This happens immediately after `postqueue_construct()`. The handler is called with two arguments:
  - Its first argument is the offset where the `INTERNALINDEX` call was generated. The handler should make sure that `yodl`'s output file is processed up to this offset. Not any further. If a simple copy is required the function `file_copy2offset()` is available. E.g.,

            file_copy2offset(global.d_out, postqueue_istream(), offset);

    Note its arguments: the output and input file pointers are available through, respectively, `global.d_out` and `postqueue_istream()`.
  - Its second argument is a pointer to the hashitem struct originally created by the matching `construct...()` function. The handler should not free the information it receives. The function `postqueue_process()` takes care of that.
  Examples of actual `construct...()` and `handle...()` functions can be found in `src/yodlpost`.

- Table of Contents
- Previous Chapter