Writing Lmakefile.py

Lmakefile.py contains 3 sections:

  • config, (some global information)
  • sources, (the list of sources)
  • rules, (the list of rules)

When reading Lmakefile.py, open-lmake:

  • imports Lmakefile
  • for each section (config, sources, rules):
    • if there is a callable with this name, call it
    • if there is a sub-module with this name, import it

The advantage of declaring a function or a sub-module for each section is that in case something is modified, only the impacted section is re-read.

The config

The config is determined by setting the variable lmake.config. Because it is predefined with all default values, it is simpler to only define fields. A typical Lmakefile.py will then contain lines such as:

lmake.config.path_max = 500 # default is 400

lib/lmake/config.py can be used as a handy helper as it contains all the fields with a short comment.

The sources

The sources are determined by setting the variable lmake.manifest.

Sources are files that are deemed as intrinsic. They cannot be derived using rules as explained in the following section.

Also, if a file cannot be derived and is not a source, it is deemed unbuildable, even if it actually exists. In this latter case, it will be considered dangling and this is an error condition. The purpose of this restriction is to ensure repeatability : all buildable files can be (possibly indirectly) derived from sources using rules.

lmake.manifest can contain :

  • Files located in the repo
  • Dirs (ending with /), in which case:
    • The whole subtree underneath the dir are considered sources.
    • They may be inside the repo or outside, but cannot contain or lie within system dirs such as /usr, /proc, /etc, etc.
    • If outside, they can be relative or absolute.

In both cases, names must be canonical, i.e. contain no empty component nor ., nor .. except initially for relative names outside repo.

The helper functions defined in lib/lmake/sources.py can be used and if nothing is said, auto_sources() is called.

The rules

Rules are described as python class'es inheriting from lmake.Rule, lmake.AntiRule or lmake.SourceRule.

Such classes are either defined directly in Lmakefile.py or you can define a callable or a sub-module called rules that does the same thing when called/imported. For example you can define :

def rules() :
	class MyRule(lmake.Rule) :
		target = 'my_target'
		cmd    = ''

Or the sub-module Lmakefile.rules containing such class definitions.

Inheriting from lmake.Rule is used to define production rules that allows deriving targets from deps.

Inheriting from lmake.AntiRule is (rarely) used to define rules that specify that matching targets cannot be built. Anti-rules only require the targets attribute (or those that translate into it, target) and may usefully have a prio attribute. Other ones are useless and ignored.

Inheriting from lmake.SourceRule may be used to define sources by patterns rather than as a list of files controlled by some sort of source-control (typically git).

Special rules

In addition to user rules defined as described hereinafter, there are a few special rules:

  • Uphill: Any file depends on its dir in a special way : if its dir is buildable, then the file is not. This is logical : if file foo is buildable (i.e. produced as a regular file or a symbolic link), there is not way file foo/bar can be built. If foo is produced as a regular file, this is the end of the story. If it is produced as a symbolic link (say with foo_real as target), the dependent job will be rerun and it will then depend on foo and foo_real/bar when it opens foo/bar. Note that if the dir applies as the star-target of a rule, then the corresponding job must be run to determine if said dir is, indeed, produced.
  • Infinite: If walking the deps leads to infinite recursion, when the depth reaches lmake.config.max_dep_depth, this special rule is triggered which generates an error. Also, if a file whose name is longer that lmake.config.path_max considered, it is deemed to be generated by this rule and it is in error. This typically happens if you have a rule that, for example builds {File} from {File}.x. If you try to build foo, open-lmake will try to build foo.x, which needs foo.x.x, which needs foo.x.x.x etc.

Dynamic values

Most attributes can either be data of the described type or a function taking no argument returning the desired value. This allows the value to be dynamically selected depending on the job.

Such functions are evaluated in an environment in which the stems (as well as the stems variable which is a dict containing the stems and the targets (as well as the targets variable) are defined and usable to derive the return value. Also, depending on the attribute, the deps (as well as the deps variable) and the resources (as well as the resources variable) may also be defined. Whether or not these are available depend on when a given attribute is needed. For example, when defining the deps, the deps are obviously not available.

For composite values (dictionaries or sequences), the entire value may be a function or each value can individually be a function (but not the keys). For dictionaries, if the value function returns None, there will be no corresponding entry in the resulting dictionary.

Note that regarding resources available in the function environment, the values are the ones instantiated by the backend.

Inheritance

python's native inheritance mechanism is not ideal to describe a rule as one would like to prepare a base class such as:

  • provide environment variables
  • provide some default actions for some files with given pattern
  • provide some automatic deps
  • ...

As these are described with dict, you would like to inherit dict entries from the base class and not only the dict as a whole. A possibility would have been to use the __prepare__ method of a meta-class to pre-define inherited values of such attributes, but that would defeat the practical possibility to use multiple inheritance by suppressing the diamond rule.

The chosen method has been designed to walk through the MRO at class creation time and:

  • Define a set of attributes to be handled through combination. This set is defined by the attribute combine, itself being handled by combination.
  • Combined attribute are handled by updating/appending rather than replacing when walking through MRO in reverse order.
  • Entries with a value None are suppressed as update never suppress a given entry. Similarly, values inserted in a set prefixed with a '-' remove the corresponding value from the set.

Because this mechanism walks through the MRO, the diamond rule is enforced.

dict's and list's are ordered so that the most specific information appear first, as if classes are searched in MRO.

Combined attributes may only be dict, set and list:

  • dict's and set's are updated, list's are appended.
  • dict's and list's are ordered in MRO, base classes being after derived classes.

paths

Some environment variables contain paths, such as $PATH.

When such an entry appears in a rule, its value is searched for occurrences of the special marker ... surrounded by separators (the start and end of the strings are deemed to be separators) And each such occurrence is replaced by the inherited value.

This makes it particularly useful to manage paths as it allows any intermediate base class to add its own entries, before or after the original ones.

For example, to add the dir /mypath after the inherited path, one would define the attribute environ as {'PATH':'...:/mypath'}. To add it before, one would use {'PATH':'/mypath:...'}.

Entries going through this step are provided by the attribute paths, which is a dict with . as keys and as values. The default value is { 'environ.PATH':':' , 'environ.LD_LIBRARY_PATH':':' , 'environ.MANPATH':':' , 'environ.PYTHONPATH':':' }