lmake - a reliable & scalable tool to execute a full flow
lmake [OPTION]... [FILE]...
The lmake utility determines automatically which pieces of a flow need to be executed and launch them. This is based in the historical make program initially written by Stuart Feldman in 1976 with notable differences :
|
• |
lmake is reliable. It ensures that necessary jobs are run in adequate order so that the asked files are in the same state as if all derived files were removed and all jobs were run. In particular, this includes automatic discovery of actual deps. | ||
|
• |
lmake is scalable. Millions of files can be derived easily (job execution may require heavy resources, but lmake will stay light). | ||
|
• |
The makefile is called Lmakefile.py and is written in python3. | ||
|
• |
Rule matching is based of regular expressions, not just a single wildcard (% with make). | ||
|
• |
Recipies can be written in bash(1) or python(1). | ||
|
• |
Several lmake commands can be run simultaneously in the same dir. Jobs needed by several of them will be shared. Simlarly sources files can be edited without risks while lmake is running. Derived files will be generated according to the state of the source files at the time the command was launched. If this is not possible because an old version of a source file need to be accessed after it has been modified, an error will be generated. Rerunning the same lmake command will take into account the new content of such source files. | ||
|
• |
File states are based on content (using a checksum) rather than dates. This makes it possible to avoid running a dependent job if a file has been reconstructed identically to its previous content. |
To reach these goals, lmake maintains a state in the LMAKE dir, instrument jobs (to catch all file accesses) during their execution and launches a daemon (called lmakeserver) that can be shared between several concurrent invocations of lmake.
lmake generates an output line for each significant event :
|
• |
When a job starts if it is long (duration that qualifies a job as long is configurable). This start line is deemed of secondary importance. | ||
|
• |
When a job terminates. Such lines are followed by the content of stderr if any and status is known. Typically the job status is not known at the end of execution when a new dep was discovered that turned out to be out-of-date. In that case, the dep is rebuilt, and if the new content is identical to the old one, lmake can decide that the job was run with up-to-date deps. | ||
|
• |
When the job status is finally known, if it was not known when it terminated. Such lines are followed by the content of stderr if any. |
During execution, if launched from a terminal, lmake also generates a progression status in the title bar. This progression status line contains :
|
• |
The number of executed jobs (split between useful, rerun and hit) | ||
|
• |
The number of queued jobs, i.e. jobs that are waiting for resources to be run | ||
|
• |
The number of waiting jobs, i.e. jbos that are waiting for deps to be run |
At the end of the execution, if the asked targets are not successfully generated, a summary is generated reminding the errors (with a configurable max, 20 by default) and the stderr of the first error. Intermediate targets are deemed of secondary importance.
Before processing arguments, lmake prepends the content of $LMAKE_ARGS, separated by spaces. This eases the management of user preferences. For example, a user may like to systematically pass the --archive and --keep-tmp options, in which case they can set LMAKE_ARGS=-a -t.
lmake manages a fully coherent dir called a repo. When it starts, it first determines the root of the repo (cf. FILES below).
Arguments and reports are systematically localized to the current working dir. For example, if you launch lmake b from dir a in your repo, the argument is file a/b from the root of the repo and reports containing file names (initially seen from the root of the repo) will be shown relative the the current working dir.
If launched from a terminal, output is colored. Colors are different depending on whether terminal is normal (black on white) or reverse (white on black) video. These colors can be configured. The colors bears a semantic:
|
• |
Green means success. | ||
|
• |
Orange means possible error, depending on future (if error is confirmed, it will be repeated in red). | ||
|
• |
Red means error. | ||
|
• |
Magenta means warning. | ||
|
• |
Blue means notes. | ||
|
• |
Gray means information of secondary importance. | ||
|
• |
Uncolored means general output. In some occasion, it may be colored by user code (e.g. gcc generates colored error messages). |
If $LMAKE_VIDEO is defined, it is processed as if provided to the --video option.
These options
are common to all tools of the open-lmake set of
utilities :
--version
Print version and exit. Version is in the form "year.month.tag (key)" where "year.month" forms the major version, "tag" the minor version and "key" is indicates the format of the persistent information.
-h, --help
Print a short help and exit. It is composed of:
|
• |
The command line synoptic. | ||
|
• |
Version as described above. | ||
|
• |
A line for each supported option with its short name, long name, whether it has an argument and a short explanation. |
-J, --job
Passed arguments are interpreted as job names rather than as file names. Job names are the names that appear, for example, on start and done lines when lmake executes a job.
-R rule, --rule=rule
When the --job option is used, this options allows the specification of a rule, given by its name. This is necessary when the job name is ambiguous as several rules may lead to the same job name.
-q, --quiet
Do not generate user oriented messages. Strictly generate what is asked. This is practical if output is meant for automatic processing.
-S, --sync
Ensure server is launched (i.e. do not connect to an existing server) and wait for its termination. This is exceptionally useful in scripts that modify Lmakefile.py.
-v, --verbose
Generate more prolix output.
-V mode, --video=mode
Explicitly ask for a video mode instead of interrogating connected terminal. If mode starts with n or N, normal video (black on white) is assumed. If it starts with r or R, reverse video (white on black) is assumed. Else output is not colorized. video mode has an impact on generated colors as nice looking colors are not the same in each case.
These options
are specific to lmake :
-a, --archive
Ensure all intermediate files are up to date, in addition to the asked targets. This is useful for example if you want to archive a fully built repo.
-b value, --backend=value
Pass value to backend (cf. backend documentation for each backend). This is used for example to pass a partition or specificities to the slurm backend for a particular command. Note that backend only impacts resources and scheduling, not the content of the targets, so specifying such an option does not hurt repeatability.
-e, --forget-old-errors
Assume encountered errors (before this command) are transicent. Contrarily to the lforget -e command, this only concerns this execution, not subsequent ones.
-j jobs, --jobs=jobs
When this option is used, lmake will limit the overall number of simultaneous jobs to jobs per backend. If several lmake commands run simultaneously, a job cannot be launched on behalf of a given command if the number of running jobs is not less than its associated jobs.
-o, --live-out
Normally, lmake does not output the stdout of jobs (such stdout is accessible with the lshow -o command). However, sometimes it is practical to have the output while jobs are running. Generating such output for all jobs would produce an intermixed flow of characters of all jobs running in parallel making such an output unreadable. When this option is used, only the jobs directly producing the asked targets have their output generated on the output of lmake. Because most of the time there is a single target, this ensures that there is a single job generating its output, avoiding the intermixing problem.
-m count, --max-submits=count
Ask lmake to limit number of submits for any job to this number. This constraint must be enforced together with the max_submits rule attribute, i.e. the min of these 2 constraints is used. This is useful to observe a job while it is supposed to rerun.
-r count, --retry-on-error=count
Ask lmake to retry jobs in case of error. This is useful for unattended execution (e.g. nightly regressions) when system reliability is not enough to guarantee correct execution at the desired level.
-l, --local
With this option, jobs are launched locally (i.e. using the local backend) instead of the backend mentioned in the rule. Note that if 2 lmake commands with different values for this option are running simultaneously, in case a job is necessary for both, it may be launched locally or remotely. The originally targetted backend is in charge of mapping required resources mentioned in the rule to local resources understandable by the local backend.
-s, --source-ok
Normally, lmake refuses to launch a job that may overwrite a source. With this option, the user instructs lmake that this is allowed.
-t, --keep-tmp
Normally, lmake washes the temporary dir allocated to a job at the end of job execution. With this option, the user instructs lmake to keep the temporary dirs, as if the keep_tmp attribute was set for all rules. The kept temporary dir can be retreived with lshow -i.
-v, --verbose
Enable the generation of some execution information from backend. This is not done systematicly as this may incur a performance hit. These information are available by using lshow -i.
lmake exits with a status of zero if the asked targets could be built or were already up-to-date. Else it exits with a non-zero status.
While lmake runs, it outputs a log. This log is also recorded in LMAKE/outputs/<start date> with the following differences:
|
• |
It is not colored. | ||
|
• |
Reported files are relative to the root of the repo, not to the current working dir where the lmake command has been launched. |
The log contains a line, possibly followed by attached information when the following events occur :
|
• |
A job is started, if the job duration is longer than the start_delay attribute of the rule. | ||
|
• |
A job is completed. | ||
|
• |
A job status is known, while it was not known when it completed. | ||
|
• |
A source file has been seen as modified. | ||
|
• |
A frozen file or a target of a frozen job is needed. |
Once the build process is complete, a summary is generated with :
|
• |
The frozen files and jobs that we necessary to carry out the build. | ||
|
• |
The jobs in error (the first of them is accompanied with its stderr). |
The content of $LMAKE_ARGS is prepended to command line arguments.
The content of $LMAKE_VIDEO is processed as if provided with the --video option.
Unless explicitly asked in Lmakefile.py, the environment is mostly ignored when lmake is run, i.e. it is not passed to the jobs. The goal is to improve repeatability by protecting jobs from the variability environment variables may cause. In particular :
|
• |
$HOME is redirected to the root of the repo. This protects the job from all specificities stored in .xxxrc files in the home dir. | ||
|
• |
$LMAKE_ARGS, although used by lmake, is not passed to jobs. | ||
|
• |
$PATH is reset to the default path for the system, plus the open-lmake bin dir. | ||
|
• |
$PYTHONPATH is set to the open-lmake lib dir. | ||
|
• |
Unless set to empty, $TMPDIR is redirected to an isolated, empty dir which is cleaned up at the end of each job execution. This way, the job can freely use this dir and need not take care of clean-up. |
Moreover, a few variables are set during job execution :
|
• |
$JOB_ID is set to an integer specific of a job. It does not change between executions, but may be different in different repo, even if strictly identical. | ||
|
• |
$SMALL_ID is set to a as small as possible integer such that a different value is set for jobs running concurrently. | ||
|
• |
$SEQUENCE_ID is set to a different value each time a job is run, they are never recycled. |
The files Lmakefile.py or Lmakefile/__init__.py are searched in the current dir and in parent dirs. If a single one is found, this determines the root of the repo. If several are found, the existence of an LMAKE dir is checked. If a single one is found, this determines the root of the repo. In other cases, lmake will not start.
lautodep(1), lcheck_deps(1), ldebug(1), ldecode(1), ldepend(1), lencode(1), lforget(1), lmark(1), lrepair(1), lrun_cc(1), lshow(1), ltarget(1), xxhsum(1)
The python module lmake.
The full open-lmake documentation in <open-lmake-installation-dir>/docs/index.html.
Copyright © 2023-2025, Doliam. This file is part of open-lmake.
open-lmake is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 of the License.
open-lmake is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.