Jobman: A Practical Job Manager for Research Computing

Jobman keeps these tasks running after the terminal closes and adds dependencies, retries, timeouts, logs, concurrency limits, and notifications. It combines the benefits of nohup and terminal job control with the features of many heavier-weight schedulers.

A example research pipeline

Consider a project with four stages:

Each stage should start only after its inputs are ready. A failure should stop dependent work rather than produce results from stale files.

Submit the download and cleaning jobs first:

$ jobman run --name fetch -- python fetch.py
$ jobman run --name clean --after-success fetch -- Rscript clean.R

Submit the estimation job:

$ jobman run --name model --after-success clean  -- stata -b do model

Once modeling succeeds, the tables and figures can run independently:

$ jobman run --after-success model -- Rscript tables.R
$ jobman run --after-success model -- python figures.py

Jobman records the dependency graph when each job is submitted. You can close the terminal while the pipeline runs.

Check progress

Use list for an overview of active jobs:

$ jobman list --active

Potential output:

ID                                    NAME   PHASE    OUTCOME  SUBMITTED
019fdd39-751b-71c6-916f-06ade0f7a684         waiting           2026-08-07T17:15:58.107673Z
019fdd39-5a8c-71a8-923c-3b81528037f7         waiting           2026-08-07T17:15:51.308744Z
019fdd39-3be6-7e53-82bc-094f4a942544  model  waiting           2026-08-07T17:15:43.462936Z
019fdd39-08f6-745c-95a7-5234c6e4aa49  clean  waiting           2026-08-07T17:15:30.423118Z
019fdd38-e548-75bd-9a3e-560e6d0713fc  fetch  running           2026-08-07T17:15:21.288224Z

Use status for one job:

$ jobman status model

Potential output:

019fdd39-3be6-7e53-82bc-094f4a942544    model   waiting

Use show for the job specification and run history:

$ jobman show model

Potential output:

ID:                       019fdd38-e548-75bd-9a3e-560e6d0713fc
Name:                     fetch
Phase:                    running
Outcome:
Submitted:                2026-08-07T17:15:21.288224Z
Executable:               python
Working directory:        /tmp/jobman-demo
Completed runs:           0
Successful runs:          0
Failed runs:              0
Dependencies:             0
Wait evaluations:         0
Admission:                active, global, 1 slot(s)
Notification deliveries:  0
Pending notifications:    0
Notification attempts:    0

RUN  PHASE    OUTCOME  STARTED                      COMPLETED  LOGS
1    running           2026-08-07T17:15:21.368636Z             available

A summary of commands for inspecting jobs:

Persist and review logs

Jobman captures stdout and stderr independently.

Show both streams for a job:

$ jobman logs model

Follow a job as it runs:

$ jobman logs --follow model

Inspect stderr only:

$ jobman logs --stream stderr model

Read only the last 50 lines:

$ jobman logs --lines 50 model

For a job with several attempts, include every run:

$ jobman logs --all model

This is particularly useful for software that reports diagnostics like convergence warnings, dropped observations, or failed specifications on stderr.

Raw job output is recorded to disk and is not automatically redacted. Avoid printing credentials or confidential data to logs.

Retry transient failures

Downloads, APIs, database connections, and licensed software can fail temporarily. A configurable retry policy handles those failures without repeatedly running a job that cannot succeed.

$ jobman run --name download \
    --retries 3 --retryable-exit-code 1 \
    --retry-delay 10s --retry-backoff exponential \
    -- python download.py

--retries 3 permits four attempts in total: the initial run and three retries.

A script can use distinct exit codes to separate transient failures from invalid inputs. Only failures (i.e., exit codes) you classify as retryable trigger another attempt.

This allows you to customize the retry policy based on the type of failure encountered:

Limit execution time

Use a run timeout to stop one attempt:

$ jobman run --run-timeout 2h -- python bootstrap.py

Use a job timeout to time-bound the entire lifecycle, including waiting and retries:

$ jobman run --job-timeout 8h -- python bootstrap.py

The two limits can be combined:

$ jobman run --run-timeout 2h --job-timeout 6h \
    --retry-timeouts --retries 2 -- python bootstrap.py

For example, timeouts can catch stalled optimizers, infinite loops, inaccessible network resources, and simulations stuck on pathological parameters.

Limit concurrent work

Parallel jobs can exhaust memory or make every model slower. Jobman supports store-wide capacity and named pools.

For example, configure separate limits for downloads and models:

concurrency:
  max_active_slots: 8
  pools:
    downloads: 2
    models: 4

Then assign work to a pool:

$ jobman run --pool models -- python model_a.py
$ jobman run --pool models -- python model_b.py

A memory- or CPU-intensive job can request several slots:

$ jobman run --pool models --slots 2 -- stata -b do simulation_a

Waiting jobs do not consume slots before their dependencies and wait conditions are satisfied.

Pools are useful for:

• preventing runs from exhausting RAM;
• limiting simultaneous database queries;
• restricting calls to rate-limited services;
• reserving capacity for different workload classes.

Wait for conditions to start

The --wait-* flags instruct Jobman to wait for various conditions before starting the job.

A job can wait for a file to exist before starting:

$ jobman run --wait-file data/raw/complete.flag -- python clean.py

It can also wait until a specified time:

$ jobman run --wait-until 2026-08-01T02:00:00Z -- python import.py

Or start after a relative delay:

$ jobman run --wait-delay 30m -- python refresh.py

Add an abort time when an input becomes useless after a deadline:

$ jobman run --wait-file data/ready \
    --wait-abort-at 2026-08-01T12:00:00Z -- python estimate.py

Organize related jobs

Attach groups and tags when submitting work:

$ jobman run --group paper_a --tag baseline -- python model.py

Filter the job list by group:

$ jobman list --group paper_a

Possible research-oriented groups include:

• paper;
• policy;
• replication;
• simulation;
• data-refresh.

Tags can also record characteristics such as baseline, robustness, or clustered-se.

Job names are labels, not unique identifiers. Reusing a name does not overwrite an earlier job.

Control the working environment

Jobs inherit the submitting shell’s environment by default, but run can override it.

Set the working directory:

$ jobman run --cwd /work/project -- Rscript analysis.R

Set an environment value:

$ jobman run --env SPEC=baseline -- python model.py

Remove an inherited environment variable:

$ jobman run --unset-env DEBUG -- python model.py

Jobman executes the target directly. It does not interpret shell operators unless you explicitly run a shell:

$ jobman run -- sh -c 'python model.py > summary.txt'

Prefer direct execution unless shell syntax is necessary.

Repeat a specification

Rerun a prior job without reconstructing its options:

$ jobman rerun models --name models-rerun

The new job copies the earlier specification but has its own ID and history.

For repeated sampling or simulations, define explicit completion limits:

$ jobman run --max-runs 100 --success-target 100 -- python simulate_once.py

Jobman can also abort after a specified number of failed runs:

$ jobman run --max-runs 110 --success-target 100 \
    --failure-limit 11 -- python simulate_once.py

This is useful when each execution produces one independent result that can be aggregated later.

Control active work

Pause and resume are useful when interactive work temporarily needs the machine’s resources:

$ jobman pause models
$ jobman resume models

Cancellation applies to the managed process tree, not only the initial process. Jobman first requests a graceful stop and can force termination after the configured grace period.

$ jobman cancel models

Waiting on a job blocks until the job completes:

$ jobman wait models

Notifications

Jobman supports configured command callbacks, webhooks (HTTPS), and email (SMTP) notifications.

For example, if a notifier named research is configured, this invocation sends a message to that notification channel if the job fails:

$ jobman run --notify research --notify-on job_failed -- python models.py

Useful events that Jobman can notify about include:

• job succeeded;
• job failed;
• job timed out;
• retry scheduled;
• run started or completed.

Notifications are especially useful for jobs running outside work hours and for remote sessions.

Use stable output in scripts

Human-readable output is intended for terminals. Use JSON for automation:

$ jobman status --json models
$ jobman show --json models
$ jobman list --json --group paper

This makes it easier to generate run manifests or record job outcomes alongside research artifacts.

For reproducibility, retain:

• the canonical job ID (displayed by jobman run);
• the source revision (e.g., commit hash);
• the input-data version (e.g., checksum);
• the software environment (e.g., uv.lock);
• the command or named job specification;
• relevant Jobman JSON output.

Jobman records execution history, but it does not replace source control, data versioning, or environment management.

Keep state manageable

Preview history cleanup before deleting anything:

$ jobman clean --older-than 30d

Apply the cleanup with --force:

$ jobman clean --older-than 30d --force

The clean subcommand avoids removing active state or metadata still required by another job.

Check that the local Jobman metadata store is healthy:

$ jobman doctor

Create a metadata backup before an upgrade or major cleanup:

$ jobman doctor --backup jobman-backup.db

Do not delete files inside the Jobman state directory by hand.

Where Jobman fits

Jobman is designed for single-user work on one machine.

Note that Jobman jobs:

• will survive a closed terminal or SSH connection;
• may survive when the operating system user session ends, depending on host configuration;
• will not survive host shutdown or reboot.

For many research workflows, this is the useful middle ground: more reliable and feature-rich than unmanaged background processes, but substantially simpler than a heavy-weight scheduler.

Basic use demo