Inside Jobman, Part 3: Scheduling Without a Central Scheduler

Inside Jobman: Part 1: Architecture · Part 2: Ownership transfer · Part 3: Scheduling · Part 4: Process control and logs

Part 2 stopped after ownership had been committed to a detached supervisor. The target command had not necessarily been started. Dependencies could still be active, a retry delay could be pending, or the configured concurrency limit could already be full.

Consider the preparation and analysis pipeline from Part 1, with the input now divided into four shards. All four analysis jobs depend on the same preparation job and use a pool with two slots.

Excerpt from jobman.yml config file:

schema_version: 1
concurrency:
  max_active_slots: 4
  pools:
    analysis: 2

Jobman analysis commands:

$ for shard in 1 2 3 4; do
    jobman run --after-success prepare_data \
      --pool analysis --slots 1 \
      --retries 2 --retryable-exit-code 1 --retry-delay 1s \
      -- python analyze_data.py --shard "$shard"
  done

Four supervisors are left waiting. When preparation succeeds, all four may observe that result within the same polling interval, but only two runs are allowed into the analysis pool.

No central Jobman process chooses the winners. Each supervisor attempts its own state transition against SQLite, and the transactions settle the race. The queue, slot allocations, retry deadlines, and evidence used to recover stale owners are all stored there. If a supervisor disappears, the scheduling record is left behind.

Eligibility before admission

The word “queued” can obscure two separate situations. A job may be unable to run because a prerequisite is missing, or it may be ready but waiting for capacity. Those cases are kept separate in Jobman.

Before an admission is requested, dependencies and wait conditions are evaluated. Cancellation, pause state, and timeouts are checked as well. Only an eligible job is allowed to compete for slots.

A file wait or dependency can last for hours. Capacity is not reserved during that time. One lightweight supervisor process remains, but no execution slot is consumed.

Long-running work is also kept out of SQLite transactions. A transaction records a decision or an observation and closes. Sleeps, filesystem probes, and target processes are handled afterward.

Recording dependency outcomes

When --after-success prepare_data is submitted, the selector is resolved immediately to a canonical job ID. Missing references, contradictory predicates, and dependency cycles are rejected before submission completes.

The edge stores the required relationship:

While preparation is active, no final observation can be made. Each shard stays in waiting and checks again later.

Once preparation completes, its revision and outcome are copied onto the dependency edge in a transaction. That observation is written once. The edge now acts like a receipt: the exact terminal state that satisfied the prerequisite has been recorded, and the relationship will not be reinterpreted through a display name that may later be reused.

A successful preparation satisfies all four shard edges. A failed preparation makes after-success impossible. Each dependent job can then be completed as aborted with the dependency_unsatisfied diagnostic instead of being left in waiting forever.

No completion message has to be delivered from one job to every dependent. If a notification were missed during a crash, extra recovery machinery would be needed. Here, the completed dependency is already durable and can be observed independently by each supervisor.

Waiting on files and probes

Some prerequisites do not belong to another Jobman job. A run may be held until an absolute time, for a delay after acceptance, until a filesystem path exists, or until an executable probe succeeds.

Multiple wait conditions can be combined with all or any. Probe arguments remain separate rather than being joined into a shell command. Execution is bounded by a timeout and an output limit, and a short diagnostic is persisted after each evaluation. When jobman show --json is used on a job that appears stuck, those observations show what has actually been checked.

Dependencies and wait conditions are initial gates. After they have been satisfied, they are not reevaluated before every retry. A wait-abort deadline or whole-job timeout may still end the job before its first run.

Only after this stage is an admission requested.

Racing for the last slot

An eligible job requests a positive slot count. When a named pool is selected, those requested slots are charged against both the store-wide limit and the pool limit. Without a named pool, only the global limit is used.

Requests that can never fit are rejected during submission. A three-slot request cannot run in the two-slot analysis pool, so no useful result would come from placing it in the queue.

A request that fits in principle but not at the moment is different. Its durable queue position is created or reloaded during each acquisition attempt. In the same short transaction, active admissions are counted, the fairness rule is applied, and either a new admission is inserted or the request remains queued.

This is the point at which the four shard supervisors meet. Suppose two slots have already been taken and the other two supervisors concurrently read an apparently available final slot. Their reads may overlap, but their writes cannot both commit on the same old state. In this way, SQLite serializes the admission transactions. The winner records its allocation; the other attempt observes the new usage or loses the compare-and-swap and stays queued.

After admission, the job enters starting. The allocation is bound to the run once that run has been reserved. A failed attempt leaves the original queue position intact instead of sending the job to the back.

Fairness by limiting queue bypasses

Strict first-in, first-out ordering behaves poorly when jobs request different numbers of slots. Suppose a pool has a capacity of 4 with this state:

The three-slot request cannot start. Under strict FIFO, the one-slot request would also be blocked and two usable slots would sit idle. In contrast, if every smaller request were always allowed through, the large request could instead be starved by a steady stream of short jobs.

To ensure some notion of “fairness” in scheduling, Jobman uses a bounded bypass policy. When the oldest competing request does not fit, a younger request that does fit may be admitted. The older request’s bypass counter is incremented in the same transaction. Once three bypasses have been recorded, younger competitors must wait for it.

In pseudocode form:

older := oldestCompetingRequest(request)

if older.fitsNow() || older.bypassCount >= 3 {
    return ErrCapacity
}
if !request.fitsNow() {
    return ErrCapacity
}

persistAdmission(request)
incrementBypassCount(older)

Competition is limited to shared finite scopes: the global limit, the same finite named pool, or both. Requests in unrelated pools are not ordered against one another just because their supervisors happen to poll at the same time.

Initial order is based on the time prerequisites were satisfied, with the canonical job ID used to break ties. This is not weighted fair sharing, and no running job is preempted. The narrower goal is to allow some backfilling without allowing for potentially unlimited starvation of larger jobs.

Expired leases do not release capacity

Every admission has a lease that is renewed by its supervisor while the run is active. It would be tempting to treat the expiration timestamp as automatic capacity reclamation, but that could violate the limit.

A laptop may have resumed from sleep, or a live supervisor may have been delayed while SQLite was busy. Its target could still be using the resource even though the lease is stale. If those slots were immediately reassigned, both the old and new runs would be counted as active outside the database while only one allocation appeared inside it.

Expiration therefore starts reconciliation. The owning job, supervisor lease, and stored process identity are checked. Capacity is released after the owner is shown to be gone or the job has already completed. When identity cannot be verified safely, the allocation is left alone.

Release slots before retrying

After an individual run ends, its result is classified as success, retryable failure, or non-retryable failure. Cancellation, whole-job timeout, success and failure counts, the run limit, and any retry deadline are then evaluated in that order.

The admission is released before another attempt is delayed or started. If the retry policy permits another run, constant, linear, or exponential delay is calculated with the configured cap and jitter. An exact next_run_at timestamp is persisted for backoff, whereas if the policy indicates no delay, the job returns directly to queued.

No slot is held during backoff. Once the timestamp is reached, admission must be won again. A flaky shard therefore cannot occupy half of the analysis pool while it waits for its next attempt.

The whole-job timeout continues across dependency waits, admission waits, and retry delays. A run timeout covers only one target invocation and can itself produce a retryable result. Actual termination of the executable target belongs to the execution boundary described in Part 4; the scheduler only decides whether another run may follow.

Why supervisors poll

Without a resident scheduler, no shared condition variable is available when a dependency completes or a slot is returned. Instead, supervisors check durable state at regular intervals.

The ordinary interval is 100 milliseconds. Uniform jitter between 90 and 110 percent is applied so the four shard supervisors do not keep waking in lockstep after being submitted together. Longer intervals can be configured for wait conditions, and backoff polling is bounded by the recorded eligibility time.

Some database reads and up to one polling interval of scheduling latency are accepted here. In exchange, a missed in-memory wake-up cannot strand a job. After a restart, no queue has to be reconstructed from a scheduler process because the necessary parts of that queue were already stored.

This tradeoff decision is aimed at Jobman’s core use case: local jobs owned by one user on one machine. Thousands of remote workers or sub-millisecond dispatch would call for a different design.

Durable scheduler state

In summary, for the shard pipeline, the dependency revision says why each job became eligible. The prerequisites-satisfied timestamp preserves its initial place in line. Admission rows and bypass counts record who received capacity and who was allowed through. Leases identify allocations that deserve investigation. Retry timestamps keep policy from being reconstructed from process memory or log text.

Each supervisor still acts for only one job. Coordination occurs where a shared fact must be changed, through a short transaction against those durable records. No supervisor is given authority to schedule the others.

Once one shard has been admitted, another boundary has to be crossed. A target may spawn descendants, fill stdout and stderr pipes, ignore graceful termination, or exit while its final output is still being drained. Part 4: Process Trees, Timeouts, and Durable Logs follows that run from reservation through its recorded result.

Previous: Part 2: Transferring Job Ownership to a Detached Supervisor

Next: Part 4: Process Trees, Timeouts, and Durable Logs

View every Jobman article