Queue Management

Control concurrent execution and resource usage with queues.

Overview

Dagu's queue system helps you:

• Limit concurrent workflow executions
• Manage resource usage
• Prioritize critical workflows
• Prevent system overload

Basic Queue Configuration

Assign to Queue

name: batch-processor
queue: "batch"              # Assign to batch queue
maxActiveRuns: 2            # Allow 2 concurrent runs
schedule: "*/10 * * * *"    # Every 10 minutes

steps:
  - name: process
    command: ./batch-process.sh

Disable Queueing

For critical workflows that should always run:

name: critical-alert
maxActiveRuns: -1           # Never queue - always run

steps:
  - name: check
    command: ./check-alerts.sh

Global Queue Configuration

Configure queues in server config:

# ~/.config/dagu/config.yaml
queues:
  enabled: true
  config:
    - name: "critical"
      maxConcurrency: 5     # 5 critical jobs concurrently
    - name: "batch"
      maxConcurrency: 1     # One batch job at a time
    - name: "reporting"
      maxConcurrency: 3     # 3 reports concurrently

Default Queue via Base Config

Set default queue for all workflows:

# ~/.config/dagu/base.yaml
queue: "default"
maxActiveRuns: 2

# All DAGs inherit these settings

Manual Queue Management

Enqueue Workflows

# Basic enqueue
dagu enqueue workflow.yaml

# With custom run ID
dagu enqueue workflow.yaml --run-id=batch-2024-01-15

# With parameters
dagu enqueue process.yaml -- DATE=2024-01-15 TYPE=daily

Remove from Queue

# Remove by DAG name
dagu dequeue workflow.yaml

# Remove specific run
dagu dequeue --dag-run=workflow:batch-2024-01-15

Queue Behavior

FIFO Processing

Queues process in First-In-First-Out order:

# First workflow submitted
name: report-1
queue: "reporting"
maxActiveRuns: 1

# Second workflow submitted
name: report-2
queue: "reporting"
maxActiveRuns: 1

# report-1 runs first, report-2 waits

Priority Handling

Use different queues for priority:

# High priority
name: critical-job
queue: "high-priority"
maxActiveRuns: 5

# Low priority
name: maintenance
queue: "low-priority"
maxActiveRuns: 1

See Also

• Execution Control - Control workflow execution
• Scheduling - Schedule workflows
• Configuration Reference - Complete configuration guide