Skip to content

ADR 004: Pluggable Queue Source for Dispatch

Status

Accepted

Context

The Dispatcher consumes tasks from a Redis list (BRPOP). This limits horizontal scaling — only one consumer can dequeue from a list at a time. Multi-Dispatcher deployments need fan-out.

Decision

Implement a pluggable queue.Source interface. Three implementations: InternalSource (Redis list, default), RedisStreamSource (Redis Streams with consumer groups), and SQS (stub). Controller gains dispatch_mode (http/bus) on DispatcherInstance to choose between HTTP POST and direct queue publishing.

Consequences

  • QUEUE_SOURCE=internal (default) preserves existing single-instance behavior
  • QUEUE_SOURCE=redis-stream enables multi-Dispatcher fan-out via XREADGROUP consumer groups
  • Controller's dispatch_mode=bus publishes directly to Redis Stream or SQS, bypassing the Dispatcher's HTTP API for task submission
  • Adding a new queue source requires implementing the Source.Dequeue() method
  • Redis Streams provide at-least-once delivery with acknowledgment (XACK after processing)