dispenserd — the HTTP-poll queue

Every long-lived web platform ends up with a queue. Currnt’s queue is unusual: it’s neither Kafka nor RabbitMQ nor Redis pub/sub nor SQS. It’s a homegrown HTTP-poll system built in-house, named dispenserd. Producers POST jobs to its HTTP endpoint; workers poll the same endpoint to pull work. Acknowledgments and idempotency live in MySQL rows.

It’s not the queue you’d design today. It’s also the queue Currnt has had for nine years, and it has been more reliable than any of its replacements would’ve been at the same headcount.

The architecture

                  ┌──────────────────────────────────────────┐
                  │              dispenserd                  │
   producers ────▶│  HTTP POST /enqueue                       │
   (Sails app)    │   ├── 5 lanes (priority classes)         │
                  │   ├── in-memory queue per lane           │
                  │   └── MySQL ack rows for at-least-once   │
                  └────────────────┬─────────────────────────┘
                                   │
                          (workers poll over HTTP)
                                   │
       ┌───────────────────────────┼───────────────────────────┐
       ▼                           ▼                           ▼
   worker A                    worker B                    worker N
   Supervisord                 Supervisord                 Supervisord
   ↓                           ↓                           ↓
   4,591-line entry            4,591-line entry            4,591-line entry
   ↓ dispatches by job type    ↓                           ↓
   19 handler scripts          (same set)                  (same set)

The five lanes

Each lane corresponds to a class of work with its own latency tolerance and resource profile. Producers specify the lane on enqueue.

The exact mapping of workers to lanes lives in Supervisord config; some workers are dedicated to a single lane, some round-robin across lanes.

The 19 handlers

Each enqueued job carries a type field. The worker entry script dispatches to one of 19 handler functions based on this type. A condensed list, grouped by purpose:

• Email-shaped: send_email, send_outreach, send_digest, send_invoice, notify_push
• Data-shaped: import_prospects, dedup_prospects, enrich_prospect, rebuild_search_index
• AI-shaped: ai_summarize_board, ai_suggest_post, gpt_ai_cofac_nudging
• Lifecycle: publish_post, close_board, archive_board
• Maintenance: purge_sessions, vacuum_logs, metric_rollup, health_check

A nineteenth handler is a generic noop used for testing.

The handler dispatch lives at the top of a 4,591-line worker entry file — by far the largest single file in the codebase. Reading it is roughly the way to understand the runtime: the queue is small, the entry is big, the handlers are inline.

Retry semantics

• Idempotency. Each enqueue produces a row in dispenserd_jobs with a unique job_id. A worker that re-pulls the same job no-ops if the row is already done.
• Retry on failure. If a handler throws, the row goes to failed and a retry is requeued with a back-off (exponential, capped at ~1 hour).
• Max retries. Five attempts; after that the job is marked dead and surfaces in the dispenserd dashboard inside Treehouse.
• No DLQ as a separate queue. Dead jobs sit in MySQL; staff inspects them in Treehouse.

Operational quirks

• No fan-out lane discovery. Workers are configured to know their lanes; adding a lane requires editing Supervisord config and restarting workers. There’s no dynamic registration.
• In-memory queue + MySQL ack. A worker pulling a job takes it out of the in-memory list and writes a pulled row. If the dispenserd process restarts, in-flight jobs that hadn’t been ack’d come back through the recovery pass.
• No priority within a lane. Jobs in the same lane are FIFO. Priority is achieved by lane choice.
• The big file. The 4,591-line entry has accumulated nine years of inline handler bodies. Several handlers reference deleted features (Calendly, Zoom, Lusha) with guards. Removing dead handlers was on the someday list.

Why this page matters for the rest

The queue is the spine that connects everything: when Domain B / Treehouse sends an outreach blast, when Domain F sends an invoice, when Domain D publishes a post, when the AI co-facilitator runs — they all go through here.