Queue API: Jobs, Priorities, Delays, Deduplication
The Queue API, every job where it belongs.
The Queue class is used to add and manage jobs: priorities, delays, retries, bulk operations, deduplication, durable writes, and DLQ configuration, all from one API.
Creating a Queue
Section titled “Creating a Queue”import { Queue } from 'bunqueue/client';
// Basic queue - embedded modeconst queue = new Queue('my-queue', { embedded: true });
// Typed queueinterface TaskData { userId: number; action: string;}const typedQueue = new Queue<TaskData>('tasks', { embedded: true });
// With default job optionsconst queue = new Queue('emails', { embedded: true, defaultJobOptions: { attempts: 3, backoff: 1000, removeOnComplete: true, }});TCP Mode (Server)
Section titled “TCP Mode (Server)”// Connect to bunqueue server (no embedded option)const queue = new Queue('tasks');
// With custom connectionconst queue = new Queue('tasks', { connection: { host: '192.168.1.100', port: 6789, token: 'secret-token', poolSize: 4, // Connection pool size }});Namespace Isolation (prefixKey)
Section titled “Namespace Isolation (prefixKey)”prefixKey lets multiple environments, tenants, or services share the same broker without their jobs, workers, cron schedulers, stats, pause state, DLQ, or rate limits overlapping. The prefix is prepended to the queue name on the server side; Queue.name keeps reporting the logical name.
// Same broker, totally isolated namespacesconst devQueue = new Queue('emails', { prefixKey: 'dev:' });const prodQueue = new Queue('emails', { prefixKey: 'prod:' });
await devQueue.add('send', { to: 'tester@example.com' });await prodQueue.add('send', { to: 'user@example.com' });
await devQueue.getJobCountsAsync(); // { waiting: 1, ... }await prodQueue.getJobCountsAsync(); // { waiting: 1, ... }, never sees dev jobsA Worker must use the same prefixKey to consume jobs from the prefixed queue:
const devWorker = new Worker('emails', processor, { prefixKey: 'dev:', connection: { port: 6789 },});// devWorker only processes jobs added via devQueue, never prodQueue.What gets isolated:
- Jobs (push / pull / ack / fail / DLQ)
- Worker registration and lock-based ownership
- Counts, stats, and
isPaused pause()/resume()/drain()/obliterate()- Rate limits and global concurrency
- Cron schedulers, two prefixes can use the same
schedulerIdwithout colliding on the global cron PRIMARY KEY (this fixes #77)
Use cases:
- Multi-environment,
dev:/staging:/prod:on a single broker, no port juggling - Multi-tenant SaaS,
tenant-${tenantId}:per customer, queue names can be reused - Monorepo, each service prefixes its queues so two
processqueues fromservice-aandservice-bdon’t collide - Test isolation,
test-${runId}:to avoidobliterate()between parallel test runs
Notes:
- Backward compatible, without
prefixKeybehavior is identical to previous releases. Queue.namealways returns the logical name. The server-side key (prefixKey + name) is internal.Job.queueNamereturned to processors will reflect the prefixed key (e.g.dev:emails). This is the only user-visible side effect.- Works in both embedded and TCP modes.
Adding Jobs
Section titled “Adding Jobs”Single Job
Section titled “Single Job”const job = await queue.add('job-name', { key: 'value' });
// With optionsconst job = await queue.add('job-name', data, { priority: 10, // Higher = processed first delay: 5000, // Delay in ms before processing attempts: 5, // Max retry attempts (default: 3) backoff: 2000, // Backoff in ms (default: 1000ms, jitter applied) // OR object form: backoff: { type: 'exponential', delay: 2000 } // 'fixed' | 'exponential' timeout: 30000, // Job timeout in ms jobId: 'custom-id', // Custom job ID for deduplication (BullMQ-style) removeOnComplete: true, // Remove job data after completion removeOnFail: false, // Keep failed jobs stallTimeout: 60000, // Per-job stall timeout (overrides queue config)});Bulk Add
Section titled “Bulk Add”// Batch optimized - single lock, batch INSERTconst jobs = await queue.addBulk([ { name: 'task-1', data: { id: 1 } }, { name: 'task-2', data: { id: 2 }, opts: { priority: 10 } }, { name: 'task-3', data: { id: 3 }, opts: { delay: 5000 } },]);Repeatable Jobs
Section titled “Repeatable Jobs”// Repeat every 5 secondsawait queue.add('heartbeat', {}, { repeat: { every: 5000, }});
// Repeat with limitawait queue.add('daily-report', {}, { repeat: { every: 86400000, // 24 hours limit: 30, // Max 30 repetitions }});
// Cron pattern (works in both embedded and TCP modes)await queue.add('weekly', {}, { repeat: { pattern: '0 9 * * MON', // Every Monday at 9am }});Updating Repeatable Job Data
Section titled “Updating Repeatable Job Data”You can update the data for the next repeat execution using updateData(). This works even after the current execution completes, the update propagates to the successor job automatically.
const job = await queue.add('sync', { endpoint: '/api/v1' }, { repeat: { every: 60000 },});
// Update data for the next executionawait job.updateData({ endpoint: '/api/v2' });// Next repeat will use { endpoint: '/api/v2' }Durable Jobs
Section titled “Durable Jobs”By default, bunqueue uses a write buffer for high throughput: jobs are batched in memory and flushed to SQLite every 10ms. This achieves ~100k jobs/sec but means jobs could be lost if the process crashes before the buffer is flushed.
For critical jobs where data loss is unacceptable, use the durable option:
// Critical job: immediate disk write, guaranteed persistenceawait queue.add('process-payment', { orderId: '123', amount: 99.99 }, { durable: true,});
// Batch of critical jobsawait queue.addBulk([ { name: 'payment-1', data: { orderId: '1' }, opts: { durable: true } }, { name: 'payment-2', data: { orderId: '2' }, opts: { durable: true } },]);Job Deduplication (BullMQ-style)
Section titled “Job Deduplication (BullMQ-style)”Use jobId to prevent duplicate jobs. When a job with the same jobId is still queued (waiting / delayed / prioritized), the existing job is returned instead of creating a duplicate. This works in both embedded and TCP modes (including auto-batched operations). This is BullMQ-compatible idempotent behavior.
// Basic deduplication with jobId (BullMQ-style idempotency)// If job with same jobId exists, returns existing job instead of creating duplicateconst job = await queue.add('send-email', { to: 'user@test.com' }, { jobId: 'email-user-123'});
// First call: creates the jobconst job1 = await queue.add('process', { orderId: 123 }, { jobId: 'order-123'});
// Second call with same jobId: returns existing job (no duplicate)const job2 = await queue.add('process', { orderId: 123 }, { jobId: 'order-123'});
console.log(job1.id === job2.id); // true - same job returned// Example: Restore jobs on service startupasync function restoreJobs(jobsToRestore: SavedJob[]) { for (const saved of jobsToRestore) { // Safe: existing jobs are returned, not duplicated await queue.add('process', saved.data, { jobId: saved.id }); }}Advanced Deduplication
Section titled “Advanced Deduplication”For more control over deduplication behavior, use the deduplication option with TTL-based unique keys and strategies.
TTL-Based Deduplication
Section titled “TTL-Based Deduplication”While jobId provides permanent idempotency (via customId), the deduplication option uses a separate uniqueKey mechanism with TTL-based expiry. The id field is required:
// TTL-based deduplication - unique key expires after 1 hourawait queue.add('notification', { userId: '123' }, { deduplication: { id: 'notify-123', // Required: unique deduplication key ttl: 3600000 // 1 hour in ms }});
// After TTL expires, the same id can create a new job// This is useful for rate-limiting or time-windowed deduplicationExtend Strategy
Section titled “Extend Strategy”The extend strategy resets the TTL of an existing job when a duplicate is detected. The existing job is returned (not replaced), but its deduplication window is extended:
// Extend strategy - reset TTL if duplicate, return existing jobawait queue.add('rate-limited-task', { action: 'sync' }, { deduplication: { id: 'sync-task', // Required: unique deduplication key ttl: 60000, extend: true // Extend TTL on duplicate }});Replace Strategy
Section titled “Replace Strategy”The replace strategy removes the existing job and inserts a new one with the updated data. This is useful when you always want the latest data to be processed:
// Replace strategy - remove old job, insert new oneawait queue.add('latest-data', { data: newData }, { deduplication: { id: 'data-job', // Required: unique deduplication key ttl: 300000, replace: true // Replace existing job with new data }});Deduplication Options Reference
Section titled “Deduplication Options Reference”| Option | Type | Default | Description |
|---|---|---|---|
id | string | (required) | Unique deduplication key |
ttl | number | - | Time in ms before unique key expires |
extend | boolean | false | Reset TTL on duplicate (returns existing job) |
replace | boolean | false | Remove old job and create new one |
Query Operations
Section titled “Query Operations”// Get job by IDconst job = await queue.getJob('job-id');
// Get job stateconst state = await queue.getJobState('job-id');// 'waiting' | 'prioritized' | 'delayed' | 'active' | 'completed'// | 'failed' | 'waiting-children' | 'unknown'
// Get job counts (sync in embedded mode; in TCP mode returns a Promise)const counts = queue.getJobCounts();// { waiting, prioritized, active, completed, failed, delayed, paused }
// Get job counts (async - works with TCP)const counts = await queue.getJobCountsAsync();
// Get jobs with filtering (sync - embedded mode only)const jobs = queue.getJobs({ state: 'waiting', start: 0, end: 10 });
// Get jobs with filtering (async - works with TCP)const jobs = await queue.getJobsAsync({ state: 'failed', start: 0, end: 50 });
// Get counts grouped by priority (sync - embedded mode only)const byPriority = queue.getCountsPerPriority();// { 0: 50, 10: 20, 100: 5 }
// Async version (works with TCP)const byPriority = await queue.getCountsPerPriorityAsync();Jobs by State
Section titled “Jobs by State”// Sync (embedded mode only)const waiting = queue.getWaiting(0, 10);const active = queue.getActive(0, 10);const completed = queue.getCompleted(0, 10);const failed = queue.getFailed(0, 10);const delayed = queue.getDelayed(0, 10);
// Async (works with TCP)const waiting = await queue.getWaitingAsync(0, 10);const active = await queue.getActiveAsync(0, 10);const completed = await queue.getCompletedAsync(0, 10);const failed = await queue.getFailedAsync(0, 10);const delayed = await queue.getDelayedAsync(0, 10);Count Methods
Section titled “Count Methods”// Async (work in both embedded and TCP modes)const waitingCount = await queue.getWaitingCount();const activeCount = await queue.getActiveCount();const completedCount = await queue.getCompletedCount();const failedCount = await queue.getFailedCount();const delayedCount = await queue.getDelayedCount();
// Total job count: sync (embedded mode only)const total = queue.count();
// Total job count: async (works with TCP)const total = await queue.countAsync();
// Check if pausedconst paused = queue.isPaused(); // syncconst paused = await queue.isPausedAsync(); // asyncBullMQ Compatibility Methods
Section titled “BullMQ Compatibility Methods”// Get prioritized jobs (priority > 0, separate state from 'waiting')const prioritized = await queue.getPrioritized(0, 10);const count = await queue.getPrioritizedCount();
// Get jobs waiting for children to complete (flow dependencies)const waitingChildren = await queue.getWaitingChildren(0, 10);const count = await queue.getWaitingChildrenCount();Queue Control
Section titled “Queue Control”// Pause processing (workers stop pulling)queue.pause();
// Resume processingqueue.resume();
// Remove all waiting jobsqueue.drain();
// Remove all queue dataqueue.obliterate();
// Remove a specific job (sync, fire-and-forget)queue.remove('job-id');
// Remove a specific job and await the removalawait queue.removeAsync('job-id');
// Wait until queue/server is readyawait queue.waitUntilReady();
// Close TCP connection (when done)queue.close();
// Async disconnect (compatibility)await queue.disconnect();Clean & Maintenance
Section titled “Clean & Maintenance”// Remove completed jobs older than 1 hour (sync, embedded mode only)// Returns the removed job idsconst removedIds = queue.clean(3600000, 100, 'completed');
// Async version (works with TCP)const removed = await queue.cleanAsync(3600000, 100, 'completed');
// Promote delayed jobs to waiting (returns number promoted)const promoted = await queue.promoteJobs({ count: 50 });
// Bulk retry failed jobs from the DLQ (only state: 'failed' is supported)await queue.retryJobs({ state: 'failed', count: 100 });Job Progress & Logs
Section titled “Job Progress & Logs”// Update job progressawait queue.updateJobProgress('job-id', 75);
// Get job logs (returns { logs: string[], count: number })const { logs, count } = await queue.getJobLogs('job-id', 0, 100);
// Add log entry to a jobawait queue.addJobLog('job-id', 'Processing step 3 completed');Job Dependencies
Section titled “Job Dependencies”// Get child job resultsconst childValues = await queue.getChildrenValues('parent-job-id');
// Get job dependencies infoconst deps = await queue.getJobDependencies('job-id');const depCounts = await queue.getJobDependenciesCount('job-id');
// Get child jobs with filterconst processed = await queue.getDependencies('parent-id', 'processed', 0, 10);const unprocessed = await queue.getDependencies('parent-id', 'unprocessed', 0, 10);
// Wait for a job to finish (requires a QueueEvents instance)import { QueueEvents } from 'bunqueue/client';const queueEvents = new QueueEvents('my-queue');const result = await queue.waitJobUntilFinished('job-id', queueEvents, 30000);Job State Transitions
Section titled “Job State Transitions”// Move job to completed with return valueawait queue.moveJobToCompleted('job-id', { success: true }, token);
// Move job to failed with errorawait queue.moveJobToFailed('job-id', new Error('reason'), token);
// Move job back to waitingawait queue.moveJobToWait('job-id', token);
// Move job to delayed with specific timestampawait queue.moveJobToDelayed('job-id', Date.now() + 60000, token);
// Move job to waiting-for-children stateawait queue.moveJobToWaitingChildren('job-id', token);Rate Limiting & Concurrency
Section titled “Rate Limiting & Concurrency”// Set global concurrency limit (max parallel jobs across all workers)queue.setGlobalConcurrency(10);queue.removeGlobalConcurrency();
// Set global rate limit (max jobs per second, token bucket)queue.setGlobalRateLimit(100); // 100 jobs per secondqueue.removeGlobalRateLimit();
// Throttle the queue to ~1 job/secawait queue.rateLimit(5000);Job Schedulers (Repeatable Jobs)
Section titled “Job Schedulers (Repeatable Jobs)”// Create or update a job schedulerawait queue.upsertJobScheduler('daily-report', { pattern: '0 9 * * *', // cron pattern // or: every: 3600000, // interval in ms}, { name: 'generate-report', data: { type: 'daily' },});
// Get a schedulerconst scheduler = await queue.getJobScheduler('daily-report');
// List all schedulersconst schedulers = await queue.getJobSchedulers(0, 100);const count = await queue.getJobSchedulersCount();
// Remove a schedulerawait queue.removeJobScheduler('daily-report');Deduplication Management
Section titled “Deduplication Management”// Look up job ID by deduplication keyconst jobId = await queue.getDeduplicationJobId('my-unique-key');
// Remove deduplication key (allows re-adding same jobId)await queue.removeDeduplicationKey('my-unique-key');Workers & Metrics
Section titled “Workers & Metrics”// List active workersconst workers = await queue.getWorkers();const count = await queue.getWorkersCount();
// Get historical metricsconst completedMetrics = await queue.getMetrics('completed', 0, 100);const failedMetrics = await queue.getMetrics('failed', 0, 100);
// Trim event logawait queue.trimEvents(1000);Stall Configuration
Section titled “Stall Configuration”Configure stall detection to recover unresponsive jobs.
queue.setStallConfig({ enabled: true, stallInterval: 30000, // 30 seconds without heartbeat = stalled maxStalls: 3, // Move to DLQ after 3 stalls gracePeriod: 5000, // 5 second grace period after job starts});
// Get current configconst config = queue.getStallConfig();See Stall Detection for more details.
DLQ Operations
Section titled “DLQ Operations”// Configure DLQqueue.setDlqConfig({ autoRetry: true, autoRetryInterval: 3600000, // 1 hour maxAutoRetries: 3, maxAge: 604800000, // 7 days maxEntries: 10000,});
// Get current DLQ config (sync; in TCP mode returns the client-side cache)const dlqConfig = queue.getDlqConfig();
// Get current DLQ config from the server (works with TCP)const serverDlqConfig = await queue.getDlqConfigAsync();
// Get DLQ entries (embedded mode only; returns [] in TCP mode)const entries = queue.getDlq();
// Filter entriesconst stalledJobs = queue.getDlq({ reason: 'stalled' });const recentFails = queue.getDlq({ newerThan: Date.now() - 3600000 });
// Get stats (embedded mode only; returns zeros in TCP mode)const stats = queue.getDlqStats();// { total, byReason, pendingRetry, expired, oldestEntry, newestEntry }
// Retry from DLQ (returns retried count in embedded mode;// fire-and-forget returning 0 in TCP mode)queue.retryDlq(); // Retry allqueue.retryDlq('job-123'); // Retry specific
// Retry by filter (embedded mode only)queue.retryDlqByFilter({ reason: 'timeout', limit: 10 });
// Purge DLQ (returns purged count in embedded mode;// fire-and-forget returning 0 in TCP mode)queue.purgeDlq();See Dead Letter Queue for more details.
Retry Completed Jobs
Section titled “Retry Completed Jobs”The retryCompleted() method allows re-queuing completed jobs for reprocessing. This is useful when you need to re-run a job that completed successfully, for example when business logic changes or you need to regenerate outputs.
// Retry a specific completed job (returns the re-queued count: 1 or 0)const retried = queue.retryCompleted('job-id-123');if (retried > 0) { console.log('Job re-queued for processing');}
// Retry all completed jobs (use with caution!)const count = queue.retryCompleted();console.log(`Re-queued ${count} completed jobs`);
// TCP mode: the sync method is fire-and-forget and returns 0.// Use the async version to get the real count back from the server.const remoteCount = await queue.retryCompletedAsync();Auto-Batching (TCP Mode)
Section titled “Auto-Batching (TCP Mode)”In TCP mode, queue.add() calls are automatically batched into PUSHB (bulk push) commands for higher throughput. This is enabled by default and requires no code changes.
How it works: If no flush is in-flight, the add is sent immediately (zero overhead for sequential await). If a flush is already in-flight, subsequent adds are buffered and sent together when the current flush completes or after maxDelayMs, whichever comes first.
// Auto-batching is enabled by default in TCP modeconst queue = new Queue('tasks');
// Sequential: no penalty, each add() sends immediatelyfor (const item of items) { await queue.add('task', item);}
// Concurrent: adds batch into a single PUSHB round-tripawait Promise.all([ queue.add('a', { x: 1 }), queue.add('b', { x: 2 }), queue.add('c', { x: 3 }),]);Configuration
Section titled “Configuration”const queue = new Queue('tasks', { autoBatch: { maxSize: 100, // Flush when buffer reaches this size (default: 50) maxDelayMs: 10, // Max ms to wait before flushing (default: 5) },});Disabling Auto-Batching
Section titled “Disabling Auto-Batching”const queue = new Queue('tasks', { autoBatch: { enabled: false },});Auto-Batch Options Reference
Section titled “Auto-Batch Options Reference”| Option | Type | Default | Description |
|---|---|---|---|
enabled | boolean | true | Enable or disable auto-batching |
maxSize | number | 50 | Max items before auto-flush |
maxDelayMs | number | 5 | Max delay in ms before auto-flush |
Store-and-Forward: queue.forward()
Section titled “Store-and-Forward: queue.forward()”Drain this queue’s jobs to a remote bunqueue server, the edge/IoT pattern (local embedded queue as offline buffer, central server as destination):
const forwarder = queue.forward({ to: { host: 'queue.example.com', port: 6789, tls: true, token: process.env.BQ_TOKEN }, queue: 'central-name', // optional remote queue name (default: same) concurrency: 4, // parallel forwards (default: 4) durable: true, // push remotely with durable: true (default: false)});
forwarder.on('forwarded', ({ id, remoteId, name }) => {});forwarder.on('error', (err) => {});await forwarder.close();Remote failures leave jobs local (retry → DLQ); forwarded jobs carry a
deterministic remote jobId (fwd:<queueKey>:<localId>), deduped server-side
within the custom-id retention window (bounded LRU; remote removeOnComplete
evicts it). Full guide: IoT & Edge.
Job Options Reference
Section titled “Job Options Reference”| Option | Type | Default | Description |
|---|---|---|---|
priority | number | 0 | Higher = processed first |
delay | number | 0 | Delay in ms before processing |
attempts | number | 3 | Max retry attempts |
backoff | number | { type, delay } | 1000 | Backoff base in ms, or { type: 'fixed' | 'exponential', delay } |
timeout | number | - | Processing timeout in ms |
jobId | string | - | Custom ID for deduplication (BullMQ-style idempotent) |
deduplication | object | - | Advanced deduplication config (ttl, extend, replace) |
removeOnComplete | boolean | false | Auto-delete after completion |
removeOnFail | boolean | false | Auto-delete after failure |
stallTimeout | number | - | Per-job stall timeout override |
repeat | object | - | Repeating job config |
durable | boolean | false | Immediate disk write (bypass buffer) |
lifo | boolean | false | Process in LIFO order (newest first) |
parent | { id, queue } | - | Parent job reference for flow dependencies |
Closing
Section titled “Closing”// Close TCP connection (no-op in embedded mode)queue.close();