bunqueue S3 Backup and Disaster Recovery for SQLite Job Queues

blog · operations

Disaster recovery, one S3 backup.

bunqueue stores everything in a single SQLite file, which makes backups simple, but simple does not mean optional. Set up automated S3 backups, retention, and a tested recovery path.

Why Backup?

SQLite is crash-safe (WAL mode + fsync), but it can’t protect against:

• Disk failure - hardware dies, data is gone
• Accidental deletion - rm -rf happens
• Corruption - filesystem bugs, power loss during write
• Migration errors - bad deploy wipes the data directory

S3 backup gives you point-in-time recovery with minimal effort.

Enabling S3 Backup

Configure via environment variables or a configuration file:

# Required
S3_BACKUP_ENABLED=1
S3_BUCKET=my-bunqueue-backups
S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Optional
S3_REGION=us-east-1               # Default: us-east-1
S3_ENDPOINT=                      # Custom endpoint (MinIO, R2, etc.)
S3_BACKUP_INTERVAL=21600000       # Every 6 hours (default)
S3_BACKUP_RETENTION=7             # Keep the last 7 backups (default)
S3_BACKUP_PREFIX=backups/         # Key prefix (default)

Or pass them when starting the server:

S3_BACKUP_ENABLED=1 \
S3_BUCKET=my-backups \
S3_REGION=us-east-1 \
bunqueue start --data-path ./data/queue.db

Compatible Storage Providers

Any S3-compatible storage works:

Provider	S3_ENDPOINT	Notes
AWS S3	(empty)	Default
Cloudflare R2	https://<account>.r2.cloudflarestorage.com	No egress fees
MinIO	http://minio:9000	Self-hosted
DigitalOcean Spaces	https://<region>.digitaloceanspaces.com	Simple setup
Backblaze B2	https://s3.<region>.backblazeb2.com	Cheapest storage

# Cloudflare R2 example
S3_BACKUP_ENABLED=1
S3_BUCKET=bunqueue-backups
S3_ENDPOINT=https://abc123.r2.cloudflarestorage.com
S3_ACCESS_KEY_ID=your-r2-key
S3_SECRET_ACCESS_KEY=your-r2-secret
S3_REGION=auto

Backup Process

The backup runs on a timer (default: every 6 hours):

1. Checkpoint WAL - PRAGMA wal_checkpoint(TRUNCATE) forces all pending writes into the main database file
2. Compress - the backup is gzip-compressed before upload
3. Upload to S3 - stored with a timestamp-based key, alongside a .meta.json sidecar with a SHA-256 checksum
4. Cleanup old backups - keeps only the most recent S3_BACKUP_RETENTION backups

Note

The backup process is non-blocking. It creates a snapshot without stopping the queue, so jobs continue processing during backup.

Backup File Naming

Backups are stored under the S3_BACKUP_PREFIX (default backups/) with this key pattern:

backups/
  bunqueue-2026-01-15T00-00-00-000Z.db            (gzip-compressed)
  bunqueue-2026-01-15T00-00-00-000Z.db.meta.json  (checksum + sizes)
  bunqueue-2026-01-15T06-00-00-000Z.db
  bunqueue-2026-01-15T06-00-00-000Z.db.meta.json

Note that the .db object is gzip-compressed even though the key has no .gz suffix; the .meta.json sidecar records the compression flag and a SHA-256 checksum of the original database.

Disaster Recovery

The built-in restore command handles download, decompression, and validation for you:

1. Stop bunqueue

   systemctl stop bunqueue
   # or: docker stop bunqueue

2. Find the backup to restore

   # Requires the same S3_* env vars plus BUNQUEUE_DATA_PATH
   bunqueue backup list

3. Restore it

   bunqueue backup restore backups/bunqueue-2026-01-15T18-00-00-000Z.db --force

   The restore is validate-before-replace: it verifies the SHA-256 checksum from the .meta.json sidecar, checks the SQLite header, runs an integrity check on a temp file, and only then atomically swaps it into place. On any failure the live database is left untouched.

4. Restart bunqueue

   systemctl start bunqueue
   # or: docker start bunqueue

If you prefer to do it manually, remember the object is gzip-compressed despite the .db key:

aws s3 cp s3://my-bunqueue-backups/backups/bunqueue-2026-01-15T18-00-00-000Z.db ./
mv /var/lib/bunqueue/queue.db /var/lib/bunqueue/queue.db.corrupted
gzip -dc bunqueue-2026-01-15T18-00-00-000Z.db > /var/lib/bunqueue/queue.db

bunqueue will recover the queue state from the restored database, reloading pending jobs, cron schedules, and DLQ entries.

Caution

Recovery restores to the backup point-in-time. Jobs added or completed between the backup and the failure are lost. Reduce S3_BACKUP_INTERVAL for critical workloads.

Backup Monitoring

Monitor backup health in production:

# Check the backup configuration
bunqueue backup status

# List backups and check the most recent timestamp
bunqueue backup list

# Trigger a backup on demand
bunqueue backup now

Set up alerts for:

• No backup in 2x the interval - backup process may be failing
• Backup size anomalies - sudden size changes may indicate issues
• S3 upload failures - check credentials and bucket permissions

Supplementary Strategies

S3 backup covers most scenarios, but consider layering additional protection:

Filesystem Snapshots

# LVM snapshot (instant, zero downtime)
lvcreate -s -n bunqueue-snap -L 1G /dev/vg0/bunqueue

# ZFS snapshot
zfs snapshot tank/bunqueue@daily

Cron-based local backup

# Copy SQLite file every hour (WAL must be checkpointed first)
0 * * * * sqlite3 /var/lib/bunqueue/queue.db ".backup /backups/queue-$(date +\%H).db"

Replication to secondary server

# rsync the database file periodically
*/30 * * * * rsync -az /var/lib/bunqueue/ backup-server:/bunqueue-replica/

Best Practices

1. Enable S3 backup from day one - don’t wait for your first data loss
2. Test recovery regularly - a backup you can’t restore from is worthless
3. Monitor backup health - alert on missed backups
4. Use retention policies - keep the last 7-30 backups depending on your needs
5. Consider R2 or B2 for cost-effective storage (no egress fees with R2)