Notifications
Queue and scale
How VitNode delivers notifications to large audiences with queue tasks, cron jobs, batched delivery with saved progress, crash recovery and retention.
Notifications run on VitNode's database-backed queue and cron. No extra service is needed, but the queue worker must run: without a working cron adapter, nothing is delivered in the background.
Queue tasks and cron jobs
All of these belong to @vitnode/core. Tasks appear in Debug Panel ā Queue (/admin/core/debug/queue) and cron jobs in Debug Panel ā Cron & queue (/admin/core/debug/cron).
| Name | Kind | Does |
|---|---|---|
notifications-fanout | Task, 10 attempts | Delivers one notification to its recipients, batch by batch |
notifications-email | Task, 5 attempts | Sends due emails and digests |
notifications-cleanup | Task, 3 attempts | Removes notifications older than the retention period |
notifications-schedule | Cron, every 5 minutes | Plans digests, recovers stalled work and queues the email task when due |
notifications-cleanup | Cron, daily at 02:00 | Queues the cleanup task |
A request that publishes starts delivery right after its response is sent. Anything that misses that, such as a publish from a queue task, waits for the queue worker, which runs every minute.
Batched delivery
The fan-out task walks the recipients in batches of fanoutBatchSize, 500 by default. Each batch is one short transaction that filters out deleted users, opted-out members and anyone access rejects, then writes inbox items, unread counts and immediate email deliveries, and saves its progress on the event. Realtime updates go out after the batch commits.
- A crash loses at most one batch. The retry continues after the last saved batch.
- A batch that runs twice changes nothing. Core keeps one receipt per notification and member, and only members without a receipt get an item.
- Large audiences do not block the queue. One task stops after 20 seconds and queues a new task for the rest.
- No deadlocks between writers. Every inbox write locks the affected members in user id order, so concurrent writers wait for each other instead.
Recovery
| Stuck work | Recovered by |
|---|---|
| Queue task left processing by a stopped worker | The queue worker, after 10 minutes (the task's lease, 600 s by default) |
| Notification unfinished with no task left | notifications-schedule, after 10 minutes, by queuing a new task |
Email left in sending | notifications-schedule, after 15 minutes, by setting it back to pending |
| Two workers on one notification | A row lock. The second waits, then continues from the saved progress |
Retention
The nightly cleanup removes, in batches:
- inbox items with no activity for
retentionDays, lowering unread counts for unread ones; - older notifications that no inbox item points to;
- finished email delivery records older than
retentionDays; - pending digest email for notifications no digest picked up within 14 days. The inbox items stay.
Configure the workers
How much work each run does and how long notifications are kept are deployment settings, so they live in your API config, not in the AdminCP. Add a notifications block to buildApiConfig:
import { buildApiConfig } from '@vitnode/core/vitnode.config'
export const vitNodeApiConfig = buildApiConfig({
// other options unchanged
notifications: {
retentionDays: 30,
fanoutBatchSize: 1000,
emailBatchSize: 100,
emailConcurrency: 8,
},
})| Name | Type | Required | Default | Description |
|---|---|---|---|---|
retentionDays | number | No | 90 | Days to keep notifications, from 1 to 3,650 |
fanoutBatchSize | number | No | 500 | Recipients per delivery transaction, from 10 to 5,000 |
emailBatchSize | number | No | 50 | Emails one worker run picks up, from 1 to 500 |
emailConcurrency | number | No | 4 | Emails sent at the same time during a run, from 1 to 20 |
A value outside its range is clamped to the nearest limit, and a value that is not a whole number falls back to the default. Restart the API to apply a change.
The defaults suit most sites. Raise fanoutBatchSize when one announcement to a huge audience needs too many runs, and emailConcurrency when your email provider allows more parallel sends. Lower emailBatchSize when a slow provider makes runs time out.