Glossary
113 terms from Queues, Jobs and Events, in plain words, each with the chapter that teaches it.
A
- Ack (acknowledgment)
- A message from a receiver to the broker saying a message was handled. A nack (negative ack) says it could not be handled, so the broker can retry it or set it aside.
- Alert threshold
- A level of queue depth, job age or failures per hour that should send a message to a person. Priya might be alerted when more than 500 jobs are waiting.
- Amazon SQS / SNS
- AWS's simple managed queue (SQS) and its fan-out service to several queues (SNS). SQS hides a message while a worker handles it and deletes it when done.
- AMQP (Advanced Message Queuing Protocol)
- An open standard protocol for message brokers. RabbitMQ speaks it.
- amqplib
- The usual Node.js client library for talking to RabbitMQ.
- Apache Kafka
- A message broker that stores messages in a durable, replayable log instead of removing them once read. Large systems use it for high-volume streams of events read by many teams.
- app.inject
- A Fastify method that runs a request through your app without starting a real network server. It is handy for timing and testing routes, for example `app.inject({ method: "POST", url: "/orders" })`.
- Asynchronous
- Work that is started now and finished later without the caller waiting. Kettle & Crane's receipt email is asynchronous once it moves to a background job.
- At-least-once delivery
- The promise that every job runs one or more times and never zero. A crash can make the same job run twice, so jobs must be idempotent.
- attempts
- A BullMQ job option for the total number of tries a job gets, the first one included. By default it is 1.
- Autoscaling
- Adding or removing worker processes automatically, for example based on how many jobs are waiting.
B
- Background work
- Work the API hands off to be done later, outside the request. The request records what is needed and answers at once.
- Backoff
- The wait between a job's retries. A fixed backoff waits the same time each try, while exponential doubles it (1 s, 2 s, 4 s...).
- BEGIN / COMMIT / ROLLBACK
- SQL commands that start a transaction, save all its changes, or undo them. They make several changes happen together or not at all.
- Binding
- A RabbitMQ rule that connects an exchange to a queue using routing keys.
- BRPOP
- A blocking Redis command that waits for an item to arrive in a list. Workers use blocking commands like this, which is why ioredis must keep waiting.
- Bull Board
- An open-source web dashboard for BullMQ queues. It shows jobs by state with their data and logs, and can retry failed jobs with a click.
- BullMQ
- A Node.js job-queue library that stores its jobs in Redis. It provides retries, delays, schedules and workers.
C
- Change data capture (CDC)
- A technique that reads the database's own change log and streams each new row, such as an outbox row, out as it commits. It replaces polling in bigger systems.
- Choreography
- A style where each service reacts to events and announces its own, with no coordinator in charge. It is loosely coupled but the whole flow is hard to see.
- Command
- A message asking one particular receiver to do a piece of work, named with a verb such as send-receipt. It differs from an event, which announces something that already happened.
- concurrency
- A worker option for how many jobs one worker runs at the same time. It speeds up I/O-bound jobs such as Alex's label calls.
- Consumer group
- A set of readers that share a stream or topic. In Kafka each partition goes to one member and the group tracks its offsets, so two groups each read every message.
- CPU-bound
- Work that keeps the processor busy the whole time, like drawing a PDF invoice. Node runs JavaScript on one thread, so these jobs need more processes, not more concurrency.
- Cron expression
- A five-field pattern (minute, hour, day of month, month, day of week) that describes a repeating time. For example, 0 6 * * * means 6:00 every day.
D
- Dead letter queue
- A place where messages that could not be delivered end up so a person can deal with them. In BullMQ it is simply the queue's failed jobs.
- Dead-letter exchange
- A RabbitMQ exchange that receives messages that were nacked too often or expired and routes them to a queue you can inspect.
- Debezium
- A popular open-source change-data-capture tool, suited to larger systems running Kafka.
- Deduplication
- A recent BullMQ option that ignores a repeated job while the first one is still waiting or running (or for a set ttl). Use it for "don't queue this twice at the same time".
- defaultJobOptions
- A queue setting that gives every job on that queue the same options, such as attempts, backoff and removeOnComplete, unless a job sets its own.
- delay
- A job option that holds a job back for a number of milliseconds before it can run. During the wait the job is in the delayed state, which also covers waiting for a retry.
- Dual-write problem
- The trouble that saving an order and adding a job are two writes to two systems, so a crash between them loses or invents work.
E
- Event
- A fact about something that has already happened, named in the past tense, like order.placed. The sender does not say what should happen next.
- Event envelope
- The standard wrapper every event arrives in, with the fields id, type, occurredAt and data, so any subscriber can read any event.
- Event versioning
- Changing events safely: adding a field is fine, but a breaking change means publishing a new type such as order.placed.v2 alongside the old one until no subscriber reads it.
- EventEmitter
- Node's built-in way to call listener functions inside one program. It loses every event in flight on a crash and other processes cannot listen.
- Eventual consistency
- The trade-off where other parts of the system catch up a moment later. When the order route replies, stock has not dropped yet, but it will be right soon.
- Eviction policy
- What Redis does when memory is full. A queue needs noeviction, so writes fail loudly instead of Redis deleting keys, which would lose jobs.
- Exactly-once
- The idea that every job runs exactly one time. No queue can promise it, because a crash can happen after the work but before it is marked done.
- Exchange
- The RabbitMQ part that receives messages from producers and decides which queues get a copy. Types include direct, fanout and topic.
F
- Fan-out
- Sending one copy of an event to every subscriber. With BullMQ, give each subscriber its own queue and add the event to all of them.
- Fastify
- A Node.js web framework for building HTTP servers and APIs. It is used here to build the Kettle & Crane order API.
- Fat event
- An event that carries the details subscribers need, such as the email and items. Subscribers work alone, but the data is a copy from that moment.
- Fire-and-forget
- Starting work, for example by skipping await, and never checking on it. It loses work on restart, never retries and has no limit.
G
- Graceful shutdown
- Stopping after current work finishes and taking no new work. A worker does this by handling SIGTERM and SIGINT and calling worker.close().
H
- HTTP status codes
- Three-digit codes in an HTTP reply that say how a request went. Examples are 200 OK (a repeat request returns the existing order), 201 Created (a new order now exists), 202 Accepted (the request was taken in and the rest happens later), 400 (bad body), 404 (no such job), 409 (conflict), 422 (key reused with a different body), 429 Too Many Requests (called too often, so retry later) and 500 (server error).
I
- I/O-bound
- Work that mostly waits on something outside the process, such as an API, email server or database. Concurrency helps these jobs a lot.
- Idempotency key
- A random id the client creates once per checkout and sends in an Idempotency-Key header on every retry. The server stores it UNIQUE with the order so a repeat gets the first order back.
- Idempotent
- Having the same effect whether it runs once or many times. Jobs should be idempotent because queues may run them twice.
- Inbox
- A consumer's record of message ids already handled, so a repeat is skipped. The processed table plays this role.
- INSERT OR IGNORE
- SQLite's way to insert a row unless one with the same primary key exists, in which case nothing happens. The changes count shows whether a new row went in.
- ioredis
- The Redis client library for Node.js that BullMQ uses to talk to Redis.
J
- Jitter
- A random spread added to backoff waits so jobs that failed together do not all retry at the same moment. jitter: 0.5 makes each wait between half and all of the delay.
- Job
- One piece of work and its data, waiting in a queue. For example, a receipt job for order 1042.
- Job counts
- The number of jobs in each state, such as waiting or failed, read with `queue.getJobCounts(...)`. They are often served from a stats route so anyone can check queue health.
- Job log
- Lines of text a processor adds to a job with job.log, read back with queue.getJobLogs. Unlike console output they stay with the job.
- Job scheduler
- A rule stored in Redis that adds a new job each time its cron pattern or interval comes round. It runs once however many servers you have.
- Job states
- The stage a job is in: waiting, active, completed, failed, delayed or prioritized. A job moves from waiting to active to completed or failed.
- jobId
- A job option for choosing your own job id, so BullMQ will not add a second job with an id it already has. It cannot contain a colon or be only digits.
- JSON Schema
- A set of rules describing the shape of a JSON body, such as required fields and allowed values. Fastify checks it before the handler runs and answers 400 if it is broken.
K
L
- limiter
- A worker option that caps how many jobs start per time window, across all workers on the queue. It keeps you under a partner API's limit.
- Lock
- A key in Redis saying "this job is mine", held by a worker while it runs a job and renewed regularly. If it expires, the job is stalled.
- Loose coupling
- The publisher does not know who listens. Parts depend on the event's shape, not on each other, so teams can change code independently.
M
- maxRetriesPerRequest: null
- An ioredis option that makes it keep waiting on Redis instead of giving up on a command. BullMQ requires it because workers use long blocking commands.
- Message
- One unit a broker carries from producer to consumer: a job, a command or an event.
- Message broker
- A server whose whole job is to move messages from programs that send them to programs that receive them. RabbitMQ and Kafka are the best known.
N
- Naturally idempotent
- Safe to repeat by its nature, like setting order 1042's status to shipped. Adding 10 loyalty points is not.
- node:sqlite
- Node's built-in SQLite database module, which needs no install. It can hold tables such as orders, outbox and processed.
O
- Offset
- A message's position in a Kafka partition (0, 1, 2...). A consumer group records the offset it has reached.
- Oldest waiting job age
- How long the longest-waiting job has been in the queue, worked out as Date.now() minus job.timestamp. It shows how late the work is running, which depth alone can hide.
- Orchestration
- A style where one coordinator sends a command to each step in turn and tracks progress. It is easier to follow but the coordinator knows everyone.
- Outbox table
- A table of events waiting to be published, saved in the same transaction as the order. A relay later sends them to the queues, so neither the order nor its event gets lost.
P
- Partition
- A slice of a Kafka topic that can live on a different server. Order is kept only within a partition, and a message's partition key decides which one it lands in.
- Permanent failure
- A failure retrying cannot fix, such as an email address with no @ or a bug in a template.
- Persistence
- Redis saving its data to disk so jobs survive a restart. Queue Redis should have it on, along with noeviction.
- Polling
- Asking again and again, on a timer, whether there is new work. The relay polls the outbox every second.
- Prefetch
- A RabbitMQ setting limiting how many unacked messages one worker may hold at once, similar to BullMQ's concurrency.
- priority
- A job option where lower numbers run first, so 1 is most urgent. Jobs with no priority run before all prioritized jobs.
- Processed table
- A table of keys of jobs already done, written in the same transaction as the work. It makes a worker safe to run twice.
- Processor
- The async function you give a worker, called with one job at a time. Returning completes the job and throwing fails it.
- Producer
- The code that adds jobs or publishes messages, such as the order route.
- Progress
- A number a job reports while it runs, such as a percentage, recorded with job.updateProgress.
- Pub/sub
- Redis messaging that sends a message to whoever is subscribed at that moment and keeps nothing. A restarting subscriber misses the event for good.
Q
- Queue
- A waiting line of jobs. The producer adds jobs and a worker takes them off.
- Queue depth
- How many jobs are waiting in a queue, the backlog. Read it with the age of the oldest waiting job.
- QueueEvents
- A BullMQ class that listens to a queue's completed, failed and progress events from any process, through Redis.
R
- RabbitMQ
- A message broker that routes messages through exchanges to queues. It suits several services in different languages needing flexible routing.
- Redis
- The in-memory data store where BullMQ keeps its queues and jobs.
- Relay
- The loop that reads unsent outbox rows, publishes each to the queues, and then marks it sent.
- removeOnComplete / removeOnFail
- Job options that keep only the newest N finished or failed jobs in Redis and delete older ones. Keep more failed jobs than completed ones so you can inspect them.
- Replay
- Reading old messages again, which Kafka allows by moving a group's offset back.
- Retention
- How long Kafka keeps messages, whether or not anyone has read them. The default is 7 days.
- Routing key
- The label on a RabbitMQ message, such as order.placed, that bindings use to decide which queues get a copy.
S
- SIGTERM / SIGINT
- Signals sent to a process. SIGTERM comes from a deploy and SIGINT from Ctrl+C. A worker should handle both to shut down gracefully.
- Stalled job
- A job that is active but whose worker's lock expired, usually because the worker died or was blocked. BullMQ moves it back to waiting so another worker runs it.
- Subscriber
- Any part of the system that receives an event and reacts to it, like the email worker reacting to order.placed.
- Synchronous
- Work where the caller waits until every step is done, like the original order route that sent the email before answering.
T
- Thin event
- An event carrying only ids, such as { orderId: 1042 }, so subscribers look up the rest. It stays small and never goes stale.
- Time zone
- A name like America/Los_Angeles telling the scheduler which local clock to follow, including daylight saving. Without tz, a schedule follows the server's clock, usually UTC.
- Timeout
- A time limit after which waiting stops. A timed-out call is often a transient failure.
- Topic
- In Kafka, a named log of messages, such as orders. Producers append to it and messages stay after being read.
- Transaction
- Several database changes saved together or not at all. Kettle & Crane saves an order and its outbox row in one.
- Transient failure
- A failure that may pass if you try again later, such as a timeout or a busy service.
U
- Unique constraint
- A database rule that refuses a second row with the same value. It stops two orders from being saved under one Idempotency-Key.
- UnrecoverableError
- A BullMQ error class that, when thrown, skips remaining attempts and fails the job at once. Use it for permanent failures like a bad address.
- Unsent row
- An outbox row with no sent_at yet, waiting for the relay to publish it.
- Upsert
- Update if it exists, insert if it doesn't. Calling upsertJobScheduler again with the same id changes the rule instead of making a second one.
- UTC
- The world's reference clock, with no daylight saving. toISOString() prints in UTC, marked by Z.
- UUID
- A random 36-character id, such as one made with crypto.randomUUID(). Checkout pages use one as an Idempotency-Key.
V
- Visibility timeout
- How long Amazon SQS hides a message from other workers while one handles it (30 seconds by default). If it isn't deleted in time, another worker gets it.
W
- Worker
- A separate program that takes jobs off a queue and runs them with a processor. Redis hands each job to exactly one worker.