Glossary

120 terms from Observability, in plain words, each with the chapter that teaches it.

Search the glossary

A

Acknowledge: Telling the pager tool that you have seen a page and are working on it, so it doesn't escalate to someone else. At Kettle & Crane the primary should acknowledge within 15 minutes.
Action items: Concrete tasks from an incident review, each with an owner and a due date, that fix the gaps which let the problem happen.
Alert: A rule that watches a number for you and tells a person when it needs attention. A good alert is urgent, actionable and about something customers feel.
Alert fatigue: What happens when people get so many pages that need no action that they start ignoring them and miss the real one.
Alerting rule: A rule in a Prometheus rules file with a PromQL `expr`, a `for` duration, labels and annotations. The alert is active while the query returns results.
Alertmanager: A separate Prometheus-project program that decides who hears about firing alerts. It groups related alerts, drops duplicates and sends them to receivers such as email, Slack or a webhook.
amtool: A command-line tool that ships with Alertmanager. It is used to add, list and expire silences, which mute matching alerts for a while.
annotations: Text fields on an alerting rule, such as `summary` and `runbook_url`, meant for the person who receives the alert.
API (OpenTelemetry): The set of calls your code makes to create spans, such as starting a span. On its own it does nothing until an SDK is loaded.
Attribute: A key-value detail attached to a span, such as `http.status_code` or `kc.sku`. Attributes are what you read when you click a span in Jaeger.
Automatic instrumentation: Packages that hook into libraries you already use (Node's `http`, `fetch`, Fastify, pino) to create spans without changing your handlers.

B

Blameless review: An incident review that assumes everyone acted sensibly with what they knew, and asks how the system allowed the mistake instead of who made it. This encourages people to share details.
Bucket: One range of values in a histogram. Each bucket counts the measurements up to its limit, written in the `le` (less than or equal) label, and includes all smaller buckets.
Burn rate: How many times faster than even spending you are using up the error budget: the error ratio divided by the budget. A burn rate of 1 uses the budget up exactly over the 30 days.

C

Collector: An OpenTelemetry program that sits between services and backends, receiving, processing and forwarding telemetry. It adds filtering, several destinations and tail sampling.
Context propagation: Passing trace details (trace id and parent span id) along with a request from one service to the next, so their spans join one trace.
Contributing factor: One of the things that together made an incident happen or made it worse. There is rarely a single cause.
Counter: A metric that only goes up, such as orders received, and resets to 0 when the service restarts. You care about how fast it grows, not its value.
curl: A command-line program for sending HTTP requests and showing the replies. It can place orders against an API or read a page such as `/metrics`.

D

Dashboard: A Grafana page of graphs that stays open on a screen and updates by itself.
Data source: A place Grafana reads data from, such as a Prometheus server at `http://localhost:9090`. Each dashboard panel runs its queries against a data source.
Drill: Causing trouble on purpose, in a safe place, to check that alerts fire and that you can find and fix the problem. For example, deliberately breaking the stock service while sending steady orders.

E

Environment variable: A named text setting passed to a program when it starts, such as `FAIL_EVERY=3 node stock.ts`. Programs read them to change behavior without code changes, for example `LOG_LEVEL` and the `OTEL_` settings.
Error budget: What an SLO allows to go wrong over its window: 100% minus the SLO. A 99.5% SLO gives 0.5%, about 100 failed orders a month for Kettle & Crane.
Error budget policy: A written-down agreement on what the team does as the error budget runs low, such as stopping new features when it is gone.
Error ratio: The share of orders that failed, found by dividing the rate of failed orders by the rate of orders that reached the stock check.
Escalation: Passing a page, or a problem, to the next person when the first can't handle it, such as from the primary to the secondary on-call.
evaluation_interval: How often Prometheus checks its alerting and recording rules. It is a separate setting from `scrape_interval`.
Event (span): A timestamped note inside a span. `recordException` adds the error's type, message and stack trace as an event.
Exporter: The part of the OpenTelemetry SDK that sends finished spans somewhere, such as over OTLP to Jaeger.
Exposition format: The plain-text layout a `/metrics` page uses: `# HELP` and `# TYPE` lines, then one line per series with name, labels and value.

F

Fastify: The Node.js web framework used for the orders and stock services. It comes with the pino logger built in.
fetch: Node's built-in function for making outgoing HTTP calls. The orders service uses it to call the stock service.

G

Gauge: A metric that goes up and down, such as bags in stock, memory in use or requests in progress.
Grafana: A free, open-source tool for building dashboards of graphs that query data sources such as Prometheus.

H

Handoff note: A short note the outgoing on-call person writes at the swap: what paged, what is still open and any risky events coming up.
Head sampling: Deciding at the start of a trace whether to keep it. The first service decides and the flag in `traceparent` passes the decision to the others.
Histogram: A metric that counts measurements, such as request times, into buckets, and also keeps `_sum` and `_count`. It is the basis for percentiles.
histogram_quantile(): A PromQL function that estimates a percentile from a histogram's buckets. Because it assumes values are spread evenly inside a bucket, the answer is only as precise as your bucket edges.
HTTP status codes: The number a server returns to say how a request went. Common ones are 200 (OK), 201 (created), 404 (not found), 409 (conflict, such as out of stock), 500 (server error) and 503 (a service depended on is unavailable).

I

Incident: Any unplanned event that hurts the service or its users and needs a response now. Not every alert is an incident, and not every incident starts with an alert.
Incident commander: The person who coordinates an incident response: decides what to try next, who does it and when to update people. They don't type the fixes themselves.
Incident review: A short meeting and document after an incident about what happened and how to make it less likely or less painful next time. It has a timeline, impact, contributing factors and action items.
increase(): A PromQL function that gives the total growth of a counter over a window. The result is an estimate, so it isn't always a whole number.
Inhibition: Alertmanager automatically muting some alerts while a related, bigger alert is firing, such as muting all orders alerts while 'whole site down' fires.
Instrumentation: The code that creates telemetry, such as spans. It can be automatic, from library packages that hook into code you already use, or manual, with spans you add yourself.

J

Jaeger: A free trace backend that stores traces and lets you search and view them. Its web UI is on port 16686 and it accepts OTLP on port 4318.
Job, target: A target is one service Prometheus scrapes; a job is a group of targets doing the same work, listed in `prometheus.yml`.
jq: A command-line tool that reads JSON and picks out or reshapes what you ask for. It is handy for filtering JSON logs such as pino's, for example with `select(.level >= 40)`.

L

Label: A key-value pair on a metric that says which thing is measured, such as `result="failed"`. Each different combination of labels is a separate time series.
Label cardinality: How many different values a label can take. High-cardinality labels such as order ids or emails create endless time series and fill up Prometheus, so keep them out of labels.
Label filter: Curly brackets after a PromQL metric name that keep only matching series, such as `kc_orders_total{result="failed"}`; `!=` means 'not'.
Log: A record of one event with details, such as 'order 18 created'. Logs answer 'what exactly happened?'
Log level: A number saying how important a pino log line is, from trace (10) to fatal (60). The logger writes only lines at or above its configured level.

M

Manual span: A span you create in your own code with a tracer, such as the `reserve stock` span around the call to the stock service.
Metric: A number measured over time, such as orders per minute or request duration. Metrics are cheap to keep and answer 'how much, how often, how fast?'
Mitigation, root cause: Mitigation stops the harm, such as rolling back a deploy or restarting a service, even if you don't know why it happened. The root cause is why it happened and can wait until customers can order again.
Monitoring: Watching for problems you already know about, such as 'tell me if the error count goes above 10'. Observability extends this to exploring problems you didn't predict.
Multiwindow burn-rate alert: An alert on how fast an error budget is being spent that checks a long window (proving the problem is big) and a short one (proving it is still happening). For example, page at burn rate 14.4 over 1 hour and 5 minutes, and open a ticket at 6 over 6 hours and 30 minutes.

N

Node.js: A JavaScript runtime that runs programs outside the browser, such as the Kettle & Crane services. Version 22.18 or later can run `.ts` files directly with `node`.
NodeSDK: The OpenTelemetry class for Node.js that `tracing.ts` creates and starts to collect spans, hand them to the exporter and switch on the instrumentations.

O

Observability: How well you can answer questions about a running system from the outside, without changing its code, even questions nobody thought of in advance.
On-call: Being the person who gets paged for a period of time and agrees to respond quickly, even at night. Dana and Lee share it a week at a time.
OpenTelemetry (OTel): An open, vendor-neutral standard for traces, metrics and logs, plus free libraries that produce them. You can switch the product that stores the data by changing settings, not code.
OTEL_ environment variables: Settings the OpenTelemetry SDK reads from environment variables starting with `OTEL_`. `OTEL_SERVICE_NAME` names the service and the sampler variables choose how many traces to keep.
OTLP: The OpenTelemetry Protocol, a standard way to send telemetry from services to a backend. It can run over HTTP, for example to Jaeger on port 4318.

P

Page, ticket: A page is an alert that rings the on-call developer's phone now, even at 3 a.m.; a ticket is work that waits for the next office day.
Panel: One graph or number on a Grafana dashboard. Each panel runs queries against a data source.
Parent span, child span: A child span is work done inside another span, its parent, and is drawn indented under it. Every span records its parent's span id, which is how Jaeger builds the tree.
Pending, firing: An alert is pending while its condition is true but the `for` time hasn't passed, and firing after that, when it is sent on. It is inactive when the query returns nothing, and `resolved` once it stops firing.
Percentile, p95: The value below which a given share of measurements fall. p95 is the time that 95% of requests beat, so it shows the slow tail that an average hides.
pino: Fastify's built-in logger, which writes one JSON object per line and adds fields such as `reqId`, `level` and `time`.
Primary and secondary: The two on-call roles: the primary gets every page first, and the secondary is the backup who receives the page if the primary doesn't acknowledge.
Production: Where your code runs for real customers, as opposed to your own computer. You can't pause it or add a `console.log` and retry.
prom-client: The standard Prometheus library for Node.js, used to create counters, gauges and histograms and to serve the `/metrics` page.
Prometheus: A free program that regularly scrapes the `/metrics` page of each service and stores the numbers as time series. Its web page is on port 9090.
PromQL: Prometheus's query language, used on the Query page, in alerting rules and in Grafana panels to ask about time series.
promtool: A program in Prometheus's archive that checks config and rule files (`check rules`) and runs rule unit tests (`test rules`) without starting Prometheus.

R

Range vector: A PromQL time window in square brackets, such as `[5m]`, that returns every sample in the window for each series, not just the latest.
rate(): A PromQL function that turns a counter's range vector into its average per-second increase over the window, and copes with counter resets after restarts. Because a raw counter only climbs, you graph its `rate()` instead.
Receiver, webhook: A receiver is where Alertmanager sends alerts, such as email, Slack or PagerDuty. A webhook is a receiver that is any URL you choose, which gets an HTTP POST with the alerts as JSON.
Recording rule: A rule that runs a query on every evaluation and saves the result as a new time series with a name you choose, such as `sli:orders_errors:ratio_rate1h`.
RED method: A rule for dashboards of any service that answers requests: show its Rate, its Errors and its Duration.
redact: A pino option that takes a list of field paths and replaces their values with `[Redacted]`, as a safety net against logging secrets.
Resource: The details attached to everything a process sends, such as `service.name`, host name and process id. It is how Jaeger knows which spans came from which service.
Retention: How long Prometheus keeps its data. The default is 15 days, so 30-day SLOs need `--storage.tsdb.retention.time=30d` or longer.
Rota: The schedule of who is on-call when. Kettle & Crane's is one week each for Dana and Lee, swapping on Monday at 10 a.m.
Rule unit test: A test, run with `promtool test rules`, that feeds made-up time series into your rules and checks which alerts fire at given fake times.
Runbook: A short page written in calm daylight that tells a tired on-call person what an alert means and exactly what to check and do first.
runbook_url: An alert annotation that links to the runbook for that alert, so the link arrives with the page.

S

Sampling: Keeping only some traces to save the CPU, network and disk cost of tracing in busy systems. Logs and metrics are unaffected.
Scrape: Prometheus visiting a target's `/metrics` page and saving what it reads as time series. It repeats at a set interval, such as every 5 seconds.
Scribe, communications lead: Two incident roles besides the commander: the scribe keeps the timeline, and the communications lead (Priya, for customers) handles updates.
SDK: The engine behind the OpenTelemetry API. It collects spans, groups them and hands them to an exporter.
Server span, client span: A server span handles a request that came in; a client span is a call out to another service, such as orders' `fetch` to stock. `span.kind` records which role a span played.
Severity level: A label saying how bad an incident is so everyone knows how hard to respond: SEV1 means nobody can order, SEV2 means many customers are affected and SEV3 means small impact. Alerts also carry a `severity` label such as page or ticket.
Signal: One of the three kinds of data a program gives you about itself: logs, metrics and traces. Each answers a different kind of question.
Signal (process): A message the operating system sends a process, such as SIGINT when you press Ctrl+C or SIGTERM when a host stops it. `tracing.ts` catches both so it can send the last spans before exiting.
Silence: A rule in Alertmanager that mutes matching alerts for a while, such as while Dana is fixing stock.
SLA: Service level agreement: a contract promising a level of service to customers, with a penalty such as a refund if missed. Make your SLO stricter than any SLA.
SLI: Service level indicator: a measure of how well the service is doing for users, as good events divided by valid events. For orders it is the share of orders that didn't fail.
SLO: Service level objective: a target for an SLI over a time window, such as 99.5% of orders succeeding over 30 days.
Span: One step of a trace: a piece of work with a name, start time, duration and attributes. Spans form a tree where child spans are work done inside their parent.
Span id, trace id: The trace id is 32 hex characters shared by every span in a trace; each span has its own 16-character span id and records its parent's. Pino lines carry `trace_id` so you can jump to the trace.
Span status: Whether a span is unset (normal) or an error. `span.setStatus({ code: SpanStatusCode.ERROR })` marks it failed, and Jaeger shows it in red.
Structured log: A log line written as data with named fields, usually one JSON object per line, so tools like jq can search and count by field.
sum by: A PromQL aggregation: `sum` adds series together, and `sum by (route)` keeps one result per value of the `route` label. Histogram queries use `sum by (le)` to keep buckets apart.
Symptom, cause: A symptom is what users feel, such as failing orders or slowness; a cause is what's behind it, such as high CPU. Page on symptoms and keep causes on dashboards.

T

Tail sampling: Deciding after a trace ends whether to keep it, for example keeping every trace with an error. It needs an OpenTelemetry Collector.
Telemetry: The logs, metrics and traces a running system sends out about itself.
Time series: One metric name with one set of labels, plus its list of samples, one value per scrape.
Trace: One request's path through the services, made of spans. Traces answer 'where did the time go?'
Trace backend: A program that stores traces and lets you search and view them. Jaeger is one example.
traceparent: The HTTP header, from the W3C Trace Context standard, that carries the trace id and parent span id to the next service. It has four parts: version, trace id, parent span id and flags.
Tracer: The object, made with `trace.getTracer()`, that you use to start manual spans with `startActiveSpan`.
Trouble switches: Environment variables in the stock service that cause problems on purpose: `SLOW_MS` makes every call wait that many milliseconds, and `FAIL_EVERY` makes every Nth call fail with a 500.

U

undici: The library that Node's built-in `fetch` is built on. The undici instrumentation makes spans for outgoing `fetch` calls.

V

Vendor-neutral: Not tied to one company's product. Because OpenTelemetry is vendor-neutral, your code doesn't care which product stores the data.

W

Waterfall: Jaeger's view of a trace, with each span drawn as a bar from its start to its end and indented under its parent. It shows where a slow request spent its time.

Y

YAML: A text format for settings that uses indentation (spaces, never tabs) for structure, `key: value` for values and `- ` for list items. Prometheus and Alertmanager config files use it.