Most data observability comes down to a threshold. Alert if the row count drops below a number. Alert if freshness exceeds a number. Alert if the null rate in a column crosses a number.

Thresholds have exactly two failure modes, and every team gets both.

They fire constantly, on days when nothing is wrong, until people build a mail rule and stop looking. Or they never fire at all, right up until the day the number is so bad that a threshold was not what you needed.

Both failures have the same cause. A threshold is a guess about where "bad" starts, and it knows nothing about how the process behaves on an ordinary day.

A pipeline is a process

Walter Shewhart worked this out at Bell Labs in the 1920s, on manufacturing lines, and the argument has not aged.

Every process varies. Some of that variation is inherent to the process itself: the ordinary day-to-day noise of a system running exactly as designed. Some of it is a signal that something changed. Shewhart called these common cause and special cause, and the entire point of a control chart is to tell them apart, because they call for opposite responses.

Chasing common cause is how you make a stable process worse. Ignoring special cause is how you ship a quiet failure for a year.

A threshold cannot make the distinction. It has one number, and the process has a distribution.

What a control chart asks instead

A control chart does not ask "is this value bad." It asks "is this value consistent with the process that produced every value before it."

You plot the metric run over run. You compute the centre line and the control limits from the process's own history, conventionally three sigma either side. Then the chart tells you something a threshold cannot.

A row count of 1.2M is a signal when the process has been running 1.0M ± 40k for six months. Nothing was below the floor. Nothing was empty. Nothing a threshold watches went wrong. But the process changed, and you want to know why today rather than in the quarterly review.

A row count of 800k is noise when the process genuinely ranges from 700k to 1.3M. Alerting on it teaches your team that the alerts are wrong, which is the most expensive thing you can teach them.

The Western Electric rules extend this. One point outside three sigma. Two of three points beyond two sigma on the same side. Eight consecutive points on one side of the mean.

That last one is the one that pays for the whole exercise.

Getting sigma right

One detail decides whether any of this works, and it is the detail most implementations get wrong. Sigma is not the standard deviation of the series.

A pipeline gives you one observation per run, which puts you on an individuals chart, and there sigma is estimated from the average moving range between consecutive points. The distinction sounds pedantic and it decides the outcome. If the process has already drifted, the series standard deviation absorbs that drift into its own value and widens the limits until nothing can ever trip them. You compute a chart that certifies the broken process as being in control. A moving range has a short memory and does not.

Seasonality is the other trap. Row counts have weekday shape and month-end shape. Chart a raw daily count against limits computed across all days and you will flag every Monday until somebody turns the chart off. Compare a day against its own kind, or chart a ratio that has the cycle already divided out.

Proportions and counts want their own chart types too — p-charts for rates, c-charts for event counts — and if you need to catch a small sustained shift faster than a run of eight will catch it, that is what EWMA and CUSUM are for.

Drift is the failure that actually costs you

A pipeline that breaks loudly gets fixed in an hour. Somebody is paged, the job is red, the dashboard is empty, and the incident has an owner before lunch.

A pipeline that quietly starts dropping two percent of records can run for a year. No threshold is crossed. No day looks wrong. The chart of daily counts looks fine, because two percent is inside the noise. And in the meantime it has trained a model, fed a report, and sat underneath a filing.

Eight consecutive points on one side of the mean catches that. Not because two percent is a big number, but because a run of eight is not something a stable process does.

That is the whole trick. A control chart is sensitive to pattern, and a threshold is only sensitive to magnitude. Drift is a pattern.

Where thresholds still belong

A control limit is not a business limit, and nothing above argues for throwing the thresholds away.

If a regulatory filing has to land by eight in the morning, 8:01 is late even when the pipeline has been consistently late for six months and the chart calls it perfectly ordinary. If a percentage cannot exceed one hundred, the historical distribution has no opinion worth hearing. Hard constraints belong in data contracts, validation rules, and SLOs, and they belong there precisely because they come from outside the process.

Control limits sit alongside them, watching for the changes that stay technically valid while ceasing to be characteristic. One kind of rule protects a boundary. The other detects a shift. A team with only the first has no idea when its process moved, and a team with only the second will eventually explain to a regulator that the late filing was within three sigma.

What to chart

Not everything. Charting everything is how you rebuild the alert fatigue you were trying to escape.

Row count per run. Null rate on the columns that matter, not all of them. Distinct cardinality of the keys you join on. The fraction of records that arrive late. Job duration. And the mean of one or two numeric fields the business would actually notice, which is where you catch an upstream unit change that nothing structural would ever see.

What it costs

Two things, and the second is why most attempts fail.

You need history. Twenty to thirty runs before the limits mean anything, and more than that if there is a weekly cycle you have to see through first. On a daily pipeline that is a few weeks, and it is a real wait.

And you have to recompute the limits after a deliberate change. When the business onboards a new region and the row count legitimately steps up, the chart will scream, and it should, once. Then somebody has to say "this is the new process," re-baseline, and write down why. Skip that step and within two months the chart is charting a process that no longer exists, everybody knows the limits are stale, and you are back to a mail rule.

That is the discipline part. It is not hard. It is just not automatic, and it will not survive without an owner.

What it buys

The technical answer is that you catch drift months earlier.

The real answer is organizational, and it is worth more. It replaces "the data looks off" with "this point is outside the control limits, and here are the three days it started."

One of those is an opinion, and opinions are settled by whoever is more senior in the room. The other is a measurement. In a regulated environment, where you will eventually have to explain to somebody why a number moved, that difference decides who wins the argument and how long it takes.

Data engineering keeps reinventing quality control, badly, and calling it observability. Manufacturing settled this a century ago and wrote it down. The tooling is different. The statistics are not.

Back to writing