Most data observability comes down to a threshold. Alert if the row count drops below a number. Alert if freshness exceeds a number. Alert if the null rate in a column crosses a number.

Thresholds have exactly two failure modes, and every team gets both.

They fire constantly, on days when nothing is wrong, until people build a mail rule and stop looking. Or they never fire at all, right up until the day the number is so bad that a threshold was not what you needed.

Both failures have the same cause. A threshold is a guess about where "bad" starts, and it knows nothing about how the process behaves on an ordinary day.

A pipeline is a process

Walter Shewhart worked this out at Bell Labs in the 1920s, on manufacturing lines, and the argument has not aged.

Every process varies. Some of that variation is inherent to the process itself: the ordinary day-to-day noise of a system running exactly as designed. Some of it is a signal that something changed. Shewhart called these common cause and special cause, and the entire point of a control chart is to tell them apart, because they call for opposite responses.

Chasing common cause is how you make a stable process worse. Ignoring special cause is how you ship a quiet failure for a year.

A threshold cannot make the distinction. It has one number, and the process has a distribution.

What a control chart asks instead

A control chart does not ask "is this value bad." It asks "is this value consistent with the process that produced every value before it."

You plot the metric run over run. You compute the mean and the control limits from the process's own history, conventionally three standard deviations either side. Then the chart tells you something a threshold cannot:

A row count of 1.2 million is a signal when the process has been running 1.0 million plus or minus 40 thousand for six months. Nothing was "below the floor." Nothing was empty. Nothing a threshold watches went wrong. But the process changed, and you want to know why today rather than in the quarterly review.

A row count of 800 thousand is noise when the process genuinely ranges from 700 thousand to 1.3 million. Alerting on it teaches your team that the alerts are wrong, which is the most expensive thing you can teach them.

The Western Electric rules extend this. One point outside three sigma. Two of three points beyond two sigma on the same side. Eight consecutive points on one side of the mean.

That last one is the one that pays for the whole exercise.

Drift is the failure that actually costs you

A pipeline that breaks loudly gets fixed in an hour. Somebody is paged, the job is red, the dashboard is empty, and the incident has an owner before lunch.

A pipeline that quietly starts dropping two percent of records gets fixed in a year. No threshold is crossed. No day looks wrong. The chart of daily counts looks fine, because two percent is inside the noise. And in the meantime it has trained a model, fed a report, and sat underneath a filing.

Eight consecutive points on one side of the mean catches that. Not because two percent is a big number, but because a run of eight is not something a stable process does.

That is the whole trick. A control chart is sensitive to pattern, and a threshold is only sensitive to magnitude. Drift is a pattern.

What to chart

Not everything. Charting everything is how you rebuild the alert fatigue you were trying to escape.

Row count per run. Null rate on the columns that matter, not all of them. Distinct cardinality of the keys you join on. The fraction of records that arrive late. Job duration. And the mean of one or two numeric fields the business would actually notice, which is where you catch an upstream unit change that nothing structural would ever see.

What it costs

Two things, and the second is why most attempts fail.

You need history. Twenty to thirty runs before the limits mean anything. That is a few weeks on a daily pipeline, and it is a real wait.

And you have to recompute the limits after a deliberate change. When the business onboards a new region and the row count legitimately steps up, the chart will scream, and it should, once. Then somebody has to say "this is the new process," re-baseline, and write down why. Skip that step and within two months the chart is charting a process that no longer exists, everybody knows the limits are stale, and you are back to a mail rule.

That is the discipline part. It is not hard. It is just not automatic, and it will not survive without an owner.

What it buys

The technical answer is that you catch drift months earlier.

The real answer is organizational, and it is worth more. It replaces "the data looks off" with "this point is outside the control limits, and here are the three days it started."

One of those is an opinion, and opinions are settled by whoever is more senior in the room. The other is a measurement. In a regulated environment, where you will eventually have to explain to somebody why a number moved, that difference decides who wins the argument and how long it takes.

Data engineering keeps reinventing quality control, badly, and calling it observability. Manufacturing settled this a century ago and wrote it down. The tooling is different. The statistics are not.

Back to writing