The Threshold Is Lying to You: Why 'Automated' Isn't the Same as 'Assessed'
Every article in this series so far has been about a number quietly misleading someone — an average, an alarm, an outlier, a trend line, a gap in the data. This one is about a line. Specifically, the line a system crosses when it stops flagging something for a person and starts deciding on its own.
The base rates piece was about a detector's own threshold working against it — even a highly accurate system can flood a human reviewer with false alarms if the cutoff is wrong. This article picks up one step further down that same road: what happens once nobody's reviewing anything at all, because the system was given the authority make the decision?
Confidence is a measurement. The threshold is a choice.
A few distinctions worth keeping straight before the worked example, because the language gets used loosely and the looseness is where the risk hides:
Automated decision — the system reaches an outcome and acts on it, full stop. No human sees the case unless something downstream forces a second look.
Augmented decision — the system scores or flags a case, but a person makes the actual call. The system is an input, not the decision-maker.
The threshold is a policy choice, not a technical fact. Wherever "confident enough, decide alone" turns into "not confident enough, send it to a person," someone picked that cutoff. That's a judgment about acceptable risk, sitting inside what looks like a setting in a config file.
"Assessed" is a separate question from "accurate." A system can be right most of the time and still never have had anyone check whether that cutoff is the right one for what's actually at stake if it's wrong.
The flagship example, part one: sorting tea
The clearest version of this lives in our own AI or Augmented Intelligence? tool, so it's worth walking through exactly as it works before the stakes get higher.
A tea shop wants to sort incoming stock as Green or Black Tea automatically. The system looks at a new sample's steep time and water temperature, compares it to a set of cups it already knows the answer for, and takes a vote among the nearest matches — that's its confidence: if all of the nearest cups agree, confidence is high; if it's close to a coin flip among them, confidence is low. A threshold slider, labeled right there in the tool as "confidence needed to decide automatically," decides what happens next. Above the line, the system sorts the tea itself — no one involved. Below it, the sample gets set aside for a person to check.
Drag that slider up near 100% and almost everything gets kicked to a person, because almost nothing is ever unanimous. Drag it down toward 50% and the system starts deciding almost everything on its own, including plenty of genuinely borderline cups. Somewhere in between is a number that someone has to actually choose — and in the tool, you choose it by dragging a slider, which is exactly the problem this article is about.
Nobody loses sleep over a mis-sorted bag of tea. That's what makes it a clean way to see the mechanic without the stakes getting in the way. But the exact same mechanic — a confidence score, and a line drawn across it — is quietly running underneath decisions where the stakes are not trivial at all.
The flagship example, part two: the same mechanic, real stakes
Picture a company hiring for a high-volume role, using an automated resume screener built on the same logic as the tea sorter: a confidence score, and a threshold that decides who reviews it. At launch:
| Score band | Outcome | Applicants | Share |
|---|---|---|---|
| 75 and above | Auto-advance to interview | 90 | 9% |
| 35–74 | Sent to a recruiter for review | 170 | 17% |
| Below 35 | Auto-reject | 740 | 74% |
Add the two automated ends together and 830 of every 1,000 applicants — 83% — never have a person look at their application before the system decides their outcome. Only the middle 17% ever reaches a human at all.
Nobody touches the model after that. No retraining, no change to the score bands — the threshold stays exactly where it was set on day one. Six months later, same cutoffs, same volume of applicants each month:
| Score band | Outcome | Applicants | Share |
|---|---|---|---|
| 75 and above | Auto-advance to interview | 40 | 4% |
| 35–74 | Sent to a recruiter for review | 110 | 11% |
| Below 35 | Auto-reject | 850 | 85% |
Now 890 of every 1,000 applicants — 89% — never reach a person, and the auto-reject rate on its own has climbed from 74% to 85%. Nobody touched the threshold. Nobody changed the model. On a dashboard, that rising rejection rate could look like a perfectly ordinary story — the applicant pool for this role just got weaker over six months, which does happen.
Here's how you'd actually tell the difference: look at what still reaches a person, not just what the model auto-decides.
| Period | Reviewed by a recruiter | Advanced to interview | Advance rate |
|---|---|---|---|
| Month 1 (launch) | 170 | 68 | 40% |
| Month 3 | 140 | 56 | 40% |
| Month 6 | 110 | 44 | 40% |
Recruiters are advancing resumes to interview at exactly the same rate every time it's checked — 40%, whether it's month one, month three, or month six. The applicants a human actually looks at are getting through at the same rate they always did. What changed wasn't who was applying, and it wasn't the cutoffs either — 75 and 35 never moved. What changed was the score the model assigned to a given applicant: the same caliber of resume that would have scored a 40 at launch could score a 28 by month six, on a model nobody retrained and a threshold nobody touched. The line stayed exactly where it was. What drifted underneath it was where any given applicant landed relative to that line.
That's model drift: a system whose confidence numbers quietly stop meaning what they meant on day one, without the threshold — or anyone's decision about it — ever changing. Nobody built a check for this into the original launch. It took someone going back, months later, and comparing what the model was doing now against what a person reviewing the same band was still finding — a deliberate, later look, not something the system flagged on its own.
This is exactly what a quarterly check is for. A recruiter-advance snapshot at the three-month mark would already have shown the auto-reject share climbing past 79% — real movement from the 74% it launched at, even though month six hadn't happened yet — while the advance rate on the resumes recruiters still saw held right at 40%, same as launch. That combination is the tell: applicants weren't getting weaker, the threshold's relationship to the data was shifting. Caught at that point, the fix isn't a new model. It's widening the reviewed band back out toward where it started, then checking again next quarter. Nobody did that here, which is the entire reason six months of drift went unnoticed instead of three.
Where the number is supposed to come from
In the tool, you set the threshold by dragging a slider wherever feels right. In real life, that's the part that should never be a feeling.
This is what an Algorithmic Impact Assessment (AIA) actually does: it rates how much is at stake if an automated decision is wrong, on a scale from Level 1 (minimal impact) to Level 4 (very high impact), and that rating is what should set how much autonomy the system is allowed — not the highest confidence number the model happens to produce. A system sorting tea can reasonably run with a low bar and a lot of autonomy, because almost nothing rides on getting it wrong. A system deciding who gets an interview, whose transaction gets flagged, or whose badge gets revoked needs a much higher bar — or a human in the loop regardless of how confident the model says it is. The threshold isn't a number the model gives you. It's a number the stakes are supposed to give you, and an assessment is how you find out what the stakes actually are before you set it.
Why one assessment isn't the end of it
There's a second gap the tea-shop demo can't show, because it's a simplification the tool deliberately makes: the tea data is fixed. Confidence numbers mean the same thing today as they did when the page loaded, until someone clicks "New data" and draws a fresh set of cups.
Real systems don't get a reset button. The applicant pool shifts. Fraud patterns evolve. Network traffic changes shape. This is exactly what happened to the resume screener above: nothing about the model changed between month one and month six, but what it was looking at did, and the confidence score stopped tracking what it used to track. Nothing about the auto-reject rate climbing looked wrong from a dashboard — it only showed up once someone compared what the model was doing now against what a person reviewing the same band was still finding.
Canada's Directive on Automated Decision-Making, which the AIA framework sits under, requires the assessment to be updated when a system's functionality or scope changes, with scheduled reviews at a frequency tied to the system's impact level. "Assessed" isn't a box checked once at launch. It's a question that's supposed to get asked again — and the drift above is exactly the kind of thing a scheduled review, not a one-time sign-off, is built to catch.
How this looks in other departments
| Department | The threshold already running | What an assessment would actually ask |
|---|---|---|
| Finance | Transactions under $2,500 auto-approve with no manual review | Was $2,500 chosen because fraud exposure was actually modeled at that level, or because it was last year’s number carried forward? |
| IT | A security tool auto-quarantines any device that crosses a risk score | Has the false-quarantine rate ever been checked by device type – new hires, devices back from repair – or does the threshold just assume the training data still looks like today’s traffic? |
| Real Property | Badge access auto-revokes on an anomalous entry pattern | Does “anomalous” still account for legitimate multi-site staff and misreading readers, or was that checked once, years ago, before the building’s own usage patterns changed? |
So what do you actually do about it?
| If… | Then… | Because… |
|---|---|---|
| A system decides autonomously above some confidence threshold | Set that threshold based on what’s actually at stake if it’s wrong, not on whatever number felt reasonable | The threshold is a policy choice about risk, dressed up as a technical setting – treating it as a config default skips the one question that actually matters |
| The decision touches someone’s opportunity, benefits, rights, or access | Score the impact level before finalizing the threshold – an AIA-style Level 1–4 rating is a ready-made structure for this | Higher-stakes decisions warrant a higher bar for full autonomy, or a mandatory human review regardless of how confident the model claims to be |
| The system has been live for a while | Re-check the threshold against current data, not just the data it launched on – comparing outcomes in the still-reviewed band over time is a cheap way to tell drift from a genuinely weaker pool | Confidence scores drift as the world underneath the model changes – a threshold that was right on day one can quietly stop being right without ever looking different |
| Either way | Be able to name the threshold, who set it, and when it was last checked | If nobody in the room can answer that about a system deciding real outcomes right now, that’s the gap this article is about |
Try it yourself
Go and use the tool embedded below (or click here to open it in another tab) AI or Augmented Intelligence? tool) — the same tea-sorting system from the top of this article — and drag the threshold slider yourself (Confidence needed to decide automatically). Watch how many cases the system decides alone versus hands to a person as the line moves. Then hit "New data" and watch the same threshold produce a different mix of outcomes on a fresh set of cups — a quick, honest reminder that the comparison set underneath a confidence score is never really fixed in the real world, even though the demo has to fix it to stay explainable. Read the page in a bit more detail to understand what the “k (number of neighbours)” slider does!
Where this shows up beyond the dashboard
This is really the same lesson as every other article in this series, one level up. A missing data point, a misleading average, a base rate nobody accounted for — all of it eventually feeds into systems that act on it, and increasingly, those systems act without waiting for a person to sign off. The volume that makes automation attractive is the same volume that makes it easy for nobody to have actually checked where the line got drawn.
DAVHILL Group's Algorithmic Impact Assessment work exists for exactly this gap: rating a system's decision impact on the same Level 1–4 scale used in the federal AIA framework, and building in the reassessment that a threshold set once and left alone will eventually need. If you've got a system deciding things on its own and nobody in the building can currently name the threshold, who set it, or when it was last checked — that's the conversation worth having.
Stephen Davies, DAVHILL Group. Connect on LinkedIn.