Trust, Complacency, and Situation Awareness

Every framework in the last lesson assumed the human at the checkpoint actually understands what's happening. Mica Endsley's 1995 model gives that assumption a name and a structure: situation awareness, defined as perceiving the elements in the environment, comprehending what they mean, and projecting how they'll develop, three levels that build directly on each other (Endsley, 1995).

Level 1, perception, is simply noticing what's there: a light is blinking, a value has changed, a message has arrived. Level 2, comprehension, is understanding what that means for the current situation, not just seeing the light but knowing what it's telling you. Level 3, projection, is anticipating what happens next well enough to act before it becomes a problem. Losing any level breaks the ones above it. You can't project what you don't comprehend, and you can't comprehend what you never perceived in the first place.

Situation awareness doesn't survive automation by default. Endsley and Esin Kiris automated a navigation task using an expert system and found that operators monitoring the automation passively had measurably worse situation awareness, and measurably slower decisions once the automation failed, than operators controlling the same task directly (Endsley & Kiris, 1995).

The mechanism matters more than the headline finding, and it's shown in FIG. 1. It wasn't that passive operators saw less information, the automation still displayed the same data. It was the shift from active to passive processing itself: actively doing a task forces you to keep building comprehension and projection as you go, while watching someone, or something, else do it lets that updating lapse, even when nothing about the displayed data changed.

Raja Parasuraman and Victor Riley gave the field a compact vocabulary for what actually goes wrong once trust enters the picture. Use is a person's voluntary decision to engage automation at all, shaped by trust, workload, and perceived risk. Misuse is relying on automation past the point its actual reliability supports, monitoring failures and decision biases baked in. Disuse is the opposite, ignoring or switching off automation that's actually trustworthy, often triggered by one too many false alarms. Abuse is a different failure entirely: automation deployed by designers or managers without regard for how it changes the human's job, imposed rather than chosen (Parasuraman & Riley, 1997).

Notice that three of these four failure modes are really about miscalibrated trust in one direction or the other. That's not a coincidence.

John Lee and Katrina See made that miscalibration precise. Trust in automation should be calibrated, meaning it tracks the system's actual capability rather than running consistently above or below it. When trust runs higher than what the system can actually deliver, the result is overtrust, which shows up as misuse. When trust runs lower than the system's real capability, the result is undertrust, which shows up as disuse, useful automation sitting there switched off (Lee & See, 2004).

None of this is a solved problem that only affects untrained users. A 2025 randomized clinical trial gave physicians who had completed twenty hours of AI-literacy training, specifically covering how to critically evaluate AI output, the option to voluntarily consult ChatGPT-4o while diagnosing clinical cases. Physicians given deliberately erroneous AI recommendations still showed significantly degraded diagnostic accuracy compared to physicians given error-free advice, even though consulting the AI was entirely their own choice and they'd been trained on exactly this risk (Qazi et al., 2025).

That's automation bias, in Parasuraman and Riley's terms, misuse, happening to people who knew the concept, had been warned about it directly, and chose to rely on the system anyway. Calibrated trust isn't a training problem that goes away once someone's been told about it.

This is what actually sits behind the checkpoint from the last lesson. A human-in-the-loop gate only does its job if the human standing at it has real situation awareness and calibrated trust at that exact moment, not just formal authority to say no. A checkpoint staffed by someone out of the loop, or someone whose trust in the system is miscalibrated in either direction, isn't much of a safeguard.

Mica Endsley's 2023 update on supporting human-AI teams argues that transparency and explainability are what's supposed to keep this from happening with modern AI specifically: giving a person enough insight into what the system is doing and why to keep comprehension and projection intact, not just a raw feed of what it perceived (Endsley, 2023). Whether that actually works with systems that reason in natural language rather than fixed control laws, and what happens to teaming when the teammate itself doesn't hold still, is where the next lesson in this series picks up.