AI Teammates: What's Actually New

Every framework the last two lessons covered, Fitts's list, Sheridan and Verplank's levels, the four-function model, situation awareness, calibrated trust, assumed automation that behaves the same way today as it did yesterday: a fixed control law, a fixed sensor, a fixed set of failure modes you could characterize once and rely on. A large language model agent breaks that assumption in three specific ways. It can change after deployment, without the person relying on it necessarily knowing. It increasingly gets handed a goal instead of a function, and decides its own steps toward it. And when it fails, it tends to fail by sounding right, not by throwing an error.

None of that makes the last two lessons wrong. It means the target they were calibrated against, stable, characterizable automation, doesn't fully describe what a lot of people are now teaming with.

A 2025 study at CHI, human-computer interaction's leading venue, tested something simple and unsettling: can people actually tell when the AI model underneath a tool they already use has been updated? Studying a real historical-photo facial recognition platform alongside a controlled experiment, the researchers found that users routinely couldn't reliably detect a model swap, that the swap measurably changed their behavior and task performance anyway, and that people who did notice something had changed often built incorrect explanations for what was different (Mohanty et al., 2025).

The paper opens with a concrete illustration of why this matters: a 2023 Tesla Full Self-Driving software update that preceded a wave of driver reports of unexpected swerving near highway medians, a visible symptom of the same underlying problem, drivers whose calibrated trust no longer matched the system now running underneath them (Mohanty et al., 2025). FIG. 1 shows the shape of that problem directly: capability jumps at each update, and trust doesn't jump with it.

The teaming loop from the first lesson in this series assigned fixed roles: the machine senses and analyzes, the human decides and acts. That's function allocation, a task broken into sub-functions assigned in advance, and it's the shape every framework in this series has assumed so far, including Parasuraman, Sheridan, and Wickens's own four-function model.

An agentic AI teammate breaks the fixed-role shape. Instead of a function, it gets handed a goal, and its own internal loop decides how to break that goal into steps, often without surfacing each one for approval. The human's role shifts from allocating functions in advance to setting a goal and a boundary, then reviewing what came back, a fundamentally different relationship even when a checkpoint still exists at the end of it.

Classical automation tends to fail in ways that announce themselves: a sensor reads out of range, an actuator doesn't respond, a system throws an error. Large language models tend to fail by sounding exactly as confident when they're wrong as when they're right, a different category of problem for situation awareness, not just a different failure rate.

The clearest recent demonstration of how far that problem reaches: an analysis of the 2025 Neural Information Processing Systems conference, one of the most competitive, expert-reviewed venues in artificial intelligence, found at least 53 accepted papers, about one percent of everything published that year, containing fabricated citations, invented authors, titles, or venues that read as entirely plausible. Every one of those papers had been reviewed by three to five domain-expert researchers, and the fabrications still weren't caught (Ansari, 2026). If confident, fluent hallucination can get past expert peer reviewers checking a paper in their own specialty, it isn't a problem situation awareness training alone is going to solve for anyone else either.

Long before language models, human factors researchers studied how many robots a single operator can actually supervise at once. Olsen and Wood's 2004 fan-out equation frames it as a ratio: how long an agent can safely be neglected before it needs attention, divided by how long the human's attention takes once it's needed (Olsen & Wood, 2004). Empirical work since has generally put fan-out in the single digits for real tasks, not the dozens a purely additive view of automation might suggest, and pushing past that point tends to raise workload faster than it raises output.

Whether that same ratio holds for a person overseeing several AI agents instead of several physical robots is an open question, not a settled one. AI agents don't need physical neglect tolerance the way a robot does, but they introduce exactly the non-stationary behavior and confident-failure problems the last two sheets described, at whatever fan-out a team decides to run.

Put together, the human-in-the-loop checkpoint from the first lesson in this series now has to survive three things the classical automation literature never had to plan for: a teammate that can change without notice, a teammate that decides its own steps toward a goal rather than executing an assigned function, and a teammate whose wrong answers don't look wrong. None of the last two lessons' tools, situation awareness, calibrated trust, use and misuse, stop being relevant. They just have to work against a harder target.

Mica Endsley's own 2023 extension of situation awareness theory to human-AI teams argued that transparency and explainability were the fix, giving a person enough insight into a natural-language reasoner to keep comprehension and projection intact (Endsley, 2023). The three problems this lesson just described, silent updates, self-directed goal decomposition, and confident wrong answers, are exactly what makes that fix harder to deliver than the paper's framing suggests. What actually designing a checkpoint against that harder target looks like is where the next lesson in this series picks up.