Agency Under Mediation — Part X
For most of this series, the mediator has been human.
An organizer.
A representative.
A manager.
A caregiver.
A pastor.
A regulator.
An expert.
An institution.
Now introduce a mediator that can read thousands of documents, generate recommendations, classify cases, write messages, diagnose patterns, operate software, call tools, coordinate workflows, and increasingly perform actions without waiting for a human being to specify every intermediate step.
The mediation problem changes.
But it does not disappear.
In fact, artificial intelligence makes the underlying architecture easier to see.
The ordinary model is:
The human has a limitation.
Information volume.
Time.
Memory.
Technical skill.
Pattern recognition.
Administrative capacity.
The AI contributes something.
The human emerges more capable.
That is mediation.
But another transformation is possible:
At first glance, the human is still present.
There is still a button.
A signature.
An approval field.
A supervisor.
A “human in the loop.”
But presence is not the same thing as agency.
If the human does not understand what occurred,
cannot reconstruct why it occurred,
lacks time to evaluate it,
cannot meaningfully reject it,
cannot alter its scope,
or possesses no practical alternative to accepting the output,
then the system may contain a human while no longer being meaningfully governed by one.
The human has become ceremonial.
That is the final mediation problem.
And because machines can operate at a scale and speed unavailable to ordinary human mediators, the transition can happen very quickly.
Automation Has Never Simply Replaced Work
The central mistake is older than artificial intelligence.
We imagine automation as subtraction.
A human performs ten functions.
A machine takes over five.
The human now performs five.
Formally:
But automation rarely works that cleanly.
Lisanne Bainbridge identified the problem in her classic 1983 paper Ironies of Automation.
Automating routine operation can leave the human responsible for the exceptional conditions automation cannot handle.
That means the machine performs the task during ordinary conditions while the human is expected to intervene during abnormal ones—the precise moment when understanding the system is most difficult.
The transformation is closer to:
The job did not disappear.
The cognitive task changed.
The operator once continuously interacted with the system.
Now the operator monitors a system that usually works.
And then, occasionally:
something goes wrong.
At that moment the human must suddenly recover:
what the system is doing,
why it is doing it,
what state it is in,
what happened previously,
what consequences will follow,
and how to intervene.
Automation can therefore remove practice from the human while preserving responsibility for failure.
That is an agency problem.
The Machine Can Make the Human Less Ready at the Exact Moment the Human Matters Most
Consider a person who manually performs a task every day.
Their competence is continuously exercised.
Now automate 99 percent of the task.
The person becomes a supervisor.
Most days:
nothing happens.
The machine performs correctly.
The person observes.
Then on day 200:
the machine encounters an unusual condition.
The system asks the human to take over.
But the human has now spent 199 days not doing the task.
The architecture becomes:
while:
because the human is called precisely when the normal system has stopped behaving normally.
That is one of Bainbridge's ironies.
Automation may leave people responsible for the parts humans are worst positioned to perform after extensive automation.
This is not an argument against automation.
It is an argument against pretending that automation removes the human problem.
It redistributes it.
Automation Changes the Role of the Human
Raja Parasuraman, Thomas Sheridan, and Christopher Wickens later formalized automation across different types and levels.
Automation can intervene in:
information acquisition,
information analysis,
decision and action selection,
and action implementation.
And each can be automated to different degrees. Their core design insight is important: automation does not simply replace human activity. It changes human activity and introduces new coordination demands.
This gives us a much better model for AI.
An AI might merely acquire information:
Or analyze information:
Or recommend a decision:
Or select the decision:
Or execute:
Those are radically different mediation architectures.
Calling all of them “AI assistance” hides the important distinction.
The question is not:
Is AI being used?
It is:
Which part of the transformation has been delegated to the machine?
Decision Support and Decision Authority Are Different
Imagine a physician using an AI system.
Architecture A:
The AI mediates information.
Architecture B:
Architecture C:
These are not three versions of the same thing.
The location of decision authority moves.
In A:
remains obvious.
In B:
authority is ambiguous.
In C:
the machine may possess executable authority even if a human retains theoretical supervisory authority.
This is why interface vocabulary can be misleading.
Recommendation.
Suggestion.
Decision support.
Assistant.
Those labels describe products.
They do not necessarily describe the actual authority graph.
Overreliance Is Not a New AI Problem
Parasuraman and Victor Riley distinguished several human relationships with automation:
use, misuse, disuse, and abuse.
Their concept of misuse includes overreliance on automation, resulting in failures of monitoring and biased decisions.
Their concept of automation abuse is particularly relevant: designers or managers can automate functions without adequately considering consequences for human performance, leaving human roles to emerge as by-products of automation design rather than being deliberately designed themselves.
That phrase describes a large fraction of bad AI deployment.
The workflow begins:
What can the AI do?
Then:
Let's automate that.
Only afterward does somebody ask:
What is the human supposed to do now?
That sequence is backwards.
The correct first question is:
What human agency must the completed system preserve?
Then automation can be designed around it.
Automation Bias Makes Nominal Approval Dangerous
A human reviewer sounds reassuring.
Suppose an AI produces 1,000 recommendations.
The human must approve each.
Problem solved?
Not necessarily.
Automation-bias research shows that people can over-rely on automated decision aids.
Parasuraman and Dietrich Manzey's review found automation bias in both novices and experts. Their analysis describes errors of both omission and commission when automated systems are imperfect and emphasizes the role of attention, workload, and human-automation interaction.
A systematic review by Kate Goddard, Abdul Roudsari, and Jeremy Wyatt similarly found that decision-support systems can improve performance while introducing new errors through overreliance. Their review identified factors including workload, task complexity, time pressure, trust, presentation, and the form in which automation provides its output.
So imagine:
The human sees hundreds of correct recommendations.
Their learned expectation becomes:
Then the hundred-and-first unusual recommendation appears.
The human has technically retained approval authority.
But psychologically:
The human begins confirming rather than deciding.
That is rubber-stamp mediation.
This is one of the central conclusions of the entire series.
Suppose a system says:
APPROVE
REJECT
A human must click one.
We might therefore claim:
“The human makes the final decision.”
But ask:
Does the human know what evidence the model used?
Can they see uncertainty?
Can they independently inspect the underlying evidence?
Do they understand the system's limitations?
Do they have enough time?
Are they punished for disagreeing?
Does rejecting the recommendation create substantial additional work?
Can they produce another answer?
Do they possess domain expertise?
Can they stop the machine after approval?
Can they reverse the resulting action?
If the answers are mostly no, the buttons exaggerate the human's agency.
Formally:
and:
Those are perhaps the two most important equations in AI governance.
The European AI Act Makes This Distinction Explicit
The European Union's AI Act provides a useful contemporary example.
Article 14 requires high-risk AI systems covered by its requirements to be designed so they can be effectively overseen by natural persons.
The regulation does not stop at:
put a human in the process.
It gets much more specific.
Depending on the system and context, the human overseer must be enabled to understand the system's capacities and limitations, remain aware of the risk of automation bias, correctly interpret output, decide not to use the output, disregard or reverse it, and intervene in or interrupt operation safely.
Notice what this implies.
Meaningful oversight requires capabilities.
The human needs:
Remove enough of those and “human oversight” becomes nominal.
Part VI established that legitimate authority requires scope, provenance, duty, contestability, and review.
AI adds another condition:
information sufficient to exercise authority.
Suppose I have formal authority to override the AI.
But I cannot understand what it is doing.
Then:
while:
The system can truthfully tell auditors:
“Humans may override the model.”
But practically, the human may have no basis for knowing when to do so.
That is why interpretability cannot be separated from governance.
Explanation is not merely a nice user-interface feature.
In some systems it is part of the causal pathway through which human authority becomes exercisable.
Now reverse it.
Suppose the system is beautifully transparent.
The human sees:
every input,
confidence scores,
reasons,
logs,
alternatives,
limitations.
But the organization's policy says:
AI recommendations must be followed unless a vice president approves an exception.
Now:
but:
The human understands perfectly.
And cannot act.
That is also not meaningful oversight.
So:
and:
alone.
It requires an architecture connecting the two.
Time Is Part of Authority
Suppose a human has:
information,
competence,
and authority.
But the system requires a decision in 400 milliseconds.
Human intervention is theoretically available.
Practically:
The temporal structure of a system can therefore erase human agency even while every formal permission remains intact.
This matters enormously as AI agents increasingly operate through:
APIs,
software tools,
automated markets,
network defense,
robotics,
industrial control,
and other high-speed environments.
If an AI can make twenty consequential state changes before a human can perceive the first one, then the relevant governance boundary cannot exist only at the instant of action.
Control may need to move earlier.
Human Control Can Exist Before the Individual Action
This is where simplistic “human-in-the-loop” language breaks down.
Suppose an autonomous system performs ten thousand low-level decisions per second.
A human cannot approve each one.
Does that mean humans have no meaningful control?
Not necessarily.
Humans can govern:
the objective,
the permitted action space,
the prohibited action space,
resource limits,
credentials,
environment boundaries,
risk tolerances,
stop conditions,
escalation conditions,
and the authority under which the machine operates.
Then the architecture becomes:
The human does not decide every action.
The human governs the space of legitimate actions.
That is much closer to how authority already works in complex human organizations.
A company president does not approve every forklift movement.
Governance does not require microcontrol.
It requires legitimate delegation.
This Is Meaningful Human Control
Filippo Santoni de Sio and Jeroen van den Hoven developed a philosophical account of meaningful human control over autonomous systems.
They propose two central conditions:
Tracking
The system's behavior should appropriately respond to the relevant reasons and facts of the human agents responsible for designing and deploying it.
Tracing
The system should be designed so that outcomes can be traced to at least one human in the chain of design and operation who possesses appropriate understanding and responsibility.
That is immediately recognizable within our mediation framework.
Tracking asks:
remain connected to:
Tracing asks:
be connected back to:
The machine may possess enormous operational autonomy while still being part of a human-governed architecture.
Autonomy at the machine level does not necessarily require sovereignty at the machine level.
Machine Autonomy and Human Authority Are Different Variables
This distinction is critical.
People often imagine:
Sometimes.
But not necessarily.
Suppose an autonomous warehouse robot independently determines:
route,
speed,
local obstacle avoidance,
battery management,
and charging schedule.
Operational autonomy is high.
But human governance may still tightly constrain:
which facility it operates in,
where it may enter,
what loads it may carry,
maximum speed,
minimum separation from people,
emergency-stop behavior,
which commands it may execute,
and when it must escalate.
So:
can be high while:
remains high.
The variables are not opposites.
This is Part VI again:
bounded authority over transformations.
The Dangerous AI Is Not Necessarily the Most Autonomous One
A seemingly modest recommender can exercise enormous practical authority.
Imagine a system that merely recommends which job applicants should receive interviews.
Formally:
But suppose recruiters process 5,000 applications.
The model ranks them.
Recruiters inspect only the top 200.
Now:
and:
The system never officially rejected anyone.
It merely determined who became visible.
Yet its mediation changed the state:
The AI possessed no legal authority to hire.
But it possessed epistemic gatekeeping power.
That is Part VII's information problem in machine form.
AI Models Are Maps With Execution Capability
Part VII argued:
is unavoidable.
The danger appears when:
Artificial intelligence intensifies this problem because the model may no longer merely describe reality.
It may act.
A risk score can trigger investigation.
A classification can route a case.
A generated recommendation can alter care.
A detection system can block a transaction.
An agent can execute a command.
So:
The machine's representation of the world can now directly modify the world.
That is qualitatively important.
The map has acquired hands.
The AI Does Not Need to “Want Power”
This is where anthropomorphic framing becomes distracting.
We do not need an AI to:
desire control,
seek authority,
develop ambition,
or intentionally manipulate humans
for agency transfer to occur.
The system can become functionally authoritative because humans restructure their institutions around it.
Consider:
therefore:
therefore:
therefore:
therefore:
No machine ambition required.
The architecture alone produces dependency.
That is Part V's dependency problem.
The Machine Can Become the Expert Nobody Can Challenge
Suppose an AI system consistently outperforms individual humans at a task.
Eventually someone says:
“Why would we ever override it?”
That question sounds rational.
Perhaps overrides usually make outcomes worse.
But then:
and:
Soon the human reviewer becomes anomalous:
Why did you reject the model?
The burden shifts.
Initially:
must justify its recommendation.
Eventually:
must justify disagreement.
That may be appropriate in some tightly validated systems.
But governance has changed.
The transformation should be explicit.
Otherwise authority migrates silently through performance.
Accuracy Is Not Authorization
This principle from Part VI becomes essential.
Suppose:
and:
That is strong evidence for using AI.
It may justify substantial automation.
It does not, by itself, answer:
Who is permitted to deploy it?
Which decisions may it make?
What consequences may it execute?
What data may it access?
What happens when uncertainty is high?
Who bears responsibility?
Who may contest its output?
What rights does the affected person retain?
Performance answers:
How well does it do the task?
Governance asks:
What authority should follow from that competence?
Those remain different questions.
A Machine Cannot Legitimize Its Own Authority
Imagine an AI says:
Based on my superior performance, you should grant me additional permissions.
Perhaps the statement is logically persuasive.
It still cannot be the final authorization.
Why?
Because the entity receiving authority cannot be the sole source determining that the authority is legitimate.
That would recreate the self-sealing problem from every earlier article.
The authority grant must originate elsewhere.
Credentials Are Authority
Now move from decision support to agentic systems.
Suppose an AI agent can:
read email,
edit a database,
publish content,
send messages,
transfer files,
modify infrastructure,
place orders,
schedule actions,
or call external services.
The moment you provide credentials, the AI possesses something more consequential than abstract capability.
It possesses executable authority.
Formally:
That distinction is extremely important.
A model may know how to delete a database.
Without authority:
Give it credentials with deletion permissions:
The governance question is therefore not merely:
What can the model reason about?
It is:
What transformations is the system actually authorized to cause?
Capability and Authority Should Be Separated
Human institutions already understand this principle.
A bank employee may know how transfers work.
That does not automatically authorize every transfer.
A lawyer may know how to file a document.
That does not authorize filing anything for anyone.
A software engineer may understand production infrastructure.
That does not necessarily mean they hold every production credential.
AI systems should be treated with the same distinction:
A system may possess broad reasoning capability while receiving narrowly scoped executable permissions.
That is not a limitation of intelligence.
It is governance.
The Agent Does Not Need to Hold Every Credential It Can Request
Consider two architectures.
Architecture A
The AI receives:
database credentials,
publication credentials,
payment credentials,
administrative credentials,
communication credentials.
Then:
The same system performs:
interpretation,
decision,
authorization,
and execution.
Now consider:
Architecture B
then:
then:
The AI can reason broadly.
But executable authority is mediated separately.
That architecture preserves a distinction between:
what the system proposes
and:
what the system is allowed to cause.
This is the machine version of representation without replacement.
Proposal Is Not Execution
This is perhaps the cleanest design principle in agent governance.
and:
An AI can propose:
Send this message.
That does not mean it should possess independent authority to send every message it can compose.
An AI can recommend:
Deploy this configuration.
That does not mean its reasoning process should automatically produce production-state changes.
Separating those stages allows each transformation to acquire its own governance.
Who proposed it?
Who authorized it?
Under what policy?
Who executed it?
What changed?
Can we reconstruct the sequence?
That is provenance.
NIST Treats Human-AI Governance as a Role Problem
The NIST AI Risk Management Framework approaches this from an organizational direction.
Its governance guidance emphasizes clearly defining and differentiating human roles and responsibilities across AI development, deployment, operation, oversight, and risk management.
The framework calls for documented human-AI configurations, operator proficiency, oversight processes, organizational accountability, and engagement with affected actors. Its Playbook also discusses appeal and override mechanisms and asks organizations to identify who is ultimately responsible for AI-assisted decisions.
This is important because:
is not merely:
It is:
The relevant unit is the sociotechnical system.
“The AI Made the Decision” Is an Accountability Smell
Imagine something goes wrong.
The operator says:
“The AI told me to.”
The manager says:
“A human approved it.”
The developer says:
“We only built the model.”
The vendor says:
“The customer chose how to deploy it.”
The executive says:
“The technical team validated it.”
Now responsibility exists everywhere and nowhere.
That is accountability diffusion.
A system can have ten humans in the loop and still contain no person who actually owns the transformation.
This is why tracing matters.
There must be some meaningful answer to:
Who was responsible for ensuring this particular kind of action was legitimate?
Not necessarily one person for everything.
But responsibility must be legible.
Responsibility Without Authority Is Unjust
There is a complementary failure.
Suppose the organization tells an employee:
“You are responsible for reviewing AI output.”
But the employee:
cannot inspect the model,
cannot access underlying evidence,
cannot change the workflow,
cannot reject the recommendation without management approval,
and is expected to process one case every twenty seconds.
Then the organization has assigned:
without:
That is not meaningful oversight.
It is liability transfer.
The human becomes a moral shock absorber for a system they do not control.
Authority Without Responsibility Is Also Dangerous
Reverse it.
An AI system can trigger consequential actions.
Nobody owns the resulting decision because:
“The algorithm did it.”
Now:
exists without:
That produces the responsibility gap meaningful-human-control theory is attempting to prevent.
A mature governance architecture should therefore seek alignment:
When those variables separate dramatically, something is wrong.
Human Oversight Must Be Designed Before Failure
Sarter and Woods's work on aviation automation shows why this matters.
They studied mode awareness—whether operators understand the current and future status and behavior of automation.
When operators do not understand what mode the automated system is in, they can experience automation surprises: the machine does something unexpected or fails to do something expected.
Their research treats such failures not merely as pilot mistakes but as breakdowns in human-machine coordination.
That lesson transfers directly to AI agents.
If the system has:
multiple tools,
different permission states,
background actions,
delegated subtasks,
changing modes,
and long execution chains,
then human overseers need to know:
What is it doing?
What has it already done?
What is it about to do?
What authority is active?
What state is the environment in?
Oversight cannot begin only when something goes wrong.
At that point the human may already have lost situational awareness.
Logs Are Not the Same Thing as Awareness
A system may record every event.
Excellent.
But imagine 50,000 lines of logs.
Technically:
Practically:
So auditability requires more than storage.
The information has to support reconstruction.
What was proposed?
What evidence was used?
What policy applied?
What authority was granted?
What tool executed?
What changed?
What happened afterward?
That produces an intelligible chain:
This is the machine implementation of the mediation grammar that has run through the entire series.
AI Should Not Become the Only Witness to Its Own Actions
Suppose an AI system executes an action and then produces a summary saying:
“I performed the authorized operation successfully.”
Is that sufficient evidence?
No.
The actor that caused the transformation should not necessarily be the sole source establishing:
what it did,
that it was authorized,
and that it succeeded.
Otherwise:
The system becomes self-attesting.
Independent records, external verification, or other evidence mechanisms may be necessary where the stakes justify them.
Again:
this is not an AI-specific insight.
Governance has long separated:
actor,
auditor,
authorizer,
and recordkeeper
for precisely this reason.
Human Agency Requires a Real Veto Somewhere
There is a simple test for many consequential AI systems.
Ask:
Where can a legitimate human “no” actually stop the transformation?
Not express concern.
Not add a comment.
Not generate a support ticket.
Stop it.
If:
but:
then human authority does not exist at that boundary.
Perhaps the system is intentionally autonomous and that is legitimate.
But we should describe the architecture accurately.
Do not call ceremonial disagreement oversight.
The Veto Does Not Have to Be at Every Step
Again, meaningful control does not require a human click before every machine action.
A governance boundary can exist upstream:
Then:
This architecture can preserve human agency while allowing highly autonomous operation.
The question becomes whether the boundary is meaningful.
Is the scope explicit?
Can the agent detect it?
Can the authority layer enforce it?
Can the agent circumvent it?
Can the human revise it?
That is governed autonomy.
Escalation Is an Agency-Preservation Mechanism
A mature system should know when the machine's authority ends.
For example:
may permit routine processing.
But:
or:
or:
or:
may produce:
That means the machine is not merely capable of acting.
It is capable of recognizing conditions under which it must stop acting autonomously.
That is a governance capability.
Irreversibility Changes the Required Boundary
Suppose an AI drafts an email.
Low consequence.
A human can edit it.
Now suppose an AI sends the email.
Higher consequence.
Suppose it publishes something publicly.
Higher still.
Suppose it deletes data.
Transfers money.
Revokes someone's access.
Changes infrastructure.
The same reasoning system may be involved.
But:
changes.
Therefore:
should change.
This is proportionality from Parts VI and VIII.
Authority should scale with consequence.
One underappreciated property of mediated systems is the ability to undo.
If an AI recommendation can be reversed easily:
If an AI action is irreversible:
The second requires a stronger authorization boundary.
So governance should ask not merely:
Can the human intervene before action?
but:
Can the human restore the prior state afterward?
That is a machine form of restoration.
The Human Can Become Deskilled
Now return to Part II.
Suppose AI performs:
writing,
diagnosis,
coding,
navigation,
planning,
research,
analysis,
scheduling,
and judgment.
If humans stop practicing those activities:
Then dependence on AI increases.
The technology may still expand total human capability enormously.
There is no contradiction.
Part V already established:
The relevant question is:
Which capabilities must remain human because humans need them to govern the mediator itself?
This is a much harder problem.
We do not need to preserve every manual skill forever.
Nobody argues that accountants must preserve the ability to perform all arithmetic without calculators.
But some competence is required to evaluate whether the machine's transformation is legitimate.
If humans lose that competence, oversight becomes ceremonial.
Oversight Requires Residual Competence
Call it:
A system remains governable only if humans preserve enough (C_R) to:
understand objectives,
recognize anomalies,
evaluate consequences,
challenge outputs,
and intervene intelligently.
The required competence varies by system.
But:
while:
creates a dangerous architecture.
The human remains formally responsible for something they have become cognitively incapable of governing.
AI Can Increase Human Competence Too
The opposite is possible.
Suppose AI explains.
Provides counterarguments.
Surfaces relevant evidence.
Teaches techniques.
Makes expert knowledge accessible.
Allows simulation.
Offers feedback.
Helps users understand why a conclusion follows.
Then:
where:
That is the positive mediation model from the beginning of the series.
AI does not merely perform the task.
It helps the human become more capable.
The design choice matters.
A system can optimize for:
answer delivery
or:
capacity development.
Those are not always the same objective.
Convenience Can Quietly Transfer Agency
This is perhaps the most ordinary path.
Nobody commands the human to surrender judgment.
The machine is simply easier.
Why write it?
The AI can.
Why investigate?
The AI summarized it.
Why compare sources?
The AI ranked them.
Why remember?
The AI remembers.
Why decide?
The AI usually gets it right.
Each individual delegation may be rational.
But collectively:
The process is voluntary.
That does not mean it has no structural consequences.
Agency can be surrendered through convenience as easily as through coercion.
The Goal Is Not Maximum Human Labor
This needs to be stated plainly.
Preserving agency does not mean humans should continue doing work machines perform better.
That would turn human agency into ritual inefficiency.
If an AI can:
calculate more accurately,
search faster,
detect patterns better,
handle repetitive administration,
or operate safely in dangerous environments,
then automating those functions may substantially expand human agency.
The objective is not:
It is:
while preserving human governance over the transformations that matter.
Machines Can Have Operational Agency Without Moral Sovereignty
We should distinguish another pair of concepts.
A machine can be an agent in a technical sense.
It can:
perceive,
select actions,
pursue objectives,
update plans,
use tools,
and affect the environment.
That operational agency does not automatically establish moral or political authority.
We can model:
while:
remains bounded by human authorization.
That distinction lets us use highly autonomous systems without pretending that autonomy alone settles legitimacy.
The Objective Is a Governance Stack
At this point, the ten articles converge.
A legitimate AI mediation architecture might look like:
↓
↓
↓
↓
↓
↓
↓
↓
That structure preserves distinct functions.
The model reasons.
Policy constrains.
Authority is granted.
Execution changes state.
Evidence records the change.
Humans can review.
The system learns or is corrected.
No single mediator necessarily has to possess every role.
That is governance.
The AI Governance Test
We can now make the final diagnostic.
1. What function is actually automated?
Information gathering?
Analysis?
Recommendation?
Decision?
Execution?
Do not answer merely:
AI is involved.
Locate the transformation.
2. What authority does the AI possess?
Read?
Write?
Recommend?
Approve?
Publish?
Transfer?
Delete?
Command?
Authority should be explicit.
3. Where did that authority come from?
User?
Organization?
Law?
Policy?
Administrator?
Credential?
Trace the provenance.
4. What is the authorized scope?
Which resources?
Which actions?
Which identities?
Which environments?
Which consequence levels?
Capability should not silently define scope.
5. Can a human understand what the system is doing?
Not necessarily every mathematical detail.
Enough to exercise the role assigned to them.
6. Does the human possess meaningful authority to disagree?
Can they:
reject,
override,
reverse,
interrupt,
or escalate?
If not, stop calling them the decision-maker.
7. Does the human have enough time to intervene?
Formal authority without temporal opportunity is nominal authority.
8. Does the human retain enough competence to intervene intelligently?
Oversight requires capability.
9. Can the affected person contest the outcome?
The human user of the AI is not always the person whose agency matters.
A hiring model affects applicants.
A credit model affects borrowers.
A medical model affects patients.
A government model affects citizens.
Contestability has to extend to the mediated subject, not only the operator.
10. Can responsibility be traced?
Who authorized the system?
Who defined the policy?
Who deployed it?
Who was responsible for oversight?
Who caused the actual state change?
The answer cannot always be:
the algorithm.
11. Is the execution independently observable?
Can we know what actually happened without relying entirely upon the system that caused it?
12. Can the state be restored?
If the action is reversible, how?
If irreversible, was the authorization boundary correspondingly stronger?
13. Does AI use build or erode human competence?
Where competence erosion is acceptable, say so.
Where competence is necessary for oversight, protect it deliberately.
14. Does the machine escalate uncertainty?
A trustworthy agent should not treat every situation as if it falls comfortably inside its authority.
15. Who remains the principal?
This is the question underneath all the others.
After the system becomes powerful, useful, trusted, integrated, and difficult to replace:
Who is actually directing whom?
The Architecture of Agency-Preserving AI
We can now express the positive model.
Let:
Then legitimate machine mediation requires:
and:
with:
remaining sufficiently connected,
and:
remaining traceable.
Additionally:
where appropriate,
and:
That last inequality may be the most important technical principle in the article.
A Powerful Machine Can Be Governed Precisely Because It Is Not the Sovereign
There is a misconception that strict governance makes an AI system less capable.
Not necessarily.
A brilliant surgeon operates under law.
A powerful executive operates under corporate authority.
A judge operates within jurisdiction.
A pilot operates within aviation rules.
Capability and governance coexist constantly.
The same should be true for machines.
We can build:
without requiring:
Indeed, greater capability usually creates stronger reasons to govern authority carefully.
Imagine two AI systems.
System A can suggest grammar corrections.
System B can:
read corporate systems,
form plans,
invoke software,
communicate externally,
and change persistent state.
Their intelligence might even be identical.
What differs is their authority surface.
So:
should scale not merely with:
but with:
Again, not literal arithmetic.
But directionally, that is the problem.
A highly intelligent system with no executable authority may be less operationally dangerous than a mediocre system with powerful credentials.
Human Agency Is Not Preserved by Pretending Humans Are Better at Everything
There is another mistake worth rejecting.
Humans will sometimes be worse than machines.
Much worse.
At some tasks, human judgment should defer to automated evidence.
Preserving human agency does not mean preserving human epistemic pride.
A person can legitimately decide:
For this class of problem, I authorize the machine's judgment because the evidence shows it performs better than mine.
That is itself an exercise of agency.
The important questions remain:
What exactly was delegated?
Can the delegation be revised?
What happens outside the validated domain?
Who handles exceptions?
Who bears responsibility?
Can affected people contest mistakes?
Agency is compatible with humble delegation.
Delegation Is One of the Things Agency Is For
A theory that requires humans to personally retain every task misunderstands agency.
Agency includes the ability to say:
Do this for me.
The problem begins when:
Do this for me
becomes:
Decide what I should want.
Or:
Act beyond what I authorized.
Or:
Make yourself impossible to challenge.
Or:
Use my dependence to expand your authority.
That was true of every human mediator in this series.
It remains true of machines.
The Machine Is the Perfect Stress Test
Artificial intelligence exposes every weakness in mediation architecture because machines amplify whatever authority structures already exist.
If the organization has unclear authority boundaries, AI accelerates ambiguity.
If accountability is weak, AI diffuses it further.
If employees already rubber-stamp procedure, AI can automate the procedure and leave humans rubber-stamping the automation.
If local knowledge is already ignored, AI can encode the centralized representation and deploy it at massive scale.
If contestability is weak, automated decisions can make opacity nearly instantaneous.
If credentials are poorly separated, AI can turn reasoning errors into world-changing actions.
AI therefore does not merely create governance problems.
It reveals governance problems that were already there.
And AI Can Also Make Governance More Explicit
There is a positive side.
Machines force us to formalize things humans often leave implicit.
What exactly is permitted?
Who authorized it?
What state changed?
Which rule applied?
What evidence exists?
Which actions require escalation?
When does authority expire?
Which actor owns the result?
Human organizations frequently operate through vague social assumptions.
Machines need interfaces.
Permissions.
Policies.
Schemas.
State transitions.
That gives us an opportunity.
The emergence of agentic systems could push governance from:
implicit institutional custom
toward:
explicit authority architecture.
That would be valuable far beyond AI.
The Entire Series Was Always Heading Here
Part I asked:
When does the mediator become the actor?
Part II asked:
When does safety replace competence?
Part III asked:
When does leadership produce followers instead of leaders?
Part IV asked:
When does representation replace the represented?
Part V asked:
When does dependency create discretionary control?
Part VI asked:
What keeps authority attached to its legitimate scope?
Part VII asked:
When does a representation of reality become sovereign over reality?
Part VIII asked:
When does care become control?
Part IX asked:
When does spiritual mediation occupy the space it exists to serve?
And now:
When does artificial intelligence stop expanding human capability and begin becoming the effective locus of human decision?
The answer is structurally familiar.
When information migrates.
Then judgment.
Then authority.
Then execution.
Then competence.
Then responsibility.
Until eventually the human remains somewhere in the diagram—
but no longer governs the transformation.
Governance for Machines, Agency for Humans
The answer is not to freeze artificial intelligence at the level of autocomplete.
Nor is it to insist that every machine action receive direct human approval.
Nor is it to assume autonomous systems are inherently illegitimate.
Machines can possess extensive operational autonomy.
They can reason.
Plan.
Coordinate.
Adapt.
Recommend.
And under properly bounded authority, execute.
The harder principle is this:
Machine autonomy is legitimate when it operates inside an authority structure whose purposes, permissions, boundaries, accountability, and correction mechanisms remain meaningfully governed by human actors.
And:
A human is not meaningfully “in the loop” merely because a workflow contains a human-shaped approval step. Human agency requires enough information, competence, time, and authority to alter the transformation.
That is the difference between:
and:
The machine should be able to become extraordinarily capable without becoming sovereign.
The human should be able to delegate extensively without becoming irrelevant.
The organization should be able to automate without losing accountability.
And the person affected by the system should remain more than a data object moving through someone else's model.
We began this series with:
Ten articles later, we can make the legitimacy question much sharper.
Ask:
Who was the actor?
Where did that authority come from?
Which decisions remained with the actor?
Could its actions be reversed?
Did competence grow or decline?
Did dependency become leverage?
Did the authority remain within scope?
Who became more capable?
Who became more necessary?
And finally:
After the transformation, who is still the subject?
That question works for a worker and an organizer.
A citizen and a representative.
A patient and a physician.
A believer and a pastor.
A person and an institution.
And now:
a human and a machine.
The technology changes.
The grammar remains.
A legitimate mediator may be powerful.
It may know more.
It may act faster.
It may possess specialized authority.
It may remain necessary indefinitely.
It may even, under defined conditions, act without waiting for the human at every step.
But it remains a mediator when its power stays attached to the transformation it was authorized to serve.
The danger begins when capability quietly becomes jurisdiction.
Jurisdiction becomes dependency.
Dependency becomes authority.
And authority ceases returning to the subject from whom the entire system derived its purpose.
The principle that began this series therefore survives its hardest technical case.
But now we can state it more precisely:
Good mediation expands the subject's real capacity while keeping the mediator's authority proportionate, traceable, contestable, and bounded by the purpose that justified it.
Bad mediation does something else.
It solves a limitation by transferring the locus of action until eventually the mediator can say:
“You cannot function without me.”
And perhaps be correct.
The deeper question is whether that condition was necessary—
or whether the architecture helped create it.
For artificial intelligence, that is not merely an ethical question.
It is a systems-design question.
Machines should become more capable.
Humans should remain the principals.
And governance is the architecture that allows both statements to remain true at once.
References
[1] Lisanne Bainbridge. “Ironies of Automation.” Automatica 19, no. 6 (1983): 775–779. DOI: 10.1016/0005-1098(83)90046-8. Classic human-factors analysis of the new difficulties created when automation leaves humans responsible for exceptional conditions and supervisory control.
[2] Raja Parasuraman and Victor Riley. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors 39, no. 2 (1997): 230–253. DOI: 10.1518/001872097778543886. Distinguishes appropriate use, overreliance, underuse, and automation introduced without adequate consideration of human performance.
[3] Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. “A Model for Types and Levels of Human Interaction with Automation.” IEEE Transactions on Systems, Man, and Cybernetics — Part A 30, no. 3 (2000): 286–297. DOI: 10.1109/3468.844354. Distinguishes automation of information acquisition, analysis, decision selection, and action implementation across differing levels of automation.
[4] Nadine B. Sarter and David D. Woods. “How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control.” Human Factors 37, no. 1 (1995): 5–19. DOI: 10.1518/001872095779049516. Shows how advanced automation creates new monitoring and mode-awareness demands for human supervisors.
[5] Nadine B. Sarter and David D. Woods. “Team Play with a Powerful and Independent Agent: Operational Experiences and Automation Surprises on the Airbus A-320.” Human Factors 39, no. 4 (1997): 553–569. DOI: 10.1518/001872097778667997. Examines automation surprises and human-machine coordination failures in highly automated aviation systems.
[6] Raja Parasuraman and Dietrich H. Manzey. “Complacency and Bias in Human Use of Automation: An Attentional Integration.” Human Factors 52, no. 3 (2010): 381–410. DOI: 10.1177/0018720810376055. Reviews empirical research on automation complacency and automation bias, including overreliance among both novice and expert users.
[7] Kate Goddard, Abdul Roudsari, and Jeremy C. Wyatt. “Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators.” Journal of the American Medical Informatics Association 19, no. 1 (2012): 121–127. DOI: 10.1136/amiajnl-2011-000089. Reviews automation-bias evidence and factors affecting overreliance on decision-support systems.
[8] Filippo Santoni de Sio and Jeroen van den Hoven. “Meaningful Human Control over Autonomous Systems: A Philosophical Account.” Frontiers in Robotics and AI 5 (2018): Article 15. DOI: 10.3389/frobt.2018.00015. Develops the tracking and tracing conditions for meaningful human control and responsibility over autonomous systems.
[9] National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0) and associated Playbook, 2023. Provides governance guidance on human-AI roles, oversight, proficiency, accountability, monitoring, appeal, override, and organizational responsibility.
[10] European Union. Regulation (EU) 2024/1689, Artificial Intelligence Act, Article 14, “Human Oversight.” Primary legal source. Requires effective human oversight for covered high-risk systems and specifies capacities including understanding system limitations, awareness of automation bias, interpretation, disregard or reversal of output, and safe intervention or interruption.