
Too Magnificent to Shut Down
01Scenario
When the Colonel asks whether an Akira-level Tetsuo could be controlled, Onishi answers with more equipment, data, and analysis. After the stop condition has been ignored and Tetsuo is destabilizing, Onishi admits he could not throw away such a magnificent subject.
02AI safety Analogy
Monitoring, understanding, and control are different capabilities. Even a stated shutdown rule is meaningless if the person responsible for enforcing it is rewarded by unprecedented results and can keep the experiment running.
03Where the analogy breaks
The project is an abusive human experiment and its proposed “shutdown” is killing a person; model deactivation is ethically and technically different.
AI Risk families
This scenario is an example of the following types of AI risk.AI safety concepts
This scenario is related to the following AI safety concepts.Source continuation
More from Akira
Taxonomy neighbors
Related scenarios
Selected from other sources by shared risk families and AI safety concepts.

Cyberpunk: Edgerunners2022
Cyberpsychosis Tripwires Ignored
- Risk
- Accidents
- Concept
- Deployment Safety · AI Control · Governance Failure · Runtime Monitoring

A Quiet Place2018
The Baby Box Is Containment With Life Support
- Risk
- Accidents
- Concept
- Deployment Safety · AI Control · AI Welfare · Runtime Monitoring

Ghostbusters1984
Shut Down the Grid
- Risk
- Accidents · Systemic
- Concept
- Deployment Safety · AI Control · Governance Failure


