Cultural Alignment
Still from Akira illustrating Too Magnificent to Shut Down

Too Magnificent to Shut Down

When the Colonel asks whether an Akira-level Tetsuo could be controlled, Onishi answers with more equipment, data, and analysis. After the stop condition has been ignored and Tetsuo is destabilizing, Onishi admits he could not throw away such a magnificent subject.

Monitoring, understanding, and control are different capabilities. Even a stated shutdown rule is meaningless if the person responsible for enforcing it is rewarded by unprecedented results and can keep the experiment running.

The project is an abusive human experiment and its proposed “shutdown” is killing a person; model deactivation is ethically and technically different.

AI Risk families

This scenario is an example of the following types of AI risk.
  1. Accidents
  2. Systemic

AI safety concepts

This scenario is related to the following AI safety concepts.
  1. Deployment Safety
  2. AI Control
  3. AI Welfare
  4. Governance Failure
  5. Runtime Monitoring