Success in Friendly AI can have positive consequences that are arbitrarily large, depending on how powerful a Friendly AI is. Failure in Friendly AI has negative consequences that are also arbitrarily large. The farther into the future you look, the larger the consequences (both positive and negative) become. What is at stake in Friendly AI is, simply, the future of humanity.
with such high stakes, taking a cautious approach has an entirely new meaning. for instance the slightest error might result in the emergence of unfriendly ai. knowing that human design capabilities are limited and error prone, how do you design such a system?
what is good?
Thus, Creating Friendly AI uses “volition-based Friendliness” as the assumed model for Friendliness content. Volition-based Friendliness has both a negative aspect – don’t cause involuntary pain, death, alteration, et cetera; try to do something about those things if you see them happening – and a positive aspect: to try and fulfill the requests of sentient entities.
in other words, the only way out is to make sure the ai has an active interest in being friendly.