We will consider later whether it might be possible through deliberate effort to construct a superintelligence that values such things, or to build one that values human welfare, moral goodness, or any other complex purpose its designers might want it to serve. The orthogonality thesis suggests that we cannot blithely assume that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom and intellectual development in humans-scientific curiosity, benevolent concern for others, spiritual enlightenment and contemplation, renunciation of material acquisitiveness, a taste for refined culture or for the simple pleasures in life, humility and selflessness, and so forth. With mature technology, basic resources such as time, space, matter, and free energy could be processed to serve almost any goal. If you looked more closely, studying the morphology of the two brains under a microscope, this impression of fundamental similarity would only be strengthened: You would see the same lamellar organization of the cortex, with the same brain areas, made up of the same types of neuron, soaking in the same bath of neurotransmitters.įirst, the value of resources depends on the uses to which they can be put, which in turn depends on the available technology.

You might even be unable to tell which brain belonged to whom. You would readily recognize them as two of a kind. Imagine their brains lying side by side in quiet repose. Certainly in terms of neural architecture, Ms. If we zoom out and consider the space of all possible minds, however, we must conceive of these two personalities as virtual clones. But this is because our intuitions are calibrated on our experience, which samples from the existing human distribution (and to some extent from fictional personalities constructed by the human imagination for the enjoyment of the human imagination). The personality differences between these two individuals may seem almost maximally large.

Consider two persons who seem extremely unlike, perhaps Hannah Arendt and Benny Hill. In this abstract space, human minds form a tiny cluster. Let us first reflect for a moment on the vastness of the space of possible minds.

Superintelligenz nick bostrom