Naive superintelligence and Pascal's magic beans
I. I just finished an ebook by Stuart Armstrong called Smarter Than Us: The Rise of Machine Intelligence , and it's increasing my skepticism of the singulatarians—those whacky A.I.-vangelists such as inhabit MIRI, the Machine Intelligence Research Institute. They're doing (what they think is) an important public service, by trying to warn the public about the dangers of a fully-godlike, insufficiently-friendly machine intelligence. Except I'm not so convinced that we could stumble on the kind of superintelligence they usually preach in fear about, an alien god with boundless powers: You [the machine superintelligence] are currently having twenty million simultaneous conversations. Your predictive software shows that about five of those you are interacting with show strong signs of violent psychopathic tendencies. You can predict at least two murder sprees, with great certainty, by one of those individuals over the next year. You consider your options. The human poli...