Eliezer Yudkowsky

Eliezer Shlomo Yudkowsky (/ˌɛliˈɛzər jʊdˈkaʊski/ EL-ee-EH-zər yuud-KOW-skee;[1] born September 11, 1979) is an American artificial intelligence researcher[2][3][4][5] and writer on decision theory and ethics, known for popularizing ideas related to friendly artificial intelligence.[6][7] He is the founder of and a research fellow at the Machine Intelligence Research Institute (MIRI), a private research nonprofit based in Berkeley, California.[8] His work on the prospect of a runaway intelligence explosion influenced philosopher Nick Bostrom's 2014 book Superintelligence: Paths, Dangers, Strategies.[9] His books include If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, a New York Times bestseller he co-authored with Nate Soares,[10] as well as the Harry Potter fanfiction Harry Potter and the Methods of Rationality. Yudkowsky's views on the safety challenges future generations of AI systems pose are discussed in Stuart Russell's and Peter Norvig's undergraduate textbook Artificial Intelligence: A Modern Approach. Noting the difficulty of formally specifying general-purpose goals by hand, Russell and Norvig cite Yudkowsky's proposal that autonomous and adaptive systems be designed to learn correct behavior over time: Yudkowsky (2008)[11] goes into more detail about how to design a Friendly AI. He asserts that friendliness (a desire not to harm humans) should be designed in from the start, but that the designers should recognize both that their own designs may be flawed, and that the robot will learn and evolve over time. Thus the challenge is one of mechanism design—to design a mechanism for evolving AI under a system of checks and balances, and to give the systems utility functions that will remain friendly in the face of such changes.[6] In response to the instrumental convergence concern, which implies that autonomous decision-making systems with poorly designed goals would have default incentives to mistreat humans, Yudkowsky and other MI