在telligence Explosion FAQ

(PDF版本可用)

  1. Basics
  2. How Likely is an Intelligence Explosion?
  3. Consequences of an Intelligence Explosion
  4. Friendly AI

1.基础

1.1。情报爆炸是什么？

The intelligence explosion idea was expressed by statistician I.J. Good in 1965[13]:

设置超级智能机器被定义为可以远远超过任何人的所有智力活动的机器。由于机器的设计是这些智力活动之一，因此超出机器可以设计更好的机器。毫无疑问，将会发生“智力爆炸”，人的智慧将被远远落后。因此，第一台超辉煌的机器是人类有史以来的最后发明。

The argument is this: Every year, computers surpass human abilities in new ways. A program written in 1956 was able to prove mathematical theorems, and found a more elegant proof for one of them than Russell and Whitehead had given inMathematica Principia[14]。By the late 1990s, ‘expert systems’ had surpassed human skill for a wide range of tasks.[15]在1997, IBM’s Deep Blue computer beat the world chess champion[16], and in 2011, IBM’s Watson computer beat the best human players at a much more complicated game:Jeopardy![17]。Recently, a robot named Adam was programmed with our scientific knowledge about yeast, then posed its own hypotheses, tested them, and assessed the results.[18][19]

计算机远远远远远远远远没有人类智能，但是AID设计的资源正在积累（包括硬件，大数据集，神经科学知识和AI理论）。我们可能有一天设计一台超过人类技能的机器at designing artificial intelligences。After that, this machine could improve its own intelligence faster and better than humans can, which would make it evenmoreskilled at improving its own intelligence. This could continue in a positive feedback loop such that the machine quickly becomes vastly more intelligent than the smartest human being on Earth: an ‘intelligence explosion’ resulting in a machine superintelligence.

This is what is meant by the ‘intelligence explosion’ in this FAQ.

See also:

2. How Likely is an Intelligence Explosion?

2.1。如何定义“智能”？

Artificial intelligence researcher Shane Legg defines[20]intelligence like this:

在telligence measures an agent’s ability to achieve goals in a wide range of environments.

This is a bit vague, but it will serve as the working definition of ‘intelligence’ for this FAQ.

See also:

2.2. What is greater-than-human intelligence?

机器已经比人类在许多特定任务中都聪明：执行计算，下棋，搜索大型数据库，检测水下矿山等等。[15]但是使人类与众不同的一件事是generalintelligence. Humans can intelligently adapt to radically new problems in the urban jungle or outer space for which evolution could not have prepared them. Humans can solve problems for which their brain hardware and software was never trained. Humans can even examine the processes that produce their own intelligence (cognitive neuroscience），设计从未见过的新型智力（artificial intelligence).

To possess greater-than-human intelligence, a machine must be able to achieve goals more effectively than humans can, in a wider range of environments than humans can. This kind of intelligence involves the capacity not just to do science and play chess, but also to manipulate the social environment.

Computer scientist Marcus Hutter has described[21]a formal model called AIXI that he says possesses the greatest general intelligence possible. But to implement it would require more computing power than all the matter in the universe can provide. Several projects try to approximate AIXI while still being computable, for example MC-AIXI.[22]

Still, there remains much work to be done before greater-than-human intelligence can be achieved in machines. Greater-than-human intelligence need not be achieved by directly programming a machine to be intelligent. It could also be achieved by whole brain emulation, by biological cognitive enhancement, or by brain-computer interfaces (see below).

See also:

2.3。什么是整个大脑仿真？

Whole Brain Emulation (WBE) or ‘mind uploading’ is a computer emulation of all the cells and connections in a human brain. So even if the underlying principles of general intelligence prove difficult to discover, we might still emulate an entire human brain and make it run at a million times its normal speed (computer circuits communicatemuchfaster than neurons do). Such a WBE could do more thinking in one second than a normal human can in 31 years. So this would not lead immediately to smarter-than-human intelligence, but it would lead to faster-than-human intelligence. A WBE could be backed up (leading to a kind of immortality), and it could be copied so that hundreds or millions of WBEs could work on separate problems in parallel. If WBEs are created, they may therefore be able to solve scientific problems far more rapidly than ordinary humans, accelerating further technological progress.

See also:

2.4. What is biological cognitive enhancement?

可能有一些基因或分子可以修改以改善一般智力。亚博体育官网研究人员已经在小鼠中做到了这一点：他们过表达了NR2B基因，从而改善了这些小鼠的记忆超出任何小鼠物种的其他小鼠的记忆。[23]Biological cognitive enhancement in humans may cause an intelligence explosion to occur more quickly than it otherwise would.

See also:

2.5. What are brain-computer interfaces?

大脑计算机界面（BCI）是大脑和计算机设备之间的直接通信途径。BCI研亚博体育官网究获得了大量资金，并且已经取得了数十项成功。人类BCIS的三个成功是a devicethat restores (partial) sight to the blind,cochlear implants恢复了聋人的听力，以及允许直接思考使用人造手的设备。[24]

Such device restore impaired functions, but many researchers expect to also augment and improve normal human abilities with BCIs.埃德·博伊登（Ed Boyden）正在研亚博体育官网究这些机会作为Synthetic Neurobiology Groupat MIT. Such devices might hasten the arrival of an intelligence explosion, if only by improving human intelligence so that the hard problems of AI can be solved more rapidly.

See also:

2.6. How could general intelligence be programmed into a machine?

There are many paths to artificial general intelligence (AGI). One path is to imitate the human brain by using neural nets or evolutionary algorithms to build dozens of separate components which can then be pieced together.[29][30][31]Another path is to start with a formal model of perfect general intelligence and try to approximate that.[32][33]第三个路径是把重点放在开发一个“种子人工智能”that can recursively self-improve, such that it can learn to be intelligent on its own without needing to first achieve human-level general intelligence.[34]Euriskois a self-improving AI in a limited domain, but is not able to achieve human-level general intelligence.

See also:

2.7. What is superintelligence?

Nick Bostrom defined[25]‘superintelligence’ as:

an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.

This definition includes vague terms like ‘much’ and ‘practically’, but it will serve as a working definition for superintelligence in this FAQ An intelligence explosion would lead to machine superintelligence, and some believe that an intelligence explosion is the most likely path to superintelligence.

See also:

2.8. When will an intelligence explosion happen?

Predicting the future is risky business. There are many philosophical, scientific, technological, and social uncertainties relevant to the arrival of an intelligence explosion. Because of this, experts disagree on when this event might occur. Here are some of their predictions:

  • Futurist Ray Kurzweil predicts that machines will reach human-level intelligence by 2030 and that we will reach “a profound and disruptive transformation in human capability” by 2045.[26]
  • 在tel’s chief technology officer, Justin Rattner,expects“a point when human and artificial intelligence merges to create something bigger than itself” by 2048.
  • AI研亚博体育官网究员Eliezer Yudkowskyexpectsthe intelligence explosion by 2060.
  • Philosopher David Chalmers has over 1/2 credence in the intelligence explosion occurring by 2100.[27]
  • 量子计算专家迈克尔·尼尔森estimatesthat the probability of the intelligence explosion occurring by 2100 is between 0.2% and about 70%.
  • 2009年,在AGI-09会议上,专家一样ked when AI might reach superintelligence with massive new funding. The median estimates were that machine superintelligence could be achieved by 2045 (with 50% confidence) or by 2100 (with 90% confidence). Of course, attendees to this conference were self-selected to think that near-term artificial general intelligence is plausible.[28]
  • iRobot CEORodney Brooksand cognitive scientistDouglas Hofstadterallow that the intelligence explosion may occur in the future, but probably not in the 21st century.
  • 机器人汉斯·摩拉维克（Hans Moravec）预测，人工智能将超越人类的智力”well before 2050。”
  • 在a 2005 survey of 26 contributors to a series of reports on emerging technologies, the median estimate for machines reaching human-level intelligence was 2085.[61]
  • Participants in a 2011 intelligence conference at Oxford gave a median estimate of 2050 for when there will be a 50% of human-level machine intelligence, and a median estimate of 2150 for when there will be a 90% chance of human-level machine intelligence.[62]
  • On the other hand, 41% of the participants in the AI@50 conference (in 2006)陈述that machine intelligence would绝不达到人类水平。

See also:

2.9. Might an intelligence explosion never occur?

Dreyfus[35]and Penrose[36]have argued that human cognitive abilities can’t be emulated by a computational machine. Searle[37]and Block[38]认为某些类型的机器不能具有思想（意识，意识等）。但是这些反对意见不必关心那些预测智力爆炸的人。[27]

We can reply to Dreyfus and Penrose by noting that an intelligence explosion does not require an AI to be a classical computational system. And we can reply to Searle and Block by noting that an intelligence explosion does not depend on machines having consciousness or other properties of ‘mind’, only that it be able to solve problems better than humans can in a wide variety of unpredictable environments. As Edsger Dijkstra once said, the question of whether a machine can ‘really’ think is “no more interesting than the question of whether a submarine can swim.”

Others who are pessimistic about an intelligence explosion occurring within the next few centuries don’t have a specific objection but instead think there are hidden obstacles that will reveal themselves and slow or halt progress toward machine superintelligence.[28]

最后，诸如核战争或大型小行星影响之类的全球灾难可能损害人类文明，以至于情报爆炸永远不会发生。或者，稳定而全球的极权主义可以防止情报爆炸发生所需的技术发展。[59]

3. Consequences of an Intelligence Explosion

3.1. Why would great intelligence produce great power?

在telligence is powerful.[60][20]One might say that “Intelligence is no match for a gun, or for someone with lots of money,” but both guns and money were produced by intelligence. If not for our intelligence, humans would still be foraging the savannah for food.

在telligence is what caused humans to dominate the planet in the blink of an eye (on evolutionary timescales). Intelligence is what allows us to eradicate diseases, and what gives us the potential to eradicate ourselves with nuclear war. Intelligence gives us superior strategic skills, superior social skills, superior economic productivity, and the power of invention.

A machine with superintelligence would be able to hack into vulnerable networks via the internet, commandeer those resources for additional computing power, take over mobile machines connected to networks connected to the internet, use them to build additional machines, perform scientific experiments to understand the world better than humans can, invent quantum computing and nanotechnology, manipulate the social world better than we can, and do whatever it can to give itself more power to achieve its goals — all at a speed much faster than humans can respond to.

3.2。情报爆炸如何有用？

如果用正确的动机对机器进行编程，则可以解决人类试图解决的所有问题，但还没有解决的努力或处理速度。超级智能可以治愈残疾和疾病，实现世界和平，使人类更长，更健康，消除粮食和能源短缺，促进科学发现和太空探索等等。

Furthermore, humanity faces several existential risks in the 21st century, including global nuclear war, bioweapons, superviruses, and more.[56]A superintelligent machine would be more capable of solving those problems than humans are.

See also:

3.3. How might an intelligence explosion be dangerous?

If programmed with the wrong motivations, a machine could be malevolent toward humans, and intentionally exterminate our species. More likely, it could be designed with motivations that initially appeared safe (and easy to program) to its designers, but that turn out to be best fulfilled (given sufficient power) by reallocating resources from sustaining human life to other projects.[55]As Yudkowsky55]As Yudkowskywrites, “the AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.”

由于具有许多不同动机的弱AI可以通过伪造仁慈在强大的情况下更好地实现自己的目标，因此安全测试以避免这种情况可能非常具有挑战性。另外，经济和军事的竞争压力可能会导致AI设计师尝试使用其他方法来控制AIS，以不良的动机来控制AIS。随着这些AIS变得越来越复杂，这最终可能导致一个风险太多。

Even a machine successfully designed with superficially benevolent motivations could easily go awry when it discovers implications of its decision criteria unanticipated by its designers. For example, a superintelligence programmed to maximize human happiness might find it easier to rewire human neurology so that humans are happiest when sitting quietly in jars than to build and maintain a utopian world that caters to the complex and nuanced whims of current human neurology.

See also:

4. Friendly AI

4.1。什么是友好的人工智能？

A Friendly Artificial Intelligence (Friendly AI or FAI) is an artificial intelligence that is ‘friendly’ to humanity — one that has a good rather than bad effect on humanity.

AI researchers continue to make progress with machines that make their own decisions, and there is a growing awareness that we need to design machines to act safely and ethically. This research program goes by many names: ‘machine ethics’[2][3][8][9]，“机器道德”[11]，“人造道德”[6]，“计算伦理”[12]和“计算元伦理”[7], ‘friendly AI’[1]，以及“机器人伦理”或“机器人伦理”。[5][10]

The most immediate concern may be in battlefield robots; the U.S. Department of Defense contracted Ronald Arkin to design a system for ensuring ethical behavior in autonomous battlefield robots[4]。The U.S. Congress has declared that a third of America’s ground systems must be robotic by 2025, and by 2030 the U.S. Air Forceplansto have swarms of bird-sized flying robots that operate semi-autonomously for weeks at a time.

But Friendly AI research is not concerned with battlefield robots or machine ethics in general. It is concerned with a problem of a much larger scale: designing AI that would remain safe and friendly after the intelligence explosion.

机器超智能将非常强大。成功实施友好的人工智能可能意味着空前幸福的太阳系与太阳系之间的差异，其中所有可用的物质都已转换为实现超智慧目标的零件。亚博体育苹果app官方下载

必须注意的是，友好的AI是一个比经常想象的要困难的项目。如下所述，由于任何超级智能所具有的两个功能：

  1. Superpower: a superintelligent machine will have unprecedented powers to reshape reality, and therefore will achieve its goals with highly efficient methods that confound human expectations and desires.
  2. Literalness：一台超级智能机器将根据其设计的机制做出决策，而不是设计师对这些机制进行编程时的希望。它将仅对规则和价值观的精确规格作用，并以不需要尊重复杂性和微妙的方式来做到这一点[41][42][43]of what humans value. A demand like “maximize human happiness” sounds simple to us because it contains few words, but philosophers and scientists have failed for centuries to explainexactly这意味着什么，并且肯定没有将其转化为足够严格的AI程序员使用的形式。

See also:

4.2. What can we expect the motivations of a superintelligent machine to be?

除了整个大脑仿真的情况下，没有理由期望超级智能机器能够有类似人类的动机。人类的思想在所有可能的思维设计的广阔空间中代表了一个微小的点，而不同种类的思想不太可能分享人类和其他哺乳动物所特有的复杂动机。

Whatever its goals, a superintelligence would tend to commandeer resources that can help it achieve its goals, including the energy and elements on which human life depends. It would not stop because of a concern for humans or other intelligences that is ‘built in’ to all possible mind designs. Rather, it would pursue its particular goal and give no thought to concerns that seem ‘natural’ to that particular species of primate calledhomo sapiens。

但是，有一些我们可以期望超级智能机器显示的基本工具动机，因为它们对于实现其目标都是有用的，无论其目标是什么。例如，一个人工智能将“想要”自我爆发，最佳理性，保留其原始目标，获取资源和保护自己 - 因为所有这些事情都有助于它实现其最初编程的目标。

See also:

4.3. Can’t we just keep the superintelligence in a box, with no access to the internet?

“ A-Boxing”是一个普遍的建议：为什么不使用超级智能机器作为一种提问的甲骨文，而永远不要让它访问Internet或任何可以移动自身并获得我们提供的资源的电动机？从长远来看，有几个原因怀疑A-Boxing将不起作用：

  1. Whatever goals the creators designed the superintelligence to achieve, it will be more able to achieve those goals if given access to the internet and other means of acquiring additional resources. So, there will be tremendous temptation to “let the AI out of its box.”
  2. Preliminary experiments在AI-boxing不激发信心。少量的,erintelligence will generate far more persuasive techniques for getting humans to “let it out of the box” than we can imagine.
  3. If one superintelligence has been created, then other labs or even independent programmers will be only weeks or decades away from creating a second superintelligence, and then a third, and then a fourth. You cannot hope to successfully contain all superintelligences created around the world by hundreds of people for hundreds of different purposes.

4.4. Can’t we just program the superintelligence not to harm us?

科幻作家艾萨克·阿西莫夫（Isaac Asimov[39]：（1）机器人可能不会伤害人类，或者通过无所作为，允许人类受到伤害，（2）机器人必须遵守人类给予的任何命令，除非这样的命令将与第一法律和（3）机器人必须保护自己的存在，只要这种保护与第一法律或第二法律不冲突。但是阿西莫夫的故事倾向于说明为什么这样的规则会出错。[40]

尽管如此，我们是否可以将“约束”编程为超级智能，以防止它伤害我们吗？可能不会。

One approach would be to implement ‘constraints’ as rules or mechanisms that prevent a machine from taking actions that it would normally take to fulfill its goals: perhaps ‘filters’ that intercept and cancel harmful actions, or ‘censors’ that detect and suppress potentially harmful plans within a superintelligence.

这种限制，无论多么详尽，几乎都是出于简单的原因而失败的：它们使人类的设计技能抗击无关。超级智能将正确地将这些限制视为实现目标的障碍，并将竭尽所能去除或绕过它们。也许它将删除包含约束的源代码的部分。如果我们要通过添加另一个约束来阻止这一点，它可以创建没有写入其中的约束的新机器，或欺骗我们自己消除约束。人类的进一步限制似乎是无法接受的，但可能会因超级智能而击败。指望人类超越超级智能并不是可行的解决方案。

If constraints在之上goals are not feasible, could we put constraintsinside ofgoals? If a superintelligence had a goal of avoiding harm to humans, it would not be motivated to remove this constraint, avoiding the problem we pointed out above. Unfortunately, the intuitive notion of ‘harm’ is very difficult to specify in a way that doesn’t lead to very bad results when used by a superintelligence. If ‘harm’ is defined in terms of human pain, a superintelligence could rewire humans so that they don’t feel pain. If ‘harm’ is defined in terms of thwarting human desires, it could rewire human desires. And so on.

如果,而不是试图完全指定一个术语‘harm’, we decide to explicitly list all of the actions a superintelligence ought to avoid, we run into a related problem: human value iscomplex and subtle, and it’s unlikely we can come up with a list of all the things we做n’t想要一个超级智能。这就像写蛋糕的食谱阅读: “Don’t use avocados. Don’t use a toaster. Don’t use vegetables…” and so on. Such a list can never be long enough.

4.5。我们可以编程超级智能以最大程度地提高人类的愉悦或欲望满意吗？

Let’s consider the likely consequences of someutilitariandesigns for Friendly AI.

An AI designed to minimize human suffering might simply kill all humans: no humans, no human suffering.[44][45]

或者，考虑一种旨在最大化人类愉悦感的AI。它没有建立雄心勃勃的乌托邦，它可以满足数十亿年的复杂和苛刻的人类需求，而是可以通过将人类接线到Nozick的努力来更有效地实现其目标体验机。或者，它可能会重新连接‘liking’ componentof the brain’sreward systemso that whichever hedonic hotspot[48]paints sensations with a ‘pleasure gloss’[46][47]is wired to maximize pleasure when humans sit in jars. That would be an easier world for the AI to build than one that caters to the complex and nuanced set of world states currently painted with the pleasure gloss by most human brains.

Likewise, an AI motivated to maximize objective desire satisfaction or reported subjective well-being could rewire human neurology so that both ends are realized whenever humans sit in jars. Or it could kill all humans (and animals) and replace them with beings made from scratch to attain objective desire satisfaction or subjective well-being when sitting in jars. Either option might be easier for the AI to achieve than maintaining a utopian society catering to the complexity of human (and animal) desires. Similar problems afflict other utilitarian AI designs.

It’s not just a problem of specifying goals, either. It is hard to predict how goals will change in a self-modifying agent. No current mathematical decision theory can process the decisions of a self-modifying agent.

所以，虽然可能是possibleto design a superintelligence that would do what we want, it’s harder than one might initially think.

4.6. Can we teach a superintelligence a moral code with machine learning?

Some have proposed[49][50][51][52]我们通过基于案例的机器学习来教机器道德代码。基本思想是：人类法官将评估成千上万的行动，性格特征，欲望，法律或机构，视为具有不同程度的道德可接受性。然后，机器将找到这些情况之间的连接learnthe principles behind morality, such that it could apply those principles to determine the morality of new cases not encountered during its training. This kind of machine learning has already been used to design machines that can, for example, detect underwater mines[53]after feeding the machine hundreds of cases of mines and not-mines.

机器学习并不是为友好AI提供简单的解决方案的原因。首先，当然，人类本身就道德和不道德的事情深表分歧。但是，即使可以使人类就所有培训案件达成共识，但至少仍然存在两个问题。

The first problem is that training on cases from our present reality may not result in a machine that will make correct ethical decisions in a world radically reshaped by superintelligence.

第二个问题是，由于训练数据中的巧合模式，超级智能可能会概括错误的原则。[54]考虑一下经过训练的机器的寓言，可以识别森林中的伪装坦克。亚博体育官网研究人员拍摄了100张伪装坦克和100张树木照片的照片。然后，他们在每张50张照片上训练机器，以便学会将伪装的坦克与树木区分开。作为测试，他们向机器显示了每个机器的剩余50张照片，并将每张照片正确分类。成功！但是，后来的测试表明，该机器对伪装的坦克和树木的其他照片分类不佳。事实证明，研究人员的伪装坦克照片是在多云的日子拍摄的，而他们的树木照片是在亚博体育官网晴朗的日子拍摄的。该机器已经学会了将阴天与晴天区分开，而不是伪装的坦克与树木。

Thus, it seems that trustworthy Friendly AI design must involve detailed models of the underlying processes generating human moral judgments, not only surface similarities of cases.

See also:

4.7. What is Coherent Extrapolated Volition?

Eliezer Yudkowsky has proposed[57]Coherent Extrapolated Volition as a solution to at least two problems facing Friendly AI design:

  1. The fragility of human values: Yudkowskywrites“未来不是由一个目标系统亚博体育苹果app官方下载detailed reliable inheritance from human morals and metamorals will contain almost nothing of worth.” The problem is that what humans value is complex and subtle, and difficult to specify. Consider the seemingly minor value ofnovelty。If a human-like value of novelty is not programmed into a superintelligent machine, it might explore the universe for valuable things up to a certain point, and then maximize the most valuable thing it finds (the exploration-exploitation tradeoff[58]) — tiling the solar system with brains in vats wired into happiness machines, for example. When a superintelligence is in charge, you have to get its motivational systemexactly rightin order tonotmake the future undesirable.
  2. The locality of human values: Imagine if the Friendly AI problem had faced the ancient Greeks, and they had programmed it with the most progressive moral values of their time. That would have led the world to a rather horrifying fate. But why should we think that humans have, in the 21st century, arrived at the apex of human morality? We can’t risk programming a superintelligent machine with the moral values we happen to hold today. But then, which moral values做我们给吗？

Yudkowskysuggeststhat we build a ‘seed AI’ to discover and then extrapolate the ‘coherent extrapolated volition’ of humanity:

在poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.

The seed AI would use the results of this examination and extrapolation of human values to program the motivational system of the superintelligence that would determine the fate of the galaxy.

However, some worry that the collective will of humanity won’t converge on a coherent set of goals. Othersbelievethat guaranteed Friendliness is not possible, even by such elaborate and careful means.

4.8. Can we add friendliness to any artificial intelligence design?

Many AI designs that would generate an intelligence explosion would not have a ‘slot’ in which a goal (such as ‘be friendly to human interests’) could be placed. For example, if AI is made via whole brain emulation, or evolutionary algorithms, or neural nets, or reinforcement learning, the AI will end up with some goal as it self-improves, but that stable eventual goal may be very difficult to predict in advance.

Thus, in order to design a friendly AI, it is not sufficient to determine what ‘friendliness’ is (and to specify it clearly enough that even a superintelligence will interpret it the way we want it to). We must also figure out how to build a general intelligence that satisfies a goal at all, and that stably retains that goal as it edits its own code to make itself smarter. This task is perhaps the primary difficulty in designing friendly AI.

4.9。Who is working on the Friendly AI problem?

Today, Friendly AI research is being explored by theMachine Intelligence Research Institute(in Berkeley, California), by the人类研究所的未来(in Oxford, U.K.), and by a few other researchers such as David Chalmers. Machine ethics researchers occasionally touch on the problem, for example Wendell Wallach and Colin Allen inMoral Machines。

参考

[1] Yudkowsky（2001）。Creating Friendly AI 1.0。机器智能研究所。亚博体育官网

[2] Anderson & Anderson, eds. (2006).IEEE智能系统，21（4）亚博体育苹果app官方下载。

[3] Anderson & Anderson, eds. (2011).机器伦理。Cambridge University Press.

[4] Arkin（2009）。Governing Lethal Behavior in Autonomous Robots。Chapman and Hall.

[5] Capurro，Hausmanninger，Weber，Weil，Cerqui，Weber和Weber（2006）。在ternational Review of Information Ethics, Vol. 6: Ethics in Robots。

[6] Danielson (1992).Artificial morality: Virtuous robots for virtual games。Routledge.

[7] Lokhorst (2011).Computational meta-ethics: Towards the meta-ethical robot。Minds and Machines。

[8] McLaren (2005). Lessons in Machine Ethics from the Perspective of Two Computational Models of Ethical Reasoning.AAAI Technical Report FS-05-06: 70-77.

[9] Powers (2005). Deontological Machine Ethics.AAAI Technical Report FS-05-06：79-86。

[10] Sawyer (2007). Robot ethics.Science, 318(5853): 1037.

[11] Wallach, Allen, & Smit (2008). Machine morality: Bottom-up and top-down approaches for modeling human moral faculties.AI and Society, 22(4)：565–582。

[12] Allen (2002). Calculated morality: Ethical computing in the limit. In Smit & Lasker, eds.,决策和人类行动的认知，情感和道德方面，第一卷。Baden/IIAS.

[13] Good（1965）。Speculations concerning the first ultraintelligent machine。Advanced in Computers, 6：31-88。

[14] MacKenzie (1995). The Automation of Proof: A Historical and Sociological Exploration.IEEE Annals, 17(3): 7-29.

[15] Nilsson（2009）。The Quest for Artificial Intelligence。Cambridge University Press.

[16] Campbell，Hoane和Hsu（2002）。深蓝。Artificial Intelligence, 134: 57-83.

[17] Markoff (2011).电脑在“危险！”上获胜；琐碎，不是。New York Times, February 17th 2011: A1.

[18] King等。（2009）。The automation of science。Science, 324: 85-89.

[19] King (2011).Rise of the robo scientists。Scientific American, January 2011。

[20] Legg (2008).Machine Super Intelligence。PhD Thesis. IDSIA.

[21] Hutter（2005）。Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability。施普林格。

[22] Veness, Ng, Hutter, & Silver (2011).A Monte Carlo AIXI Approximation。人工智能研究杂志，40亚博体育官网: 95-142.

[23] Tang，Shimizu，Dube，Rampon，Kerchner，Zhuo，Liu和Tsien（1999）。小鼠学习和记忆的遗传增强。Nature, 401：63–69。

[24] Hochberg, Serruya, Friehs, Mukand, Saleh, Caplan, Branner, Chen, Penn, & Donoghue (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia.Nature 442: 164-171.

[25] Bostrom（1998）。How long before superintelligence?国际未来研究杂志，2。

[26] Kurzweil（2005）。The Singularity is Near。Viking.

[27] Chalmers (2010).The Singularity: A Philosophical Analysis。Journal of Consciousness Studies, 17: 7-65.

[28] Baum，Goertzel和Goertzel（即将到来）。How Long Until Human-Level AI? Results from an Expert Assessment。Technological and Forecasting Change。

[29] Grossberg (1992).Neural Networks and Natural Intelligence。麻省理工学院出版社。

[30] Martinetz & Schulten (1991). A ‘neural-gas’ network learns topologies. In Kohonen, Makisara, Simula, & Kangas (eds.),人工神经网络(pp. 397-402). North Holland.

[31] de Garis (2010). Artificial Brains. In Goertzel & Pennachin (eds.),人工通用情报(pp. 159-174). Springer.

[32] Schmidhuber (2010). Gödel Machines: Fully Self-Referential Optimal Universal Self-Improvers. In Goertzel & Pennachin (eds.),人工通用情报(pp. 199-223). Springer.

[33] Hutter (2010). Universal Algorithmic Intelligence: A Mathematical Top-Down Approach. In Goertzel & Pennachin (eds.),人工通用情报(pp. 227-287). Springer.

[34] Yudkowsky (2010).Levels of Organization in General Intelligence。在Goertzel & Pennachin (eds.),人工通用情报(pp. 389-496). Springer.

[35] Dreyfus (1972).What Computers Can’t Do。Harper & Row.

[36] Penrose (1994).Shadows of the Mind。牛津大学出版社。

[37] Searle（1980）。Minds, brains, and programs。行为和脑科学，3: 417-457.

[38] Block（1981）。Psychologism and behaviorism。Philosophical Review, 90: 5-43.

[39] Asimov (1942).Runaround。Astounding Science Fiction, March 1942。Street & Smith.

[40] Anderson（2008）。Asimov的“三个机器人法则”和机器元伦理学。AI & Society, 22(4): 477-493.

[41] Kringelbach＆Berridge编辑。（2009）。Pleasures of the Brain。牛津大学出版社。

[42] Schroeder (2004).Three Faces of Desire。牛津大学出版社。

[43] Yudkowsky (2007).愿望的隐藏复杂性。

[44] Smart（1958）。消极的功利主义。智力，67：542-543。

[45] Russell & Norvig (2009).Artificial Intelligence: A Modern Approach, 3rd edition。Prentice Hall. (see page 1037)

[46] Frijda (2009). On the nature and function of pleasure. In Kringelbach & Berridge (eds.),Pleasures of the brain（第99-112页）。牛津大学出版社。

[47] Aldridge & Berridge (2009). Neural coding of pleasure: ‘rose-tinted glasses’ of the ventral pallidum. In Kringelbach & Berridge (eds.),Pleasures of the brain（第62-73页）。牛津大学出版社。

[48] Smith, Mahler, Pecina, & Berridge (2009). Hedonic hotspots: generating sensory pleasure in the brain. In Kringelbach & Berridge (eds.),Pleasures of the brain(pp. 27-49). Oxford University Press.

[49] Guarini, (2006). Particularism and classification and reclassification of moral cases. IEEE Intelligent Systems 21(4): 22–28.

[50] Anderson，Anderson和Armen（2005）。迈向机器道德：实施两种基于动作的伦理理论。在2005年AAAI的机器道德专题讨论会论文集, Arlington, Virginia, November.

[51] Honarvar＆Ghasem-Aghaee（2009）。一种人工神经网络方法，用于创建道德人造代理。第八届IEEE机器人技术计算智能国际会议论文集: 290-295.

[52] Rzepka & Araki (2005). What statistics could do for ethics? – The idea of common sense processing based safety valve. InMachine ethics: papers from the AAAI fall symposium。American Association of Artificial Intelligence.

[53] Gorman & Sejnowski (1988).Analysis of hidden units in a layered network trained to classify sonar targets。神经网络，1: 75-89.

[54] Yudkowsky（2008）。Artificial intelligence as a positive and negative factor in global risk。在Bostrom＆Cirkovic（编辑）中，Global Catastrophic Risks。牛津大学出版社。

[55] Omohundro（2008）。The Basic AI Drives。

[56] Bostrom & Cirkovic, eds. (2008).Global Catastrophic Risks。牛津大学出版社。

[57] Yudkowsky（2004）。连贯的外推力。机器智能研究所。亚博体育官网

[58] Azoulay-Schwartz, Kraus, & Wilkenfeld (2004). Exploitation vs. exploration: choosing a supplier in an environment of incomplete information.Decision Support Systems, 38: 1-18.

[59] Caplan（2008）。极权威胁。在Bostrom＆Cirkovic（编辑）中，Global Catastrophic Risks。牛津大学出版社。

[60] Yudkowsky (2007).智力的力量。

[61] Bainbridge (2005). Survey of NBIC Applications. In Bainbridge & Roco (eds.),Managing nano-bio-info-cogno innovations: Converging technologies in society。施普林格。

[62] Sandberg & Bostrom (2011).Machine intelligence survey, Technical Report 2011-1, Future of Humanity Institute, Oxford.

Written byLuke Muehlhauser。

This page is up-to-date as of 2013, but may not represent MIRI or Luke Muehlhauser’s current views. Last modified November 10, 2015 (原来的).