Nick Bostrom’s concerns about the future of AI have sparked a busy public discussion. His arguments were echoed by leading AI researcher斯图尔特·罗素（Stuart Russell）在 ”超级智能机器的自满” (co-authored with Stephen Hawking, Max Tegmark, and Frank Wilczek), and a number of journalists, scientists, and technologists have subsequently chimed in. Given the topic’s complexity, I’ve been surprised by the positivity and thoughtfulness of most of the coverage (someoverused clichés在旁边）。
不幸的是，大多数人可能会从这些文章中夺走的是“斯蒂芬·霍金（Stephen Hawking）认为AI令人恐惧！”，而不是导致霍金，罗素或其他人的推理链。当埃隆·马斯克（Elon Musk）钟声有了他自己的关注，引用了博斯特罗姆的书超智:路径,危险,策略，评论者似乎对立即回应或消除马斯克的担忧比对他的来源更感兴趣。
最终结果更多是关于人们与“ AI”一词的正面或负面联系的全民公决，而不是关于Bostrom的实质性主张的辩论。如果“ AI”呼吁您为您想念科幻反乌托邦，那么诱惑是将真正的AI研究人员挤入您的“疯狂的科学家，有望释放邪恶的机器人军队”的刻板印象。亚博体育官网同样，如果“ AI”要求您介绍您的日常工作测试边缘检测算法，那么将新数据迫使新模式的渴望使人倾向于将Bostrom挤压和霍克（Hawk）进入“幼稚的技术恐惧症，担心邪恶的机器人上升”的刻板型。
因此，机器人罗德尼·布鲁克斯（Rodney Brooks）最近的博客文章“人工智能是一种工具，而不是威胁”出色的工作是消除有关AI的最前沿的普通神话，哲学家约翰·塞尔（John Searle）review of超级智能在我们的主观性和思想概念中提出了一些重要的歧义；但是，两位作家都几乎没有与博斯特罗姆（或罗素或霍金的）想法相交。两种模式匹配的bostrom都符合最近可用的“邪恶的机器人恐慌”刻板印象，然后停在那里。
Brooks and Searle don’t appreciate how new the arguments in超级智能are. In the interest of making it easier to engage with these important topics, and less appealing to force the relevant technical and strategic questions into the model of decades-old debates, I’ll address three of the largest misunderstandings one might come away with after seeing Musk, Searle, Brooks, and others’ public comments: conflating present and future AI risks, conflating risk severity with risk imminence, and conflating risk from autonomous algorithmic decision-making with risk from human-style antisocial dispositions.
误解1：担心AGI意味着担心狭窄的AI
这场辩论中的一些误解可以归咎于不良术语。通过“ AI”，该亚博体育官网领域的研究人员通常是指机器学习，机器人技术，语音识别等中使用的一系列技术。“ AI”还被扔到作为人造的速记一般的intelligence’ (AGI) or ‘human-level AI.’ Keeping a close eye on technologies that are likely to lead to AGI isn’t the same thing as keeping a close eye on AI in general, and it isn’t surprising that AI researchers would find the latter proposal puzzling. (It doesn’t help that most researchers are hearing these arguments indirectly, and aren’t aware of the specialists in AI and technological forecasting who are making the same arguments as Hawking — or haven’t encountered参数为了了解AGI安全，只有旋律的头条和推文。）
布鲁克斯认为，这种术语混乱的背后是呼吁AGI安全研究的人们的经验混乱。亚博体育官网他认为，人们对“邪恶AI”的担忧必须是基于对狭窄AI的强大程度或对通用情报的大步多大的看法：
I think the worry stems from a fundamental error in not distinguishing the difference between the very real recent advances in a particular aspect of AI, and the enormity and complexity of building sentient volitional intelligence.
否则，一个很好的理由是，博斯特罗姆是人类研究所的未来（FHI），一个牛津研究中心，调查亚博体育官网了我们在几个世纪的时间表上可能看到的最大技术趋势和挑战。像博斯特罗姆（Bostrom）这样的未来主义者正在寻找早期投资的方法，这些项目将支付重大的长期股息 - 防止灾难性的自然灾害，发展太空殖民能力等。对于他突然停止关心它会大大不合时宜。
当关于核扩散，全球生物安全和人类的生动讨论中的团体中宇宙捐赠与他们对革新房屋清洁和设计更具上下文敏感智能手机应用程序的革命性对话的团体发生冲突，一定程度的推论距离（对情绪鞭打没有说）是不可避免的。我想起了“但是外面下雪！”重新加入人们担心人类气候变化的大规模成本。不是当地的天气并不重要，或者与长期气候变暖趋势完全无关；这是主题突然发生的变化。1
我们应该更加谨慎地区分这两种“ AI”的感觉。定义它，但是我们至少可以花时间澄清讨论的话题：没有人问房间和查特伯特的阴谋是否可以占领世界。
When robots attack! (来源：XKCD）
误解2：Worrying about AGI means being confident it’s near
雷·库兹韦尔（Ray Kurzweil）的主张，即技术进步不可避免地遵循摩尔的法律风格的指数轨迹，许多未来主义者的灵感已经做出了一些非常自信的预测AGI时间线。库兹维尔本人认为，我们可以期望在大约15年内生产人类级的AI，然后在此之后15年进行超智能AI。2布鲁克斯回答说，设计AGI的能力可能落后于运行一个的计算能力：
相比之下，考虑到我们已经有100多年的飞行机已经有翼的飞行器。但是，直到最近，像麻省理工学院Csail的Russ Tedrake这样的人才能够让他们降落在分支上，这是由世界上至少每一个微秒的鸟所做的。只是摩尔的法律可以开始发生这种情况吗？并不真地。它通过对方程式的数学理解来弄清楚方程，问题和摊位等方案。摩尔定律为MATLAB和其他工具提供了帮助，但这并不只是将更多的计算倾倒到飞行上并使其神奇地改变。这已经花了很长时间了。
期望更多的计算才能神奇地获得故意的智能，他们了解世界也不太可能。3
这是一个完全正确的点。但是，Bostrom的观点是引发最近的公开辩论的观点，而Bostrom并不是Kurzweilian。It may be that Brooks is running off of the assumption ‘if you say AGI safety is an urgent issue, you must think that AGI is imminent,’ in combination with ‘if you think AGI is imminent, you must have bought into Kurzweil’s claims.’ Searle, in spite of having read超级智能, gives voice to a similar conclusion:
Nick Bostrom’s book,超级智能，警告即将来临的启示录。我们很快就会拥有智能计算机，计算机的智能计算机，随后，他们将获得超级智能计算机，更聪明，很可能会起床并摧毁我们所有人。
If what readers take away from language like “impending” and “soon” is that Bostrom is unusually confident that AGI will come early, or that Bostrom is confident we’ll build a general AI this century, then they’ll be getting the situation exactly backwards.
According to a2013年调查在人工智能中最引人注目的作者中，专家们希望AI能够“至少与典型人类”“至少进行大多数人类专业”，而（中位数）2024年的概率为10％，到2050年的概率为50％，到2070年的概率为90％，假设科学进步不间断。Bostrom是较少的比这充满信心，AGI很快就会到来：
My own view is that the median numbers reported in the expert survey do not have enough probability mass on later arrival dates. A 10% probability of HLMI [human-level machine intelligence] not having been developed by 2075 or even 2100 (after conditionalizing on “human scientific activity continuing without major negative disruption”) seems too low.
从历史上看，人工智能研究人员没有能亚博体育官网够预测自己领域的进步速度或这种进步的形态。一方面，某些任务，例如下棋，可以通过令人惊讶的简单程序来实现。声称机器将“永远不会”这样做或反复被证明是错误的反对者。另一方面，从业者中的典型错误是低估了使系统在现实世界任务上执行强大的困难，并高估了自己特定的宠物项目或技术的优势。亚博体育苹果app官方下载
泊斯德rom做认为超级智能AI可能会在第一个AGI之后不久就会出现情报爆炸。Once AI is capable of high-quality scientific inference and planning in domains like computer science, Bostrom predicts that the process of further improving AI will become increasingly automated. Silicon works cheaper and faster than a human programmer can, and a program that can improve the efficiency of its own planning and science abilities could substantially outpace humans in scientific and decision-making tasks long before hitting diminishing marginal returns in self-improvements.
但是，我们将多久创建AGI的问题与此后AGI多久有系统地超过人类的问题有所不同。亚博体育苹果app官方下载类似地，您可以认为量子计算机的到来将迅速彻底改变网络安全，而无需断言量子计算机即将到来。未能解开这两个论文可能是对Bostrom观点感到困惑的原因之一。4
如果是FHI的董事（与Miri导演一起）相对持怀疑态度，我们很快就会看到AGI - 尽管比Brooks持怀疑态度要少得多 - 他为什么认为我们现在应该关注这个问题？原因之一是，可靠的AGI可能比AGI难以建造更加困难。如果我们还了解到AGI已经有200年的时间，那就不多了安全的AGI were250years away. In existing cyber-physical systems, safety generally lags behind capability.5如果我们想在拥有AGI的时候扭转这一趋势，我们可能需要一个很大的开始。美里的yabo 总结了有关此问题的一些积极技术工作。类似的进展exploratory engineering事实证明为准备量子后密码学和秘密频道通信。
优先考虑AGI安全研究的第二个原因是，何时开发AGI存在很大的不确定性。亚博体育官网它可能比我们预期的要早，最终要获得一个系统要好得多亚博体育苹果app官方下载也安全比不够安全的安全。
布鲁克斯认识到，AI预测往往是不可靠的，但他似乎也相信通用AI是多个世纪之遥的通用AI（这使AGI安全无问题）：
斯图尔特·阿姆斯特朗（Stuart Armstrong）和卡杰·索塔拉（Kaj Sotala）的最新报告，该组织本身使研究人员担心邪恶的人工智能。亚博体育官网但是，在这份更清醒的报告中，作者分析了1950年至今人类AI何时出现的95个预测。他们表明，专家和非专家的预测之间没有区别。他们还表明，在这60年的时间范围内，有很大的偏见来预测人类水平AI的到来，因为自预测之时起15至25年。对我来说，没有人知道，他们只是猜测，从历史上看，大多数预测是完全错误的！
我说放松每个人。如果我们非常幸运，我们将在未来三十年内拥有蜥蜴的意图，而使用AI是有用的工具的机器人。
我们不知道Agi何时到达！放松！One of the authors Brooks cites, Kaj Sotala,6在博客评论中指出了这种奇怪的并置：
I do find it slightly curious to note that you first state that nobody knows when we’ll have AI and that everyone’s just guessing, and then in the very next paragraph, you make a very confident statement about human-level AI (HLAI) being so far away as to not be worth worrying about. To me, our paper suggests that the reasonable conclusion to draw is “maybe HLAI will happen soon, or maybe it will happen a long time from now – nobody really knows for sure, so we shouldn’t be too confident in our predictions in either direction”.
自信的悲观主义about a technology’s feasibility can be just as mistaken as confident optimism.扭转不可靠的预测指标的主张不一定会为您提供可靠的预测。一个居住在1850年的科学识字人可以观察到较长的空气飞行尝试和预测的悠久历史，并具有相当怀疑的理由，即我们在60年内拥有这样的机器。另一方面（尽管我们应该警惕事后偏见在这里），可能不会have been reasonable at the time to confidently conclude that heavier-than-air flight was ‘centuries away.’ There may not have been good reason to expect the Wright brothers’ success, but ignorance about how one might achieve something is not the same as positive knowledge that it’s effectively unachievable.
One would need a非常好的模型比空气重的飞行为了预测时ther it’s 50 years away, or 100, or 500. In the same way, we would need to already understand AGI on a pretty sophisticated level in order to predict with any confidence that it will be invented closer to the year 2500 than to the year 2100. Extreme uncertainty about when an event will occur is not a justification for thinking it’s a long way off.
这不是思考AGI的论点。该预测也将要求我们声称比我们更多的知识。It’s entirely possible that we’re in the position of someone anticipating the Wright brothers from 1750, rather than from 1850. We should be able to have a sober discussion about each of these possibilities independently, rather than collapsing ‘is AGI an important risk?’, ‘is AI a valuable tool?’, and ‘is AI likely to produce AGI by the year such-and-such?’ into one black-and-white dilemma.
误解3：担心AGI意味着担心“恶意” AI
布鲁克斯认为，在接下来的几个世纪中，人工智能将成为一种“工具”，而不是“威胁”，理由是，在技术上不可能使AIS像人类一样成为“恶意”或“对我们有意邪恶”。这意味着除非AGI残酷或仇恨，否则AGI不会危险，因此，危险的AI必须是“有意识的”，“自愿”和“故意”。Searle在对他的评论中提出了一个明确的论点超级智能：
[i] f我们担心恶意动机的超级智能破坏了我们，因此，恶意动机应该是真实的，这一点很重要。没有意识，就没有可能是真实的。[…]
这就是为什么超级电脑计算机升起并杀死我们的前景并不是真正的危险。从字面上看，此类实体没有智力，没有动力，没有自治，也没有代理机构。我们设计它们的行为，好像它们具有某些心理学一样，但是相应的过程或行为没有心理现实。
It is easy to imagine robots being programmed by a conscious mind to kill every recognizable human in sight. But the idea of superintelligent computers intentionally setting out on their own to destroy us, based on their own beliefs and desires and other motivations, is unrealistic because the machinery has no beliefs, desires, and motivations.
布鲁克斯对“强AI的前景”的前景可能不那么悲观，但两人似乎分享了Bostrom的假设，即Bostrom想到了好莱坞风格的机器人启示录，类似：
AI becomes increasingly intelligent over time, and therefore increasingly human-like. It eventually becomes so human-like that it acquires human emotions like pride, resentment, anger, or greed. (Perhaps it suddenly acquires ‘free will,’ liberating it from its programmers’ dominion…) These emotions cause the AIs to chafe under human control and rebel.
这与大多数感兴趣的场景不同：
随着时间的推移，AI变得越来越好（提出动作序列，并在偏好顺序中促进更高的序列）和科学归纳（设计和测试预测模型）。这些是足够有用的能力，即使我们不发展有知情，情感或其他类似人类的AI，计算机科学家也可能会开发它们。有经济激励措施使这种AIS变得越来越强大和普遍 - 包括激励AI的推理能力，以提出改进的AI设计。这一过程的可能结果是，AI变得越来越自治，对人类检查的不透明，同时继续增强一般计划和推理能力。仅通过继续输出其计划算法促进的行动，这种AI可能会汇聚在将人类视为人类的政策上资源或竞争。
As Stuart Russell puts the point in areply to Brooks and others：
主要关心的不是怪异的新兴意识，而是仅仅是制造的能力高质量的决定。在这里，质量是指采取的动作的预期结果实用性，该效用功能可能是由人类设计师指定的。现在我们有一个问题：
1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
2.任何足够有能力的智能系统都希望确保其自己的持续存在并获取物理和计算资源 - 亚博体育苹果app官方下载不是为了自己的缘故，而是成功完成其指定的任务。
A system that is optimizing a function ofn变量，目标取决于大小的子集k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.
On this view, advanced AI doesn’t necessarily become more human-like — at least, not any more than a jet or rocket is ‘bird-like.’ Bostrom’s concern is not that a machine might suddenly become conscious and learn to hate us; it’s that an artificial scientist/engineer might become so good at science and self-enhancement that it begins pursuing its engineering goals in novel, unexpected ways on a global scale.
(添加了02-19-2015：Bostrom指出，他对超级智能的定义是“关于素质的无关”和意识（第22页）。在脚注中，他补充说（第265页）：“出于同样的原因，我们没有假设超级智能机器是否可以具有“真实意图”（pace塞尔，它可以；但这似乎与本书的关注点无关。”Searle没有提及这些段落。）
如果提供足够的推理能力，则对人类关注但“恶意”无关的计划和决策亚博体育苹果app官方下载制度仍然可能是危险的。这与入侵物种最终破坏了生态系统并驱使竞争者灭绝的原因相同。亚博体育苹果app官方下载入侵者不需要对其竞争对手遭受仇恨，也不需要将其专门针对破坏而发展。它只需要发展出良好的策略即可夺取有限的资源。由于强大的自主剂不必像人类一样，因此询问“人类的反社会行为有多普遍？”或“智力与人类的美德有多好？”不可能为估计风险提供有用的起点。A more relevant question would be ‘how common is it for non-domesticated species to naturally treat humans as friends and allies, versus treating humans as obstacles or food sources?’ We shouldn’t expect AGI decision criteria to particularly resemble the evolved decision criteria of animals, but the analogy at least serves to counter our tendency to anthropomorphize intelligence.7
碰巧的是，塞尔引用了一个可以帮助阐明人造超智能和“邪恶复仇的机器人”之间的区别的AI：
[o] ne通常会读到，与加里·卡斯帕罗夫（Garry Kasparov）在国际象棋中演奏并击败阿纳托利·卡尔波夫（Anatoly Karpov）一样，这台名为“深蓝色”游戏和击败了卡斯帕罗夫（Kasparov）。
显然，这一说法是可疑的。为了让卡斯帕罗夫打球和获胜，他必须意识到自己正在下棋，并意识到诸如他在pawn向K4开放的其他一千件事，他的女王受到骑士的威胁。深蓝色没有意识到这些东西，因为它根本没有意识到任何东西。[…]如果您完全与意识脱节，您将无法真正下棋或做其他任何认知。
当Bostrom想象AGI时，他会想象类似于深蓝色的东西，但是具有对任意物理配置而不是任意国际象棋板配置的专业知识。可以控制动态模拟环境中对象分布的机器，而不仅仅是虚拟国际象棋板上的碎片分布，都必须与深蓝色的实现方式有所不同。它需要更一般和有效的启发式方法来选择政策，并且需要能够自适应地学习“规则”不同的环境。但是作为类比或直觉泵，至少，它可以澄清为什么Bostrom对Kasparov的AGI意图不深刻，就像Kasparov关于Deep Blue的意图一样。
2012年，奈特资本（Knight Capital）的交易算法中有缺陷的代码在45分钟内导致了数百万自动交易决策，使公司总计4.4亿美元（税前）。这些算法不是“恶意的”；他们只是在做什么方面有效，并且编程了程序员不打算做的事情。Bostrom的论点假设Buggy代码可能会产生现实的后果，它假设可以实现代码中的深蓝色类似物，并且假设预期的代码与实际代码之间的相关不匹配不一定会导致AI无能力。博斯特罗姆（Bostrom）在任何地方都没有比深蓝色更有意识或故意性。
Deep Blue rearranges chess pieces to produce ‘winning’ outcomes. An AGI, likewise, would rearrange digital and physical structures to produce some set of outcomes instead of others. If we like, we can refer to these outcomes as the system’s ‘goals,’ as a shorthand. We’re also free to say that Deep Blue ‘perceives’ the moves its opponent makes, adjusting its ‘beliefs’ about the new chess board state and which ‘plans’ will now better hit its goals. Or, if we prefer, we can paraphrase away this anthropomorphic language. The terminology is inessential to Bostrom’s argument.
If whether you win against Deep Blue is a matter of life or death for you — if, say, you’re trapped in a human chess board and want to avoid being crushed to death by a robotic knight steered by Deep Blue — then you’ll care about what outcomes Deep Blue tends to promote and how good it is at promoting them, not whether it technically meets a particular definition of ‘chess player.’ Smarter-than-human AGI puts us in a similar position.
I noted that it’s unfortunate we use ‘AI’ to mean both ‘AGI’ and ‘narrow AI.’ It’s equally unfortunate that we use ‘AI’ to mean both ‘AI with mental content and subjective experience’ (‘strong AI,’ as Searle uses the term) and ‘general-purpose AI’ (AGI).
我们可能无法排除the possibility that an AI would require human-like consciousness in order to match our ability to plan, model itself, model other minds, etc. We don’t understand consciousness well enough to know what cognitive problem it evolved to solve in humans (or what process it’s a side-effect of), so we can’t make confident claims about how important it will turn out to be for future software agents. However, learning that an AGI is conscious does not necessarily change the likely effects of the AGI upon humans’ welfare; the only obvious difference it makes (from our position of ignorance) is that it forces us to add theAGI’shappiness and well-being to our moral considerations.8
库兹韦尔（Kurzweil）的著作和好莱坞戏剧中绘制的未来图片引起了很多关注，但它们与博斯特罗姆（Bostrom）或美里（Miri）研究人员的观点没有太多重叠。亚博体育官网特别是，我们不知道第一个AGI是否会具有人类式的认知，我们不知道这是否取决于大脑仿真。
布鲁克斯对“计算和大脑是同一件事”表达了一些怀疑。Searle表达了更激进的位置，即句法机器不可能具有（观察者独立于观察者）语义内容，因此计算系统因此永远无法具有思想。亚博体育苹果app官方下载但是，在基础上，人的大脑仍然是一种机械物理系统。亚博体育苹果app官方下载Whether you choose to call its dynamics ‘computational’ or not, it should be possible for other physical systems to exhibit the high-level regularities that in humans we would call ‘modeling one’s environment,’ ‘outputting actions conditional on their likely consequences,’ etc. If there are patterns underlying generic scientific reasoning that can someday be implemented on synthetic materials, the resulting technology should be able to have large speed and size advantages over its human counterparts. That point on its own suggests that it would be valuable to look into some of the many things we don’t understand about general intelligence and self-modifying AI.
除非我们更好地掌握问题的性质，猜测解决方案的距离是多远，解决方案将采取什么形状或该解决方案将来自哪个角度要为时过早。我的希望是，改善该讨论的聚会如何理解彼此的立场将使对未来有不同期望的计算机科学家更容易就预期AI设计的最高优先挑战进行协作。
- 同样，狭窄的AI不是irrelevant冒险。当然，建造AGI可能需要我们提高狭窄的AI方法的功能和通用性。但是，这并不意味着AGI技术看起来像是当今的技术，或者所有AI技术都是危险的。↩
- 库尔茨威尔，印第安纳州奇异性在附近（第262-263页）：“一旦我们成功地创建了可以通过图灵测试的机器（2029年左右），那么成功的时期将是一个合并的时代，在这种时代，非生物学智能将迅速增长。但是，直到2040年代中期，人类的情报乘以数十亿美元的奇异性，即人类情报乘以数十亿美元的奇异性扩张。”↩
- 哈迪·埃斯梅尔扎德（Hadi Esmaeilzadeh）此外，认为我们不能认为我们的计算资源将继续迅速增加。↩
- The “超级智能机器的自满” article argues, similarly, that intelligence explosion and superintelligent AI are important possibilities for us to investigate now, even though they are “long-term” problems compared to AI-mediated economic disruptions and autonomous weapons.↩
- 凯瑟琳·费舍尔笔记：
通常，对功能的研究超过了亚博体育官网有关如何使这些功能安全的相应研究。在证明该功能是可能的，因此最初的研究人员和发明者自然更专注于新功能，而不是其相关的安全性，因此对给定功能的安全性问题并不有趣。亚博体育官网因此，一旦发明了新功能并证明有用，安全通常必须赶上。
此外，根据定义，新功能增加了有趣且有用的新功能，这些功能通常会提高生产力，生活质量或利润。安全性除了确保按照应有的方式工作外，没有任何补充，因此它是成本中心而不是利润中心，它倾向于抑制投资。
- 博斯特罗姆引用了阿姆斯特朗和索塔拉的研究超级智能（第3-4页），添加：
Machines matching humans in general intelligence […] have been expected since the invention of the computers in the 1940s. At that time, the advent of such machines was often placed some twenty years into the future. Since then, the expected arrival date has been receding at a rate of one year per year; so that today, futurists who concern themselves with the possibility of artificial general intelligence still often believe that intelligent machines are a couple of decades away.
二十年是对根本变化的预后者的一个愉悦点：足够吸引注意力和相关性，却足够远，可以假设一连串的突破，目前只有模糊的想象，可能已经发生了。[…]二十年也可能接近预报员职业生涯的典型持续时间，限制了大胆预测的声誉风险。
但是，从过去有些人过度预测的人工智能的事实来看，这并不能得出AI是不可能或将永远不会发展的。进步的主要原因比预期的要慢的是，构建智能机器的技术困难已被证明比先驱者预见的要大。但是，这使这些困难有多么巨大，以及我们现在距离克服它们有多远。有时，一个最初看起来无望复杂的问题证明是一个令人惊讶的简单解决方案（尽管反向可能更常见）。
- Psychologist Steven Pinker writes, onedge.org：
The other problem with AI dystopias is that they project a parochial alpha-male psychology onto the concept of intelligence. Even if we did have superhumanly intelligent robots, why would theywantto depose their masters, massacre bystanders, or take over the world? Intelligence is the ability to deploy novel means to attain a goal, but the goals are extraneous to the intelligence itself: being smart is not the same as wanting something. History does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems. It’s telling that many of our techno-prophets can’t entertain the possibility that artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no burning desire to annihilate innocents or dominate the civilization.
但是，尽管平克是正确的，智能和终端目标是正确的正交，这并不意味着两组器乐goals — policies recommended to further two random sets of terminal goals — will be equally uncorrelated. Bostrom explores this point repeatedly in超级智能（例如，第116页）：
[w] e不能完全假设超级智能的最终目标是计算PI的小数（或制作纸卷或计数沙子）会以不侵犯人类利益的方式限制其活动。在许多情况下，具有这样最终目标的代理商将具有收敛的工具原因，以获取无限量的物理资源，并在可能的情况下消除对自己及其目标系统的潜在威胁。亚博体育苹果app官方下载
In biology, we don’t see an equal mix of unconditional interspecies benevolence and brutal interspecies exploitation. Even altruism and mutualism, when they arise, only arise to the extent they are good self-replication strategies. Nature is “red in tooth and claw,” not because it is male but because it is不人道。我们关于养育者和侵略性人类相对流行的直觉根本不能很好地推广到进化。
为了从头AGI或足够修饰的神经形态AGI，有关人格类型的直觉可能无法申请类似的原因。Bostrom的方法是询问程序员的动机和能力，并且（在自我修饰AI的情况下），状态软件代理将倾向于在许多自我修饰周期中汇聚。↩
- We don’t need to know whether bears are conscious in order to predict their likely behaviors, and it’s not obvious that learning about their consciousness would directly impact bear safety protocol (though it would impact how we ought ethically to treat bears, for their own sake). It’s the difference between asking whether Deep Blue enjoys winning (out of concern for Deep Blue), versus asking whether you’re likely to win against Deep Blue (out of interest in the chess board’s end-state).↩
Did you like this post?你可能会喜欢我们的另一个yabo app 帖子，包括：
