AGI outcomes and civilizational competence

||yabo app

大卫维克多

[最新的IPCC]报告称，“如果您建立所有这些技术和国际协议，我们仍然可以在[公正] 2度上停止变暖。”我自己的评估是，你需要做的行动的种类是如此英勇的是，我们不会在这个星球上看到它们。

-David Victor，1UCSD国际关系教授

一段时间后，我参加了来自科学，技术，金融和政治的“搬运工和振动器”会议。我们正在讨论我们最喜欢改善世界的大想法。一个人的大想法是复制国家之间的最佳实践。例如，当它显示国家可以dramatically improve organ donation rates通过使用退出而不是选择选择，其他国家应该只复制该解决方案。

Everyone thought this was a boring suggestion, because it wasobviously一个好主意，没有辩论。当然，他们都同意了它也是impossibleand永远无法成为标准练习。所以我们搬到另一个更具易行的大想法。

后来，在与一群类似的人的会晤时，我告诉一些经济学家，他们对某个问题的建议是“直截了当的econ 101”，我没有任何反对分享。相反，我问道，“但我们如何获得政策制定者来实施ECON 101解决方案？”经济学家笑着说，“好吧，是的，我们不知道。我们可能无法。“

我怎么把它换来？这不是应该与自我改善的AGIS一起使用的文明。2

这反铲is a powerful, labor-saving invention, but I wouldn’t put a two-year-old in the driver’s seat. That’s roughly how I feel about letting 21st century humans wield something as powerful as自我改善AGI。3.I wish we had more time to grow up first. I think the kind of actions we’d need to handle self-improving AGI successfully “are so heroic that we’re not going to see them on this planet,” at least not anytime soon.4.

But I suspect we won’t all resist the temptation to build AGI for long, and neither do most top AI scientists.5.这re’s just too much incentive to build AGI: a self-improving AGI could give its makers — whether Google or the NSA or China or somebody else — history’s greatest first-mover advantage. Even if the first few teams design their AGIs wisely, the passage of time will only make it easier for smaller and less-wise teams to cross the finish line.摩尔的疯狂科学法则那and all that.

有些人比我对自我改善的AGI令人担忧不太担心。毕竟，一个可能在1950年预测，我们不会有文明的能力，以避免未来半个世纪的全能核战争，但我们做过躲开它 （if只要仅仅）。6.所以也许我们不应该这么担心AGI。

虽然我认为重要的考虑这类第二次猜测的论点，但我一般试图在面临的面临世界。当我看看我们成功的东西，以及我们失败的各种事物，从Agi看起来很好much harder而不是我们经常失败的事情，如令人兴奋的方式切换到Opt-Out Out Out Out Offult offorms捐赠的程序。

但我不会假装这个文明能力的问题已经解决。如果可以解决问题，请执行此操作，如果不多，则需要一本书长度参数。（Nick Bostrom'sSuperintelligence关于为什么AGI控制问题很难说很多，但它并没有说出人类是否可能上升到这一挑战。7.的）

试图回答这个问题是什么点？如果我的视图是正确的，我认为结果是我们需要重新评估我们的社会差分投资在全球挑战中。如果你想在NBA中取得成功，但你是只有5'3“高,那么你就不得不投入更多的时间和effort on your basketball goals than you do on other goals for which you’re more naturally suited. And if we want our civilization to survive self-improving AGI, but our civilization can’t even manage to switch to opt-out programs for organ donation, then we’ll just have to start earlier, try harder, spend more, etc. on surviving AGI than we do when pursuing other goals for which our civilization is more naturally suited, like building awesome smartphones.

But if I’m wrong, and our civilization is on course to handle AGI just like it previously handled, say,氟氯化碳那then there may be more urgent things to be doing than亚博体育官网 。(Still, it would be surprising if more Friendly AI work wasn’t good on the展示margin, given that there are fewer than 5 full-time Friendly AI researchers in the world right now.)

我不会在这篇文章中争论自己的观点。相反，我只是想问一下：怎么可能学习这个文明能力和阿加的到来的问题？I’d probably split the analysis into two parts: (1) the apparent shape and difficulty of the AGI control problem, and (2) whether we’re likely to have the civilizational competence to handle a problem of that shape and difficulty when it knocks on our front door.

Note that这篇文章中的一切都是毛重简化。Problem Difficulty and Civilizational Competence aren’t one-dimensional concepts, though to be succinct I sometimes talk as if they are. But a problem like AGI control is difficult to different degrees in different ways, some technical and others political, and different parts of our civilization are differently competent in different ways, and those different kinds of competence are undergoing different trends.

问题的难度

AGI控制问题有多难，并且在哪种方式是难以置信的？为了说明这样的分析看起来像什么，我可能会达到自己的想法，如下所示：

  1. 这control problems that are novel to AGI look really hard. For example, getting good outcomes from self-improving AGI seems to require as-yet unobserved philosophical success — philosophical success that isnot仅仅需要写安全的自动驾驶仪软件。More generally, there seem to be several novel problems that arise when we’re trying to control a system more generally clever and powerful than ourselves — problems we have no track record of solving for other systems, problems which seem analogous to the hopeless prospect of chimpanzees getting humans to reliably do what the chimps want. (SeeSuperintelligence。）
  2. 此外，在我们在AGI能力中达到“无回头”点之前，我们可能会得到相对较少的实体机会来解决这些新的AGI控制问题。特别是（1）稳定武器竞赛between nations and/or companies, incentivizing speed of development over safety of development, seems likely, and (2) progress可能很快当新的控制问题变得相关时。作为一种物种，我们并不是那么擅长在我们的第一个尝试之一获得完全正确的东西 - 相反，我们通常通过学习几十或数百个初始失败来获得一些正确的东西。但我们可能没有这种奢侈品，具有新的AGI控制问题。
  3. 这AGI control challenge looks especially susceptible to a problem known as “位置外部性“ - 武器比赛是一个例子 - 以及相关的协调问题。（我在脚注中解释了“位置外部性”的概念。8.的）

但这只是一个粗略的草图，其他思想家可能具有不同模型的AGI控制问题的形状和难度。

文明竞争力

Second, will our civilization rise to the challenge? Will our civilizational competence at the time of AGI invention be sufficient to solve the AGI control problem?

My own pessimism on this question doesn’t follow from any conceptual argument or any simple extrapolation of current trends. Rather, it comes from the same kind of multi-faceted empirical reasoning that you probably do when you try to think about whether we’re more likely to have, within 30 years, self-driving taxis or a Mars colony. That is, I’m combining different models I have about how the world works in general: the speed of development in space travel vs. AI, trends in spending on both issues, which political and commercial incentives are at play, which kinds of coordination problems must be solved, what experts in the relevant fields seem to think about the issues, what kinds of questions experts are better or worse at answering, etc. I’m also adjusting that initial combined estimate based on some specific facts I know: about赤裸裸的那aboutradiation hazards那about自动车辆， 等等。9.

Unfortunately,因为one’s predictions about AGI outcomes can’t be strongly supported by simple conceptual arguments, it’s a labor-intensive task to try to explain one’s views on the subject, which may explain why I haven’t seen a good, thorough case for任何一个关于AGI结果的乐观或悲观。人们只是有他们的观点那based on what they anticipate about the world, and it takes a lot of work to explain in detail where those views are coming from. Nevertheless, it’d be nice to see someone try.

作为a start, I’ll link to some studies which share some methodological features of the kind of investigation I’m suggesting:

如果您决定执行一些小型此分析项目，请在下面的评论中链接到您的工作。

  1. Quote taken from theRadiolab.第一个名为“在这个星球的尘埃中。“↩
  2. 在Superintelligence那Bostrom made the point this way (p. 259):

    Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct… For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will全部找到危险的东西的意义似乎几乎可以忽略不计，我们可以通过逃跑来达到安全，因为智力爆炸的爆​​炸会带来整个穹苍。也没有长大的视线。

    ↩

  3. 由“美国国际集团”我的意思是一个计算机系统,可以通过亚博体育苹果app官方下载something like Nilsson’s employment test (see什么是AGI？）。通过“自我改善的AGI”，我的意思是通过自己的原始计算机科学和机器人研究来提高自己能力的AGI（并不完全由说，收集更多关于世界或获取更多计算资源的数据）。亚博体育官网“它自己的能力”我的意思是包括AGI本身创造目标的继承系统的能力。亚博体育苹果app官方下载在本文中，我通常是互换的“AGI”和“自我改善AGI”，而不是因为所有AGIS都是强烈的感觉自我改善，而是因为我期望第一个AGIS对某些人没有自我改善原因，自我改善的AGIS将在几十年之后遵循，如果不是更快的话。从宇宙学的角度来看，这样的延迟只是眨眼。↩
  4. 我故意完全没有钉住what关于我们的文明似乎不足以满足AGI控制的挑战;当在气候变化方面发表评论时，David Victor的选择是相同的选择。我认为，由于许多原因，我们的文明竞争力不足以挑战，但我也对每个原因产生不同程度的不确定性以及他们适用的问题，这些细节难以表达。↩
  5. See the AI timeline predictions for the TOP100 poll inMüller＆Bostrom（2014）。作者询问了顶级生活AI科学家的样本：“出于这个问题的目的，假设人类科学活动继续存在严重的负面破坏。在哪一年会看到[AGI]存在的（10％/ 50％/ 90％）存在？“每个置信水平的中位数答复分别为2024,2050和2070。

    Why trust AI scientists at all? Haven’t they been wildly optimistic about AI progress from the beginning? Yes, there are embarrassing quotes from early AI scientists about how fast AI progress would be, but there are also many now-disproven quotes from early AI skeptics about what AI不会能够做到。最早的民意调查of AI scientists we have is从1973年起，以及对该调查关于AGI时间表的问题的最受欢迎的回应是最悲观的选择，“超过50年”。（假设我们没有收到2023年的AGI，最终将是正确的。）↩

  6. An interesting sub-question: Does humanity’s competence keep up with its capability? When our capabilities jump, as they did with the invention of nuclear weapons, does our competence in controlling those capabilities also jump, out of social/moral necessity or some other forces? Einstein said “Nuclear weapons have changed everything, except our modes of thought,” suggesting that he expected us not to mature as “adults” quickly enough to manage nuclear weapons wisely. We haven’t exactly handled them “wisely,” but we’ve at least handled them wisely enough to avoid global nuclear catastrophe so far.↩
  7. WhatSuperintelligencesays on the topic can be found in chapter 14.↩
  8. 弗兰克（1991年）explains the concept this way:

    在Micromotives和Macrobehavior.，Thomas Schelling观察曲棍球运动员，留给自己的设备，几乎从不戴头盔，尽管几乎所有人都会投票给秘密选票中的头盔规则。没有戴头盔增加了获胜的几率，也许是稍微更容易看到和听到......同时，没有戴着头盔会增加受伤的几率。如果玩家重视胜利的速度越高，而不是重视额外的安全，它是理性的，不要戴头盔。讽刺，斯科利观察，就是当所有人丢弃他们的头盔时，竞争平衡就像所有人都磨损一样。

    头盔问题是我们可能称之为的例子位置外部性。这decision to wear a helmet has important effects not only for the person who wears it, but also for the frame of reference in which he and others operate. In such situations, the payoffs to individuals depend in part on their positions within the frame of reference. With hockey players, what counts is not their playing ability in any absolute sense, but how they perform relative to their opponents. Where positional externalities… are present, Schelling has taught us, individually rational behavior often adds up to a result that none would have chosen.

    军备竞赛是一种很好的地位外部性。作为亚历山大（2014）把它放在上帝的视野中，最好的解决方案是世界和平，根本没有任何国家。从系统内部，没有国家可以单方面强亚博体育苹果app官方下载制执行，因此他们的最佳选择是继续将钱投入导弹......“↩

  9. 在other words, my reasons for AGI outcomes pessimism look like a模型组合和调整。Or you can think of it in terms of what Holden Karnofsky calls集群思维。或作为我的早期读者之一，“正常日常推理。”↩

Did you like this post?你可以享受我们的其他yabo app 帖子，包括：