John Horgan interviews Eliezer Yudkowsky

John Horgan：When someone at a party asks what you do, what do you tell her?

Eliezer Yudkowsky：Depending on the venue: “I’m a decision theorist”, or “I’m a cofounder of the Machine Intelligence Research Institute”, or if it wasn’t that kind of party, I’d talk about my fiction.

John：What’s your favorite AI film and why?

Eliezer：一个I in film is universally awful.Ex Machina与现实的询问一样，与此规则的例外一样。

John：Is college overrated?

Eliezer：如果大学是underrated,考虑到社会赞许性偏见的支持college. So far as I know, there’s no reason to disbelieve the economists who say that college has mostly become a positional good, and that previous efforts to increase the volume of student loans just increased the cost of college and the burden of graduate debt.

John：Why do you write fiction?

Eliezer：To paraphrase Wondermark, “Well, first I tried not making it, but then that didn’t work.”

beyond that, nonfiction conveys knowledge and fiction conveys经验。如果您想了解一个proof of Bayes’s Rule，我可以使用图。如果我要你感觉what it is to use Bayesian reasoning, I have to write a story in which some character is doing that.

John：一个re you religious in any way?

Eliezer：No. When you make a mistake, you need to avoid the temptation to go defensive, try to find some way in which you were a little right, look for a silver lining in the cloud. It’s much wiser to just say “Oops”, admit you were not even a little right, swallow the whole bitter pill in one gulp, and get on with your life. That’s the attitude humanity should take toward religion.

John：如果您是世界之王，那么您的“要做”名单将是什么？

Eliezer：I once observed, “The libertarian test is whether, imagining that you’ve gained power, your first thought is of the laws you would pass, or the laws you would repeal.” I’m not an absolute libertarian, since not everything I want would be about repealing laws and softening constraints. But when I think of a case like this, I imagine trying to get the world to a condition where some unemployed person can offer to drive you to work for 20 minutes, be paid five dollars, and then nothing else bad happens to them. They don’t have their unemployment insurance phased out, have to register for a business license, lose their Medicare, be audited, have their lawyer certify compliance with OSHA rules, or whatever. They just have an added $5.

I’d try to get to the point where employing somebody was once again as easy as it was in 1900. I think it can make sense nowadays to have some safety nets, but I’d try to construct every safety net such that it didn’t disincent or add paperwork to that simple event where a person becomes part of the economy again.

我会尽力做聪明的经济学家已经大喊一段时间的所有事情，但几乎没有国家从未做到。用消费税和土地价值税代替投资税和所得税。用负工资税代替最低工资。机构NGDP级别针对中央银行的制度，并让太大的失败挂起。需要专利法中的失败者付费，并将版权恢复到28年。消除住房建设的障碍。从新加坡的医疗保健设置中复制和粘贴。从爱沙尼亚的电子政务设置中复制和粘贴。尝试用特定的个人决策者替换委员会，并详细说明其决策者的决定将被公开记录和责任。运行不同政府设置的对照试验，并实际上注意结果。 I could go on for literally hours.

从2000万年的角度来看，这一切可能直接至关重要。但是，当我试图弄清楚对人工智能做什么时，由此产生的经济繁荣产生的善意可能会稳定我的政府。我想，显而易见的事情将是曼哈顿在一个岛上的一个项目，与顶级对冲基金具有竞争力，在这里人们可以合作研究人工通用情报问题的某些部分，而无需自动使我们更接近我们的工作，亚博体育官网世界末日。我们仍在为未知的截止日期努力，那时我不会感到放松。除非我们假设我具有真正的神奇力量或完全不可动摇的政权，否则我看不出我可以合理的法律如何在很长时间内延迟AI时间表的任何法律在已经无处不在的地球上很长时间。

全部of this is an impossible thought experiment in the first place, and I see roughly zero hope of it ever coming to pass in real life.

John：贝叶斯定理有什么好处？

Eliezer：For one thing, Bayes’s Theorem is incredibly deep. So it’s not easy to give a brief answer to that.

I might answer that Bayes’s Theorem is a kind of Second Law of Thermodynamics for cognition. If you obtain a well-calibrated posterior belief that some proposition is 99% probable, whether that proposition is milk being available at the supermarket or global warming being anthropogenic, then you must have processed some combination of sufficiently good priors and sufficiently strong evidence. That’s not a normative demand, it’s a law. In the same way that a car can’t run without dissipating entropy, you simply don’t get an accurate map of the world without a process that has Bayesian structure buried somewhere inside it, even if the process doesn’t explicitly represent probabilities or likelihood ratios. You had strong-enough evidence and a good-enough prior or you wouldn’t have gotten there.

在个人层面上，我认为贝叶斯必须为我们提供的主要灵感只是一个事实是规则，那里是铁的法律控制着一种思维方式是否有效地绘制现实。摩门教徒被告知，他们会通过在心中感受到燃烧的感觉来了解摩尔门经的真实性。让我们保守地将《摩尔门经》的先前概率设定为一十亿美元（反对）。然后，我们询问假​​设摩尔门经书是错误的，有人在被告知期待一个人后会感到内心燃烧的感觉。如果您了解贝叶斯的规则，您可以立即看到证据的不可能与试图提出的假设的不可能相称。您甚至不必构成数字就可以看到这些数字并没有加起来 - 正如菲利普·泰洛克（Philip Tetlock）在对超级孔子的研究中发现的那样，超级孔子经常知道贝叶斯的规则，但很少构成特定的概率。在某种程度上，如果您只是在肠道上意识到，很难被愚弄that there is math, that there is一些您要做的数学要获得证据的确切优势，以及是否足以提高假设的先前不可能。您不能仅仅组成东西，并相信自己想相信的东西，因为那行不通。

John:贝叶斯脑假说给你留下深刻印象？

Eliezer：我认为这场辩论中的一些人可能正在互相交谈。询问大脑是否是贝叶斯算法，就像问本田雅阁是否在卡诺热发动机上运行。If you have one person who’s trying to say, “Every car is a thermodynamic process that requires fuel and dissipates waste heat” and the person on the other end hears, “If you draw a diagram of a Carnot heat engine and show it to a mechanic, they should agree that it looks like the inside of a Honda Accord” then you are going to have some fireworks.

当有些人打开内燃机并找到气缸时，他们也会感到非常兴奋，然后说：“我敢打赌，这会将热量转化为压力并有助于向前推动汽车！”他们会说对了，但是随后您会发现其他人说：“您专注于更大的汽车零件库中的一个组件；催化转化器也很重要，并且在您的Carnot热发动机图中没有出现任何地方。为什么，有时我们会运行空调，这与您说的热引擎的工作方式完全相反。”

我认为，我认为采取卓越态度并说：“您显然不熟悉现代汽车维修的人们，这并不令人惊讶。您需要一个不同方法的工具箱来构建汽车引擎，例如火花塞和催化转换器，而不仅仅是这些热力学过程您一直在谈论“缺少一个关键的抽象水平。

但是，如果您想知道大脑是否literally贝叶斯引擎，而不是做认知工作，我们可以以贝叶斯的方式理解其性质，然后我的猜测是“哎呀，不”。该发动机中可能有一些令人兴奋的贝叶斯气缸，但其中更多的看起来像奇怪的临时安全带和空调。None of which is going to change the fact that to correctly identify an apple based on sensory evidence, you need to do something that’s ultimately interpretable as resting on an inductive prior that can learn the apple concept, and updating on evidence that distinguishes apples from nonapples.

John：你太合理了吗？

Eliezer：您可以遇到我们所谓的“不良理性谷”。如果您以前以平衡或取消的多种方式不合理，那么成为一半理性的人可能会使您比以前更糟。如果您选择错误的地方来首先投资技能点，那么变得更加理性的是，您会逐渐变得更糟。

but I would not recommend to people that they obsess over that possibility too much. In my experience, people who go around talking about cleverly choosing to be irrational strike me as, well, rather nitwits about it, to be frank. It’s hard to come up with a realistic non-contrived life situation where you know that it’s a good time to be irrational and you don’t already know the true answer. I think in real life, you just tell yourself the truth as best you know it, and don’t try to be clever.

On an entirely separate issue, it’s possible that being an ideal Bayesian agent is ultimately incompatible with living the life best-lived from a fun-theoretic perspective. But we’re a long, long, long way from that being a bigger problem than our current self-destructiveness.

John：您对奇异性的看法与Ray Kurzweil的视野有何不同？

Eliezer：

• I don’t think you can time AI with Moore’s Law. AI is a software problem.
• I don’t think that humans and machines “merging” is a likely source for the first superhuman intelligences. It took a century after the first cars before we could even begin to put a robotic exoskeleton on a horse, and a real car would still be faster than that.
• I don’t expect the first strong AIs to be based on algorithms discovered by way of neuroscience any more than the first airplanes looked like birds.
• 我认为纳米INFO-BIO“融合”可能不可避免，不可避免，定义明确或理想。
• I think the changes between 1930 and 1970 were bigger than the changes between 1970 and 2010.
• 我购买的生产力目前正在发达国家停滞不前。
• 我认为将摩尔的定律图示出了您所说的技术进步的图表，即它预测它比人类更聪明的AI很奇怪。智能的人AI聪明地打破了您的图形。
• 一些分析师，例如Illka Tuomi，声称摩尔的法律在00年代破裂。我并不特别不相信这一点。
• 我关心的唯一关键技术阈值是AI（即AI软件）具有强大自我改善的情况。We have no graph of progress toward this threshold and no idea where it lies (except that it should not be high above the human level because humans can do computer science), so it can’t be timed by a graph, nor known to be near, nor known to be far. (Ignorance implies a wide credibility interval, not being certain that something is far away.)
• I think outcomes are not good by default — I think outcomes can be made good, but this will require hard work that key actors may not have immediate incentives to do. Telling people that we’re on a default trajectory to great and wonderful times is false.
• I think that the “Singularity” has become a suitcase word with too many mutually incompatible meanings and details packed into it, and I’ve stopped using it.

John：Do you think you have a shot at becoming a superintelligent cyborg?

Eliezer：The conjunction law of probability theory says thatp(一个&b) ≤p(一个） - A和B发生的概率远小于单独发生的概率。可以使人类分配的实验条件p(一个&b) >p(一个） 对于一些一个&b是said to exhibit the “conjunction fallacy” — for example, in 1982, experts at the International Congress on Forecasting assigned higher probability to “A Russian invasion of Poland, and a complete breakdown of diplomatic relations with the Soviet Union” than a separate group did for “A complete breakdown of diplomatic relations with the Soviet Union”. Similarly, another group assigned higher probability to “An earthquake in California causing a flood that causes over a thousand deaths” than another group assigned to “A flood causing over a thousand deaths somewhere in North America.” Even though adding on additional details necessarily makes a story less probable, it can make the story sound more plausible. I see understanding this as a kind of Pons Asinorum of serious futurism — the distinction between carefully weighing each and every independent proposition you add to your burden, asking if you can support that detail independently of all the rest, versus making up a wonderful vivid story.

I mention this as context for my reply, which is, “Why the heck are you tacking on the ‘cyborg’ detail to that? I don’t want to be a cyborg.” You’ve got to be careful with tacking on extra details to things.

John：Do you have a shot at immortality?

Eliezer：什么，字面上的永生？字面上的永生似乎很难。寿命比几万亿年的时间要长得多，这要求我们对不断扩展的宇宙的预期命运犯错。寿命比Googolplex年长的时间更长，要求我们对物理定律的基本特征是错误的，而不仅仅是细节。

Even if some of the wilder speculations are true and it’s possible for our universe to spawn baby universes, that doesn’t get us literal immortality. To live significantly past a googolplex years without repeating yourself, you need computing structures containing more than a googol elements, and those won’t fit inside a single Hubble volume.

一个nd a googolplex is hardly infinity. To paraphrase Martin Gardner, Graham’s Number is still relatively small because most finite numbers are very much larger. Look up the fast-growing hierarchy if you really want to have your mind blown, well, eternity is longer than that. Only weird and frankly terrifying anthropic theories would let you live long enough to gaze, perhaps knowingly and perhaps not, upon the halting of the longest-running halting Turing machine with 100 states.

but I’m not sure that living to look upon the 100th Busy Beaver Number feels to me like it matters very much on a deep emotional level. I have some imaginative sympathy with myself a subjective century from now. That me will be in a position to sympathize with their future self a subjective century later. And maybe somewhere down the line is someone who faces the prospect of their future self not existing at all, and they might be very sad about that; but I’m not sure I can imagine who that person will be. “I want to live one more day. Tomorrow I’ll still want to live one more day. Therefore I want to live forever, proof by induction on the positive integers.” Even my desire for merely physical-universe-containable longevity is an abstract want by induction; it’s not that I can actually imagine myself a trillion years later.

John：I’ve described the Singularity as an “逃避现实，伪科学” fantasy that distracts us from climate change, war, inequality and other serious problems. Why am I wrong?

Eliezer：because you’re trying to forecast empirical facts by psychoanalyzing people. This never works.

假设我们达到了一个足够聪明的AI的地步，可以做与人类使AI更聪明的工作相同的工作。它可以自我调整，可以进行计算机科学，可以发明新算法。它可以自我爆发。在那之后发生了什么 - 它会变得更加聪明，看到更多的改进并迅速增强能力至一定高度的极限？还是没有太多令人兴奋的事情发生？

It could be that, (A), self-improvements of sizeδ倾向于使AI足够聪明，它可以返回并找到大小的新潜在自我完善kÅδ然后k大于一个，这是一个足够扩展的政权的继续，这是一系列迅速级联的自我完善，导致超级智能。I. J. Good称之为情报爆炸。或者可能是（b），k不到一个或这样的政权很小，也不会导致超级智能，或者是不可能的，而您的超级智能是不可能的，而您会产生嘶嘶声而不是爆炸。哪个是a或b？如果您实际上在某种特定的智能上建立了AI，并且实际上试图做到这一点，那么实际上会在经验现实世界中发生某些事情，并且该事件将取决于有关算法和可实现的改进的背景事实的决定。

You can’t get solid information about that event by psychoanalyzing people. It’s exactly the sort of thing that Bayes’s Theorem tells us is the equivalent of trying to run a car without fuel. Some people will be escapist regardless of the true values on the hidden variables of computer science, so observing some people being escapist isn’t strong evidence, even if it might make you feel like you want to disaffiliate with a belief or something.

There is a misapprehension, I think, of the nature of rationality, which is to think that it’s rational to believe “there are no closet goblins” because belief in closet goblins is foolish, immature, outdated, the sort of thing that stupid people believe. The true principle is that you go in your closet and look. So that in possible universes where there are closet goblins, you end up believing in closet goblins, and in universes with no closet goblins, you end up disbelieving in closet goblins.

It’s difficult but not impossible to try to sneak peeks through the crack of the closet door, to ask the question, “What would look different in the universe now if you couldn’t get sustained returns on cognitive investment later, such that an AI trying to improve itself would fizzle? What other facts should we observe in a universe like that?”

So you have people who say, for example, that we’ll only be able to improve AI up to the human level because we’re human ourselves, and then we won’t be able to push an AI past that. I think that if this is how the universe looks in general, then we should also observe, e.g., diminishing returns on investment in hardware and software for computer chess past the human level, which we did not in fact observe. Also, natural selection shouldn’t have been able to construct humans, and Einstein’s mother must have been one heck of a physicist, et cetera.

You have people who say, for example, that it should require more and more tweaking to get smarter algorithms and that human intelligence is around the limit. But this doesn’t square up with the anthropological record of human intelligence; we can know that there were not diminishing returns to brain tweaks and mutations producing improved cognitive power. We know this because population genetics says that mutations with very low statistical returns will not evolve to fixation at all.

一个nd hominids definitely didn’t need exponentially vaster brains than chimpanzees. And John von Neumann didn’t have a head exponentially vaster than the head of an average human.

一个nd on a sheerly pragmatic level, human axons transmit information at around a millionth of the speed of light, even when it comes to heat dissipation each synaptic operation in the brain consumes around a million times the minimum heat dissipation for an irreversible binary operation at 300 Kelvin, and so on. Why think the brain’s software is closer to optimal than the hardware? Human intelligence is privileged mainly by being the least possible level of intelligence that suffices to construct a computer; if it were possible to construct a computer with less intelligence, we’d be having this conversation at that level of intelligence instead.

但这不是一个简单的辩论，为了详细的考虑，我将人们指向我的一篇古老的非正式论文，“”情报爆炸微观经济学“不幸的是，这可能仍然是最好的来源。但是，这些是人们必须要求尝试使用我们当前可访问的证据来推理我们是否会俗称什么的问题的类型。一个I FOOM” - 是否有一个延长的政权δ认知的改善，重新投资到自我优化，产生大于δ进一步的改进。

至于您关于机会成本的问题：

There is a conceivable world where there is no intelligence explosion and no superintelligence. Or where, a related but logically distinct proposition, the tricks that machine learning experts will inevitably build up for controlling infrahuman AIs carry over pretty well to the human-equivalent and superhuman regime. Or where moral internalism is true and therefore all sufficiently advanced AIs are inevitably nice. In conceivable worlds like that, all the work and worry of the Machine Intelligence Research Institute comes to nothing and was never necessary in the first place, representing some lost number of mosquito nets that could otherwise have been bought by the Against Malaria Foundation.

还有一个可能的世界里,你工作哈rd and fight malaria, where you work hard and keep the carbon emissions to not much worse than they are already (or use geoengineering to mitigate mistakes already made). And then it ends up making no difference because your civilization failed to solve the AI alignment problem, and all the children you saved with those malaria nets grew up only to be killed by nanomachines in their sleep. (Vivid detail warning! I don’t actually know what the final hours will be like and whether nanomachines will be involved. But if we’re happy to visualize what it’s like to put a mosquito net over a bed, and then we refuse to ever visualize in concrete detail what it’s like for our civilization to fail AI alignment, that can also lead us astray.)

I think that people who try to do thought-out philanthropy, e.g., Holden Karnofsky of GiveWell, would unhesitatingly agree that these are both conceivable worlds we prefer not to enter. The question is just which of these two worlds is more probable as the one we should avoid. And again, the central principle of rationality is not to disbelieve in goblins because goblins are foolish and low-prestige, or to believe in goblins because they are exciting or beautiful. The central principle of rationality is to figure out which observational signs and logical validities can distinguish which of these two conceivable worlds is the metaphorical equivalent of believing in goblins.

我认为这是第一个不可能的世界，也是第二个可能的世界。I’m aware that in trying to convince people of that, I’m swimming uphill against a sense of eternal normality — the sense that this transient and temporary civilization of ours that has existed for only a few decades, that this species of ours that has existed for only an eyeblink of evolutionary and geological time, is all that makes sense and shall surely last forever. But given that I do think the first conceivable world is just a fond dream, it should be clear why I don’t think we should ignore a problem we’ll predictably have to panic about later. The mission of the Machine Intelligence Research Institute is to do today that research which, 30 years from now, people will desperately wish had begun 30 years earlier.

John：您的妻子Brienne相信奇异性吗？

Eliezer：Brienne回复：

如果有人问我是否“相信奇异性”，我会抬起眉毛，问他们是否“相信”机器人卡车。这是一个奇怪的问题。我对第一批机器人货车的舰队不了解，或者他们需要多长时间完全取代当代地面运输。而且，如果有一个文化上装载的手提箱术语“ Robotruckism”，其中包括许多特定的技术主张以及整个经济和社会学范式，我会犹豫地说我“相信”无人驾驶卡车。我自信地预测，无人驾驶地面运输将取代当代人类经营的地面运输，因为这显然是我们前往的地方，如果什么都没发生。同样，我自信地预测了情报爆炸。显然，如果什么都没发生，那就是我们要去的地方。我不太确定“奇异性”手提箱中的其他物品。

为了避免偏见结果，布里恩（Brienne）在没有看到我的其他答案的情况下撰写了她的答复。我们只是匹配的。

John：Can we create superintelligences without knowing how our brains work?

Eliezer：Only in the sense that you can make airplanes without knowing how a bird flies. You don’t need to be an expert in bird biology, but at the same time, it’s difficult to know enough to build an airplane without realizing一些high-level notion of how a bird might glide or push down air with its wings. That’s why I write about human rationality in the first place — if you push your grasp on machine intelligence past a certain point, you can’t help but start having ideas about how humans could think better too.

John超智想要什么?他们会哈哈ve anything resembling sexual desire?

Eliezer：Think of an enormous space of possibilities, a giant multidimensional sphere. This is Mind Design Space, the set of possible cognitive algorithms. Imagine that somewhere near the bottom of that sphere is a little tiny dot representing all the humans who ever lived — it’s a tiny dot because all humans have basically the same brain design, with a cerebral cortex, a prefrontal cortex, a cerebellum, a thalamus, and so on. It’s conserved even relative to chimpanzee brain design. Some of us are weird in little ways, you could say it’s a spiky dot, but the spikes are on the same tiny scale as the dot itself; no matter how neuroatypical you are, you aren’t running on a different cortical algorithm.

问“超级智能想要什么”是一个错误的问题。超级智能并不是这个怪异的部落，这些人以迷人的异国风情生活在水面上。“人工智能”只是在小人物之外的整个可能性的名称。With sufficient knowledge you might be able to reach into that space of possibilities and deliberately pull out an AI that wanted things that had a compact description in human wanting-language, but that wouldn’t be because this is a kind of thing that those exotic superintelligence people naturally want, it would be because you managed to pinpoint one part of the design space.

在追求物质和能量之类的东西时，我们可能会暂定地期望部分但不是完全的融合 - 似乎应该有很多可能的超级智能，这些超智能会在工具上想要物质和能量，以便提供多种多样的最终偏好。但是即使在那里，一切都受到特殊情况的击败。如果您不想拆卸备用原子，那么如果您足够了解设计空间，请伸手去拿不想伤害您的特定机器智能。

So the answer to your second question about sexual desire is that if you knew exactly what you were doing and if you had solved the general problem of building AIs that stably want particular things as they self-improve and if you had solved the general problem of pinpointing an AI’s utility functions at things that seem deceptively straightforward to human intuitions, and you’d solved an even harder problem of building an AI using the particular sort of architecture where ‘being horny’ or ‘sex makes me happy’ makes sense in the first place, then you could perhaps make an AI that had been told to look at humans, model what humans want, pick out the part of the model that was sexual desire, and then want and experience that thing too.

You could also, if you had a sufficiently good understanding of organic biology and aerodynamics, build an airplane that could mate with birds.

I don’t think this would have been a smart thing for the Wright Brothers to try to do in the early days. There would have been absolutely no point.

It does seem a lot wiser to figure out how to reach into the design space and pull out a special case of AI that will lack the default instrumental preference to disassemble us for spare atoms.

John：我喜欢认为超智能生物是非暴力的，因为他们会意识到暴力是愚蠢的。我天真吗？

Eliezer：I think so. As David Hume might have told you, you’re making a type error by trying to apply the ‘stupidity’ predicate to an agent’s terminal values or utility function. Acts, choices, policies can be stupid given some set of preferences over final states of the world. If you happen to be an agent that has meta-preferences you haven’t fully computed, you might have a platform on which to stand and call particular guesses at the derived object-level preferences as ‘stupid’.

一个paperclip maximizer is not making a computational error by having a preference order on outcomes that prefers outcomes with more paperclips in them. It is not standing from within your own preference framework and choosing blatantly mistaken acts, nor is it standing within your meta-preference framework and making mistakes about what to prefer. It is computing the answer to a different question than the question that you are asking when you ask, “What should I do?” A paperclip maximizer just outputs the action leading to the greatest number of expected paperclips.

The fatal scenario is an AI that neither loves you nor hates you, because you’re still made of atoms that it can use for something else. Game theory, and issues like cooperation in the Prisoner’s Dilemma, don’t emerge in all possible cases. In particular, they don’t emerge when something is sufficiently more powerful than you that it can disassemble you for spare atoms whether you try to press Cooperate or Defect. Past that threshold, either you solved the problem of making something that didn’t want to hurt you, or else you’ve already lost.

John：超级智能会解决意识的“严重问题”？

Eliezer：是的，回想起来，从我们的角度来看，答案看起来会很明显。

John：超级智能会拥有自由意志吗？

Eliezer：是的，但是他们不会有自由意志的幻想。

John：What’s your utopia?

Eliezer：我向您的读者介绍我的非小说有趣的理论序列, since I have not as yet succeeded in writing any novel set in a fun-theoretically optimal world.

The original interview can be found atAI有远见的Eliezer Yudkowsky在奇异性，贝叶斯大脑和壁橱哥布林。

其他功能MI的对话RI researchers have included:Yudkowsky on “What can we do now?”;Yudkowsky on logical uncertainty;benya Fallenstein on the Löbian obstacle to self-modifying systems;andYudkowsky, Muehlhauser, Karnofsky, Steinhardt, and Amodei on MIRI strategy。