The following is a fictional dialogue building off ofAI Alignment: Why It’s Hard, and Where to Start。
((琥珀色, a philanthropist interested in a more reliable Internet, and珊瑚, a computer security professional, are at a conference hotel together discussing what Coral insists is a difficult and important issue: the difficulty of building “secure” software.)
琥珀色:因此,珊瑚,我了解您认为,在创建软件时,将该软件成为您所谓的“安全”非常重要。
珊瑚:特别是如果它连接到互联网,或者控制金钱或其他贵重物品。但是,是的。
琥珀色:I find it hard to believe that this needs to be a separate topic in computer science. In general, programmers need to figure out how to make computers do what they want. The people building operating systems surely won’t want them to give access to unauthorized users, just like they won’t want those computers to crash. Why is one problem so much more difficult than the other?
珊瑚:That’s a deep question, but to give a partial deep answer: When you expose a device to the Internet, you’re potentially exposing it to intelligent adversaries who can find special, weird interactions with the system that make the pieces behave in weird ways that the programmers did not think of. When you’re dealing with that kind of problem, you’ll use a different set of methods and tools.
琥珀色:任何崩溃亚博体育苹果app官方下载的系统都以程序员没有期望的方式行为,程序员已经需要阻止这种情况发生。这种情况有何不同?
珊瑚:好的,所以……想象您的系统将每次会议进行一千千字节的输入亚博体育苹果app官方下载。(尽管这本身就是我们要提出的一种假设,并问如果它得到了投入的兆字节,但不介意。8,000possible inputs, or about 102,400or so. Again, for the sake of extending the simple visualization, imagine that a computer gets a billion inputs per second. Suppose that only a googol, 10100,在10个2,400可能的输入会导致系统的行为以某种方式原始设计师不打算。亚博体育苹果app官方下载
If the system is getting inputs in a way that’s uncorrelated with whether the input is a misbehaving one, it won’t hit on a misbehaving state before the end of the universe. If there’s an intelligent adversary who understands the system, on the other hand, they may be able to find one of the very rare inputs that makes the system misbehave. So a piece of the system that would literally never in a million years misbehave on random inputs, may break when an intelligent adversary tries deliberately to break it.
琥珀色:因此,您要说的是,这更加困难,因为程序员正在与可能比自己更聪明的对手打击他们的智慧。
珊瑚:That’s an almost-right way of putting it. What matters isn’t so much the “adversary” part as the optimization part. There are systematic, nonrandom forces strongly selecting for particular outcomes, causing pieces of the system to go down weird execution paths and occupy unexpected states. If your system literally has no misbehavior modes at all, it doesn’t matter if you have IQ 140 and the enemy has IQ 160—it’s not an arm-wrestling contest. It’s just very much harder to build a system that doesn’t enter weird states when the weird states are being selected-for in a correlated way, rather than happening only by accident. The weirdness-selecting forces can search through parts of the larger state space that you yourself failed to imagine. Beating that does indeed require new skills and a different mode of thinking, what Bruce Schneier called “security mindset”.
琥珀色:Ah, and what is this security mindset?
珊瑚:I can say one or two things about it, but keep in mind we are dealing with a quality of thinking that is not entirely effable. If I could give you a handful of platitudes about security mindset, and that would actually cause you to be able to design secure software, the Internet would look very different from how it presently does. That said, it seems to me that what has been called “security mindset” can be divided into two components, one of which is much less difficult than the other. And this can fool people into overestimating their own safety, because they can get the easier half of security mindset and overlook the other half. The less difficult component, I will call by the term “ordinary paranoia”.
琥珀色:Ordinaryparanoia?
珊瑚:许多程序员有能力想象试图威胁他们的对手。他们想象对手能够以特定的方式攻击他们的可能性有多大,然后他们试图阻止对手的威胁。想象攻击,包括怪异或巧妙的攻击,并通过您想象的措施将其招待;那是普通的妄想症。
琥珀色:Isn’t that what security is all about? What do you claim is the other half?
珊瑚:To put it as a platitude, I might say… defending against mistakes in your own assumptions rather than against external adversaries.
琥珀色:你能给我一个差异的例子吗?
珊瑚:An ordinary paranoid programmer imagines that an adversary might try to read the file containing all the usernames and passwords. They might try to store the file in a special, secure area of the disk or a special subpart of the operating system that’s supposed to be harder to read. Conversely, somebody with security mindset thinks, “No matter what kind of special system I put around this file, I’m disturbed by needing to make the assumption that this file can’t be read. Maybe the special code I write, because it’s used less often, is more likely to contain bugs. Or maybe there’s a way to fish data out of the disk that doesn’t go through the code I wrote.”
琥珀色:他们想象,对手可能能够获取信息,并阻止这些途径!因为他们有更好的想象力。
珊瑚:好吧,我们有点做,但这不是关键区别。我们真正想做的是为计算机检查不依赖计算机存储密码的密码的方法根本, anywhere。
琥珀色:啊,就像加密密码文件一样!
珊瑚:No, that just duplicates the problem at one remove. If the computer can decrypt the password file to check it, it’s stored the decryption key somewhere, and the attacker may be able to steal that key too.
琥珀色:But then the attacker has to steal two things instead of one; doesn’t that make the system more secure? Especially if you write two different sections of special filesystem code for hiding the encryption key and hiding the encrypted password file?
珊瑚:That’s exactly what I mean by distinguishing “ordinary paranoia” that doesn’t capture the full security mindset. So long as the system is capable of reconstructing the password, we’ll always worry that the adversary might be able to trick the system into doing just that. What somebody with security mindset will recognize as a deeper solution is to store a one-way hash of the password, rather than storing the plaintext password. Then even if the attacker reads off the password file, they still can’t give what the system will recognize as a password.
琥珀色:啊,那很聪明!但是我看不出该措施之间的质量不同,以及我对密钥和加密密码文件的措施分开的措施。我同意您的措施更加聪明,更优雅,但是当然,您会比我了解更好的标准解决方案,因为您在该领域的职业工作。我看不到定性线将您的解决方案与我的解决方案分开。
珊瑚:Um, it’s hard to say this without offending some people, but… it’s possible that even after I try to explain the difference, which I’m about to do, you won’t get it. Like I said, if I could give you some handy platitudes and transform you into somebody capable of doing truly good work in computer security, the Internet would look very different from its present form. I can try to describe one aspect of the difference, but that may put me in the position of a mathematician trying to explain what looks more promising about one proof avenue than another; you can listen to everything they say and nod along and still not be transformed into a mathematician. So Iamgoing to try to explain the difference, but again, I don’t know of any simple instruction manuals for becoming Bruce Schneier.
琥珀色:我承认,对某些人所拥有的这种据说无法言喻的能力有点怀疑,而其他人则没有 -
珊瑚:在许多专业中都有类似的事情。有些人在五岁时浏览了为TRS-80编写的基本程序的页面来挑选编程,而有些人在二十五岁时确实很难掌握基本的python。这不是因为这位五岁的孩子知道您可以口头传达给二十五岁的孩子。
而且,是的,五岁的孩子会随着练习而变得更好。这并不是说我们谈论的是不可吸引的天才。您可能会有陈词滥调告诉这位25岁的年轻人,这将帮助他们少努力。但是有时候,一个职业需要以异常的方式思考,有些人的思想更容易在特定的维度上侧向侧面。
琥珀色:Fine, go on.
珊瑚:Okay, so… you thought of putting the encrypted password file in one special place in the filesystem, and the key in another special place. Why not encrypt the key too, write a third special section of code, and store the key to the encrypted key there? Wouldn’t that make the system even more secure? How about seven keys hidden in different places, wouldn’t that be extremely secure? Practically unbreakable, even?
琥珀色:Well, that version of the idea does feel a little silly. If you’re trying to secure a door, a lock that takes two keys might be more secure than a lock that only needs one key, but seven keys doesn’t feel like it makes the door that much more secure than two.
珊瑚:Why not?
琥珀色:It just seems silly. You’d probably have a better way of saying it than I would.
珊瑚:嗯,奇特的方式描述愚蠢是that the chance of obtaining the seventh key is not conditionally independent of the chance of obtaining the first two keys. If I can read the encrypted password file, and read your encrypted encryption key, then I’ve probably come up with something that just bypasses your filesystem and reads directly from the disk. And the more complicated you make your filesystem, the more likely it is that I can find a weird system state that will let me do just that. Maybe the special section of filesystem code you wrote to hide your fourth key is the one with the bug that lets me read the disk directly.
琥珀色:So the difference is that the person with atruesecurity mindset found a defense that makes the system simpler rather than more complicated.
珊瑚:Again, that’s almost right. By hashing the passwords, the security professional has made theirreasoningabout the system less complicated. They’ve eliminated the need for an assumption that might be put under a lot of pressure. If you put the key in one special place and the encrypted password file in another special place, the system as a whole is still able to decrypt the user’s password. An adversary probing the state space might be able to trigger that password-decrypting state because the system is designed to do that on at least some occasions. By hashing the password file we eliminate that whole internal debate from the reasoning on which the system’s security rests.
琥珀色:But even after you’ve come up with that clever trick, something could still go wrong. You’re still not absolutely secure. What if somebody uses “password” as their password?
珊瑚:Or what if somebody comes up a way to read off the password after the user has entered it and while it’s still stored in RAM, because something got access to RAM? The point of eliminating the extra assumption from the reasoning about the system’s security is not that we are then absolutely secure and safe and can relax. Somebody with security mindset isnever关于推理的大厦说系统是安全的。亚博体育苹果app官方下载
For that matter, while there are some normal programmers doing normal programming who might put in a bunch of debugging effort and then feel satisfied, like they’d done all they could reasonably do, programmers with decent levels of ordinary paranoia about ordinary programs will go on chewing ideas in the shower and coming up with more function tests for the system to pass. So the distinction between security mindset and ordinary paranoia isn’t that ordinary paranoids will relax.
It’s that… again to put it as a platitude, the ordinary paranoid is running around putting out fires in the form of ways they imagine an adversary might attack, and somebody with security mindset is defending against something closer to “what if an element of this reasoning is mistaken”. Instead of trying really hard to ensure nobody can read a disk, we are going to build a system that’s secure even if somebody does read the disk, andthatis our first line of defense. And then we are also going to build a filesystem that doesn’t let adversaries read the password file, as asecondline of defense in case our one-way hash is secretly broken, and because there’s no positive need to let adversaries read the disk so why let them. And then we’re going to salt the hash in case somebody snuck a low-entropy password through our system and the adversary manages to read the password anyway.
琥珀色:So rather than trying to outwit adversaries, somebody with true security mindset tries to make fewer assumptions.
珊瑚:Well, we think in terms of adversaries too! Adversarial reasoning is easier to teach than security mindset, but it’s still (a) mandatory and (b) hard to teach in an absolute sense. A lot of people can’t master it, which is why a description of “security mindset” often opens with a story about somebody failing at adversarial reasoning and somebody else launching a clever attack to penetrate their defense.
You need to master two ways of thinking, and there are a lot of people going around who have the first way of thinking but not the second. One way I’d describe the deeper skill is seeing a system’s security as resting on a story about why that system is safe. We want that safety-story to be as solid as possible. One of the implications is resting the story on as few assumptions as possible; as the saying goes, the only gear that never fails is one that has been designed out of the machine.
琥珀色:But can’t you also get better security by adding more lines of defense? Wouldn’t that be more complexity in the story, and also better security?
珊瑚:There’s also something to be said for preferring disjunctive reasoning over conjunctive reasoning in the safety-story. But it’s important to realize that you do want a primary line of defense that is supposed to just work and be unassailable, not a series of weaker fences that you think might maybe work. Somebody who doesn’t understand cryptography might devise twenty clever-seeming amateur codes and apply them all in sequence, thinking that, even if one of the codes turns out to be breakable, surely they won’tallbe breakable. The NSA will assign that mighty edifice of amateur encryption to an intern, and the intern will crack it in an afternoon.
There’s something to be said for redundancy, and having fallbacks in case the unassailable wall falls; it can be wise to have additional lines of defense, so long as the added complexity does not make the larger system harder to understand or increase its vulnerable surfaces. But at the core you need a simple, solid story about why the system is secure, and a good security thinker will be trying to eliminate whole assumptions from that story and strengthening its core pillars, not only scurrying around parrying expected attacks and putting out risk-fires.
That said, it’s better to use two true assumptions than one false assumption, so simplicity isn’t everything.
琥珀色:I wonder if that way of thinking has applications beyond computer security?
珊瑚:I’d rather think so, as the proverb about gears suggests.
For example, stepping out of character for a moment, the author of this dialogue has sometimes been known to discussthe alignment problem for Artificial General Intelligence。他一度在谈论试图量ure rates of improvement inside a growing AI system, so that it would not do too much thinking with humans out of the loop if a breakthrough occurred while the system was running overnight. The person he was talking to replied that, to him, it seemed unlikely that an AGI would gain in power that fast. To which the author replied, more or less:
It shouldn’t be your job to guess how fast the AGI might improve! If you write a system that will hurt youifa certain speed of self-improvement turns out to be possible, then you’ve written the wrong code. The code should just never hurt you regardless of the true value of that background parameter.
一个更好的方法来设置AGI会来衡量how much improvement is taking place, and if more thanXimprovement takes place, suspend the system until a programmer validates the progress that’s already occurred. That way even if the improvement takes place over the course of a millisecond, you’re still fine, so long as the system works as intended. Maybe the system doesn’t work as intended because of some other mistake, but that’s a better problem to worry about than a system that hurts you即使it works as intended.
Similarly, you want to design the system so that if it discovers amazing new capabilities, it waits for an operator to validate use of those capabilities—not rely on the operator to watch what’s happening and press a suspend button. You shouldn’t rely on the speed of discovery or the speed of disaster being less than the operator’s reaction time. There’s noneedto bake in an assumption like that if you can find a design that’s safe regardless. For example, by operating on a paradigm of allowing operator-whitelisted methods rather than avoiding operator-blacklisted methods; you require the operator to say “Yes” before proceeding, rather than assuming they’re present and attentive and can say “No” fast enough.
琥珀色:Well, okay, but if we’re guarding against an AI system discovering cosmic powers in a millisecond, that does seem to me like an unreasonable thing to worry about. I guess that marks me as a merely ordinary paranoid.
珊瑚:确实,安全专业人员的标志之一是,他们花了很多时间担心边缘案例,这些案例不会引起普通的偏执,因为边缘案件听起来不像对手可能会做的事情。这是一个例子from the Freedom to Tinker blog:
对“无害失败”的这种兴趣(对手可能引起异常但不是直接有害结果的情况)是安全心态的另一个标志。并非所有“无害的失败”会导致大麻烦,但令人惊讶的是,聪明的对手可以多久将一堆看似无害的失败堆积到危险的麻烦之塔中。无害的失败是不良的卫生。我们尝试在可能的时候将它们排除在外…
To see why, consider the donotreply.com email story that hit the press recently. When companies send out commercial email (e.g., an airline notifying a passenger of a flight delay) and they don’t want the recipient to reply to the email, they often put in a bogus From address like donotreply@donotreply.com. A clever guy registered the domain donotreply.com, thereby receiving all email addressed to donotreply.com. This included “bounce” replies to misaddressed emails, some of which contained copies of the original email, with information such as bank account statements, site information about military bases in Iraq, and so on…
The people who put donotreply.com email addresses into their outgoing email must have known that they didn’t control the donotreply.com domain, so they must have thought of any reply messages directed there as harmless failures. Having gotten that far, there are two ways to avoid trouble. The first way is to think carefully about the traffic that might go to donotreply.com, and realize that some of it is actually dangerous. The second way is to think, “This looks like a harmless failure, but we should avoid it anyway. No good can come of this.” The first way protects you if you’re clever; the second way always protects you.
“The first way protects you if you’re clever; the second way always protects you.” That’s very much the other half of the security mindset. It’s what this essay’s author was doing by talking about AGI alignment that runs on whitelisting rather than blacklisting: you shouldn’t assume you’ll be clever about how fast the AGI system could discover capabilities, you should have a system that doesn’t use not-yet-whitelisted capabilities even if they are discovered very suddenly.
如果您的AGI会在一毫秒内获得全部宇宙力量,这意味着您建立了一种认知过程,从某种意义上说,试图伤害您并仅由于您认为缺乏能力而失败。这是very badand you should be designing some other AGI system instead. AGI systems should never be running a search that will hurt you if the search comes up non-empty. You should not be trying to fix that by making sure the search comes up empty thanks to your clever shallow defenses closing off all the AGI’s clever avenues for hurting you. You should fix that by making sure no search like that ever runs. It’s a silly thing to do with computing power, and you should do something else with computing power instead.
Going back to ordinary computer security, if you try building a lock with seven keys hidden in different places, you are in some dimension pitting your cleverness against an adversary trying to read the keys. The person with security mindset doesn’t want to rely on having to win the cleverness contest. An ordinary paranoid, somebody who can master the kind of default paranoia that lots of intelligent programmers have, will look at the Reply-To field saying donotreply@donotreply.com and think about the possibility of an adversary registering the donotreply.com domain. Somebody with security mindset thinks in assumptions rather than adversaries. “Well, I’m assuming that this reply email goes nowhere,” they’ll think, “but maybe I should design the system so that I don’t need to fret about whether that assumption is true.”
琥珀色:Because as the truly great paranoid knows, what seems like a ridiculously improbable way for the adversary to attack sometimes turns out to not be so ridiculous after all.
珊瑚:同样,这是一种不恰当的方式来提出它。当我不设置电子邮件以源自donotreply@donotreply.com时,这不仅仅是因为我感谢注册Donotreply.com的对手比新手想象的更有可能。就我所知,当弹跳电子邮件无处发送时,可能会发生各种事情!也许弹跳的电子邮件的工作方式是,该电子邮件被路由到寻找该地址的怪异地方。我不知道,我不想学习它。相反,我会问:我可以做到这一点,以便弹跳的电子邮件不会产生答复吗?我可以做到这一点,以便弹跳的电子邮件不包含原始消息的文字吗?也许我可以查询电子邮件服务器,以确保它在我尝试发送消息之前仍然具有该名称的用户?如果对于未经授权的人来说,这是非常不好的,也许我不应该通过电子邮件将其发送。
琥珀色:So the person with true security mindset understands that where there’s one problem, demonstrated by what seems like a very unlikely thought experiment, there’s likely to be more realistic problems that an adversary can in fact exploit. What I think of as weird improbable failure scenarios are canaries in the coal mine, that would warn a truly paranoid person of bigger problems on the way.
珊瑚:同样,这不是完全正确的。The person with ordinary paranoia hears about donotreply@donotreply.com and may think something like, “Oh, well, it’s not very likely that an attacker will actually try to register that domain, I have more urgent issues to worry about,” because in that mode of thinking, they’re running around putting out things that might be fires, and they have to prioritize the things that are most likely to be fires.
如果您向具有安全心态的人展示了一个怪异的边缘思想实验,那么他们就不会看到更有可能是火的东西。他们认为:“哦,不,我相信那些弹跳电子邮件无处可寻!”构建安全操作系统的OpenBSD项目在传递中也建立了一个非常强大的操作系统,因为从他们的角度来看亚博体育苹果app官方下载,任何可能崩溃的错误被认为是关键的安全漏洞。普通的偏执狂会看到一个崩溃的输入,并认为:“崩溃并不像有人窃取我的数据那样糟糕。亚博体育苹果app官方下载直到您向我证明该错误可以由对手使用来窃取数据极其critical.” Somebody with security mindset thinks, “Nothing inside this subsystem is supposed to behave in a way that crashes the OS. Some section of code is behaving in a way that does not work like my model of that code. Who knows what it might do? The system isn’t supposed to crash, so by making it crash, you have demonstrated that my beliefs about how this system works are false.”
琥珀色:I’ll be honest: Ithassometimes struck me that people who call themselves security professionals seem overly concerned with what, to me, seem like very improbable scenarios. Like somebody forgetting to check the end of a buffer and an adversary throwing in a huge string of characters that overwrite the end of the stack with a return address that jumps to a section of code somewhere else in the system that does something the adversary wants. How likely is thatreallyto be a problem? I suspect that in the real world, what’s more likely is somebody making their password “password”. Shouldn’t you be mainly guarding against that instead?
珊瑚:You have to do both. This game is short on consolation prizes. If you want your system to resist attack by major governments, you need it to actually be pretty darned secure, gosh darn it. The fact that some users may try to make their password be “password” does not change the fact that you also have to protect against buffer overflows.
琥珀色:But even when somebody with security mindset designs an operating system, it often still ends up with successful attacks against it, right? So if this deeper paranoia doesn’t eliminate all chance of bugs, is it really worth the extra effort?
珊瑚:If you don’t have somebody who thinks this way in charge of building your operating system, it hasnochance of not failing immediately. People with security mindset sometimes fail to build secure systems. People without security mindsetalwaysfail at security if the system is at all complex. What this way of thinking buys you is achancethat your system takes longer than 24 hours to break.
琥珀色:That sounds a little extreme.
珊瑚:History shows that reality has not cared what you consider “extreme” in this regard, and that is why your Wi-Fi-enabled lightbulb is part of a Russian botnet.
琥珀色:Look, I understand that you want to get all the fiddly tiny bits of the system exactly right. I like tidy neat things too. But let’s be reasonable; we can’t always get everything we want in life.
珊瑚:您认为您正在与我进行谈判,但您确实正在与墨菲定律进行谈判。恐怕墨菲先生的要求在历史上一直是不合理的,而且对那些拒绝与他们见面的人的宽容。我不是在为您提倡一项政策,只是告诉您如果您不遵守该政策会发生什么。也许您认为,如果您的灯泡在爱沙尼亚的一家床垫商店进行拒绝服务攻击,这并不是特别糟糕。但是,如果您确实希望系统安全,则需要做某些亚博体育苹果app官方下载事情,而这部分更像是自然法则,而不是可转让的需求。
琥珀色:Non-negotiable, eh? I bet you’d change your tune if somebody offered you twenty thousand dollars. But anyway, one thing I’m surprised you’re not mentioning more is the part where people with security mindset always submit their idea to peer scrutiny and then accept what other people vote about it. I do like the sound of that; it sounds very communitarian and modest.
珊瑚:我想这是许多程序员拥有的普通偏执狂的一部分。向他人审查提交想法的目的并不难理解,尽管当然有很多人甚至没有这样做。如果我有任何原始的评论可以为计算机安全方面的那个破旧的话题做出贡献,我会指出,它是对明智的偏执狂的建议,但是当然,需要更多的人是快乐的无辜者。
琥珀色:Happy innocents?
珊瑚:甚至缺乏普通偏执狂的人。快乐的无辜者倾向于设想他们的系统工作的方法,但不问亚博体育苹果app官方下载根本how their system might fail, until somebody prompts them into that, and even then they can’t do it. Or at least that’s been my experience, and that of many others in the profession.
有一个令人难以置信的可怕加密系统,相当于傻瓜伴侣在国际象棋中的伴侣,有时会被最多的业余爱好者(即快速XO亚博体育苹果app官方下载R)融合在一起。那就是选择密码,重复密码并使用重复的密码字符串X键置。发明该系统的人可能根本无法采取对手的看法。亚博体育苹果app官方下载He想s his marvelous cipher to be unbreakable, and he is not able to truly enter the frame of mind of somebody who wants his cipher to be breakable. If you ask him, “Please,try想象一下可能出了什么问题,”他可能会说,“好吧,如果密码丢失,数据将永远无法恢复,因为我的加密算法太强了;我想这可能会出现问题。”或者,“也许有人会破坏我的代码”,或者,“如果您真的坚持认为我发明了牵强的方案,那么计算机也许会自发决定不服从我的编程。”当然,任何有能力的普通偏执狂都会询问他们可以找到的最熟练的人来看一个好主意并试图将其击落,因为其他思想可能以不同的角度出现或知道其他标准技术。但是,我们说“不要滚动您自己的加密货币!”的另一个原因是另一个原因。和“让安全专家看看您的好主意!”希望能够接触到许多无法根本扭转目标的极性 - 他们不会自发地认为,如果您试图强迫他们这样做,他们的想法会朝着无效的方向发展。
琥珀色:Like… the same way many people on the Right/Left seem utterly incapable of stepping outside their own treasured perspectives to pass theIdeological Turing Test左/右。
珊瑚:我不知道如果它是完全相同的精神齿轮or capability, but there’s a definite similarity. Somebody who lacks ordinary paranoia can’t take on the viewpoint of somebody who wants Fast XOR to be breakable, and pass that adversary’s Ideological Turing Test for attempts to break Fast XOR.
琥珀色:Can’t, or won’t? You seem to be talking like these are innate, untrainable abilities.
珊瑚:Well, at the least, there will be different levels of talent, as usual in a profession. And also as usual, talent vastly benefits from training and practice. But yes, it has sometimes seemed to me that there is a kind of qualitative step or gear here, where some people can shift perspective to imagine an adversary that truly wants to break their code… or a reality that isn’t cheering for their plan to work, or aliens who evolved different emotions, or an AI that doesn’t想to conclude its reasoning with “And therefore the humans should live happily ever after”, or a fictional character who believes in Sith ideology and yetdoesn’t believe they’re the bad guy。
It does sometimes seem to me like some people simply can’t shift perspective in that way. Maybe it’s not that they truly lack the wiring, but that there’s an instinctive political off-switch for the ability. Maybe they’re scared to let go of their mental anchors. But from the outside it looks like the same result: some people do it, some people don’t. Some people spontaneously invert the polarity of their internal goals and spontaneously ask how their cipher might be broken and come up with productive angles of attack. Other people wait until prompted to look for flaws in their cipher, or they demand that you argue with them and wait for you to come up with an argument that satisfies them. If you ask them to predict themselves what you might suggest as a flaw, they say weird things that don’t begin to pass your Ideological Turing Test.
琥珀色:You do seem to like your qualitative distinctions. Are there better or worse ordinary paranoids? Like, is there a spectrum in the space between “happy innocent” and “true deep security mindset”?
珊瑚:One obvious quantitative talent level within ordinary paranoia would be in how far you can twist your perspective to look sideways at things—the creativity and workability of the attacks you invent. Like theseexamples布鲁斯·施尼耶(Bruce Schneier)给了:
Uncle Milton Industries has been selling ant farms to children since 1956. Some years ago, I remember opening one up with a friend. There were no actual ants included in the box. Instead, there was a card that you filled in with your address, and the company would mail you some ants. My friend expressed surprise that you could get ants sent to you in the mail.
I replied: “What’s really interesting is that these people will send a tube of live ants to anyone you tell them to.”
Security requires a particular mindset. Security professionals—at least the good ones—see the world differently. They can’t walk into a store without noticing how they might shoplift. They can’t use a computer without wondering about the security vulnerabilities. They can’t vote without trying to figure out how to vote twice. They just can’t help it.
Smartwater是一种液体,具有与特定所有者相关的唯一标识符。当我第一次了解这个想法时,我写道:“这个想法是我的贵重物品作为所有权证明。”“我认为更好的主意是我在您的贵重物品上画画,然后报警。”
Really, we can’t help it.
This kind of thinking is not natural for most people. It’s not natural for engineers. Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail…
I’ve often speculated about how much of this is innate, and how much is teachable. In general, I think it’s a particular way of looking at the world, and that it’s far easier to teach someone domain expertise—cryptography or software security or safecracking or document forgery—than it is to teach someone a security mindset.
To be clear, the distinction between “just ordinary paranoia” and “all of security mindset” is my own; I think it’s worth dividing the spectrum above the happy innocents into two levels rather than one, and say, “This business of looking at the world from weird angles is only half of what you need to learn, and it’s the easier half.”
琥珀色:也许Bruce Schneier自己不明白哟u mean when you say “security mindset”, and you’ve simply stolen his term to refer to a whole new idea of your own!
珊瑚:No, the thing with not wanting to have to reason about whether somebody might someday register “donotreply.com” and just fixing it regardless—a methodology that doesn’t trust you to be clever about which problems will blow up—that’s definitely part of what existing security professionals mean by “security mindset”, and it’s definitely part of the second and deeper half. The only unconventional thing in my presentation is that I’m factoring out an intermediate skill of “ordinary paranoia”, where you try to parry an imagined attack by encrypting your password file and hiding the encryption key in a separate section of filesystem code. Coming up with the idea of hashing the password file is, I suspect, a qualitatively distinct skill, invoking a world whose dimensions are your own reasoning processes and not just object-level systems and attackers. Though it’s not polite to say, and the usual suspects will interpret it as a status grab, my experience with other reflectivity-laden skills suggests this may mean that many people, possibly including you, will prove unable to think in this way.
琥珀色:我确实发现这很不礼貌。
珊瑚:It may indeed be impolite; I don’t deny that. Whether it’s untrue is a different question. The reason I say it is because, as much as I want ordinary paranoids totryto reach up to a deeper level of paranoia, I want them to be aware that it might not prove to be their thing, in which case they should get help and then listen to that help. They shouldn’t assume that because they can notice the chance to have ants mailed to people, they can also pick up on the awfulness of donotreply@donotreply.com.
琥珀色:Maybe you could call that “deep security” to distinguish it from what Bruce Schneier and other security professionals call “security mindset”.
珊瑚:“Security mindset” equals “ordinary paranoia” plus “deep security”? I’m not sure that’s very good terminology, but I won’t mind if you use the term that way.
琥珀色:假设我以面值为准。早些时候,您描述了一个幸福的无辜试图并没有成为普通偏执狂时可能出了什么问题。当普通偏执狂试图做需要深厚安全技能的事情时,会发生什么?
珊瑚:他们认为,他们明智地确定了不良密码,因为需要推出的真正大火,并花所有的时间写越来越聪明的检查以获取不良密码。他们对检测到不良密码的努力以及对系统安全表现出了多少关注的努力给他们留下了深刻的印象。亚博体育苹果app官方下载They fall prey to the standard cognitive bias whose name I can’t remember, where people want to solve a problem using one big effort or a couple of big efforts and then be done and not try anymore, and that’s why people don’t put up hurricane shutters once they’re finished buying bottled water. Pay them to “try harder”, and they’ll hide seven encryption keys to the password file in seven different places, or build towers higher and higher in places where a successful adversary is obviously just walking around the tower if they’ve gotten through at all. What these ideas have in common is that they are in a certain sense “shallow”. They are mentally straightforward as attempted parries against a particular kind of envisioned attack. They give you a satisfying sense of fighting hard against the imagined problem—and then they fail.
琥珀色:Are you saying it’snota good idea to check that the user’s password isn’t “password”?
珊瑚:No, shallow defenses are often good ideas too! But even there, somebody with the higher skill will try to look at things in a more systematic way; they know that there are often deeper ways of looking at the problem to be found, and they’ll try to find those deep views. For example, it’s extremely important that your password checker doesnotrule out the password “correct horse battery staple” by demanding the password contain at least one uppercase letter, lowercase letter, number, and punctuation mark. What you really want to do is measure password entropy. Not envision a failure mode of somebody guessing “rainbow”, which you will cleverly balk by forcing the user to make their password be “rA1nbow!” instead.
你想要的密码输入字段checkbox that allows showing the typed password in plaintext, because your attempt to parry the imagined failure mode of some evildoer reading over the user’s shoulder may get in the way of the user entering a long or high-entropy password. And the user is perfectly capable of typing their password into that convenient text field in the address bar above the web page, so they can copy and paste it—thereby sending your password to whoever tries to do smart lookups on the address bar. If you’re really that worried about some evildoer reading over somebody’s shoulder, maybe you should be sending a confirmation text to their phone, rather than forcing the user to enter their password into a nearby text field that they can actually read. Obscuring one text field, with no off-switch for the obscuration, to guard against this one bad thing that you imagined happening, while managing to step on your own feet in other ways and not even really guard against the bad thing; that’s the peril of shallow defenses.
“普通偏执狂,他认为他真的很努力,但实际上只是堆积了很多浅的预防措施”是一个原型的角色Harry Potterseries, who has a whole room full of Dark Detectors, and who also ends up locked in the bottom of somebody’s trunk. It seems Mad-Eye Moody was too busy buying one more Dark Detector for his existing room full of Dark Detectors, and he didn’t invent precautions deep enough and general enough to cover the unforeseen attack vector “somebody tries to replace me using Polyjuice”.
And the solution isn’t to add on a special anti-Polyjuice potion. I mean, if you happen to have one, great, but that’s not where most of your trust in the system should be coming from. The first lines of defense should have a sense about them of depth, of generality. Hashing password files, rather than hiding keys; thinking of how to measure password entropy, rather than requiring at least one uppercase character.
琥珀色:Again this seems to me more like a quantitative difference in the cleverness of clever ideas, rather than two different modes of thinking.
珊瑚:真实的分类往往是模糊的,但对我来说these seem like the product of two different kinds of thinking. My guess is that the person who popularized demanding a mixture of letters, cases, and numbers was reasoning in a different way than the person who thought of measuring password entropy. But whether you call the distinction qualitative or quantitative, the distinction remains. Deep and general ideas—the kind that actually simplify and strengthen the edifice of reasoning supporting the system’s safety—are invented more rarely and by rarer people. To build a system that can resist or even slow down an attack by multiple adversaries, some of whom may be smarter or more experienced than ourselves, requires a level of professionally specialized thinking that isn’t reasonable to expect from every programmer—not even those who can shift their minds to take on the perspective of a single equally-smart adversary. What you should ask from an ordinary paranoid is that they appreciate that deeper ideas exist, and that they try to learn the standard deeper ideas that are already known; that they know their own skill is not the upper limit of what’s possible, and that they ask a professional to come in and check their reasoning. And then actually listen.
琥珀色:But if it’s possible for people to think they have higher skills and be mistaken, how do you know that你are one of these rare people whotrulyhas a deep security mindset? Might your high opinion of yourselfjust be due to the Dunning-Kruger effect?
珊瑚:…Okay, that reminds me to give another caution.
是的,会有一些无辜者不敢相信他们所缺乏的称为“偏执狂”的才能,如果您要求他们更担心他们出色的加密想法中的缺陷,他们会想出奇怪的模仿。还会有一些人在严重的情况下阅读此书social anxiety and underconfidence。Readers whoarecapable of ordinary paranoia and even security mindset, who might not try to develop these talents, because they are terribly worried that they might just be one of the people who only imagine themselves to have talent. Well, if you think you can feel the distinction between deep security ideas and shallow ones, you should at least try now and then to generate your own thoughts that resonate in you the same way.
琥珀色:But won’t that attitude encourage overconfident people to think they can be paranoid when they actually can’t be, with the result that they end up too impressed with their own reasoning and ideas?
珊瑚:I strongly suspect that they’ll do that regardless. You’re not actually promoting some kind of collective good practice that benefits everyone, just by personally agreeing to be modest. The overconfident don’t care what you decide. And if you’re not just as worried about underestimating yourself as overestimating yourself, if your fears about exceeding your proper place are asymmetric with your fears about lost potential and foregone opportunities, then you’re probably dealing with an emotional issue rather than a strict concern with good epistemology.
琥珀色:If somebody does have the talent for deep security, then, how can they train it?
珊瑚:…这是一个好问题。已经为普通偏执狂开发了一些有趣的培训方法,例如学生必须弄清楚如何在计算机科学环境之外攻击日常系统。亚博体育苹果app官方下载一位教授进行了一个测试,其中一个问题是“ PI的前100位数字是什么?” - 关键是您需要找到一些作弊的方法才能通过测试。如果您还没有这样做,则应首先训练这种普通的偏执狂。
琥珀色:And then what? How do you graduate to deep security from ordinary paranoia?
珊瑚:…Try to find more general defenses instead of parrying particular attacks? Appreciate the extent to which you’re building ever-taller versions of towers that an adversary might just walk around? Ugh, no, that’s too much like ordinary paranoia—especially if you’re starting out with just ordinary paranoia. Let me think about this.
…
好的,我有一个古怪的proba的的建议bly not going to work. Write down the safety-story on which your belief in a system’s security rests. Then ask yourself whether you actually included all the empirical assumptions. Then ask yourself whether you actually believe those empirical assumptions.
琥珀色:So, like, if I’m building an operating system, I write down, “Safety assumption: The login system works to keep out attackers”—
珊瑚:No!
Uh, no, sorry. As usual, it seems that what I think is “advice” has left out all the important parts anyone would need to actually do it.
That’s not what I was trying to handwave at by saying “empirical assumption”. You don’t want to assume that parts of the system “succeed” or “fail”—that’s not language that should appear in what you write down. You want the elements of the story to be strictly factual, not… value-laden, goal-laden? There shouldn’t be reasoning that explicitly mentions what you want to have happen or not happen, just language neutrally describing the background facts of the universe. For brainstorming purposes you might write down “Nobody can guess the password of any user with dangerous privileges”, but that’s just a proto-statement which needs to be refined into more basic statements.
琥珀色:I don’t think I understood.
珊瑚:“Nobody can guess the password” says, “I believe the adversary will fail to guess the password.” Why do you believe that?
琥珀色:I see, so you want me to refine complex assumptions into systems of simpler assumptions. But if you keep asking “why do you believe that” you’ll eventually end up back at the Big Bang and the laws of physics. How do I know when to stop?
珊瑚:您要做的是将故事减少到您谈论实现目标的事件“对手失败”的地步,而是谈论该事件的中立事实。就目前而言,只需回答我:为什么您认为对手无法猜测密码?
琥珀色:Because the password is too hard to guess.
珊瑚:The phrase “too hard” is goal-laden language; it’s your own desires for the system that determine what is “too hard”. Without using concepts or language that refer to what you want, what is a neutral, factual description of what makes a password too hard to guess?
琥珀色:The password has high-enough entropy that the attacker can’t try enough attempts to guess it.
珊瑚:We’re making progress, but again, the term “enough” is goal-laden language. It’s your own wants and desires that determine what is “enough”. Can you say something else instead of “enough”?
琥珀色:The password has sufficient entropy that—
珊瑚:我不是说找到“足够”的同义词。我的意思是,使用不采用目标的不同概念。这将涉及更改您写下的内容的含义。
琥珀色:I’m sorry, I guess I’m not good enough at this.
珊瑚:Not yet, anyway. Maybe not ever, but that isn’t known, and you shouldn’t assume it based on one failure.
Anyway, what I was hoping for was a pair of statements like, “I believe every password has at least 50 bits of entropy” and “I believe no attacker can make more than a trillion tries total at guessing any password”. Where the point of writing “I believe” is to make yourself pause and question whether you actually believe it.
琥珀色:Isn’t saying no attacker “can” make a trillion tries itself goal-laden language?
珊瑚:Indeed, that assumption might need to be refined further via why-do-I-believe-that into, “I believe the system rejects password attempts closer than 1 second together, I believe the attacker keeps this up for less than a month, and I believe the attacker launches fewer than 300,000 simultaneous connections.” Where again, the point is that you then look at what you’ve written and say, “Do I really believe that?” To be clear, sometimes the answer will be “Yes, I sure do believe that!” This isn’t a social modesty exercise where you show off your ability to have agonizing doubts and then you go ahead and do the same thing anyway. The point is to find out what you believe, or what you’d need to believe, and check that it’s believable.
琥珀色:And this trains a deep security mindset?
珊瑚:…Maaaybe? I’m wildly guessing it might? It may get you to think in terms of stories and reasoning and assumptions alongside passwords and adversaries, and that puts your mind into a space that I think is at least part of the skill.
In point of fact, the real reason the author is listing out this methodology is that he’s currently trying to do something similar on the problem of aligning Artificial General Intelligence, and he would like to move past “I believe my AGI won’t want to kill anyone” and into a headspace more like writing down statements such as “Although the space of potential weightings for this recurrent neural net does contain weight combinations that would figure out how to kill the programmers, I believe that gradient descent on loss functionL仅在子空间内访问结果Qwith propertiesP, and I believe a space with propertiesPdoes not include any weight combinations that figure out how to kill the programmer.”
Though this itself is not really a reduced statement and still has too much goal-laden language in it. A realistic example would take us right out of the main essay here. But the author does hope that practicing this way of thinking can help lead people into building more solid stories about robust systems, if they already have good ordinary paranoia and some fairly mysterious innate talents.
Continued in:安全心态和后勤成功曲线。
Did you like this post?You may enjoy our otheryabo app posts, including: