约翰Fox on AI safety

卢克·穆尔豪瑟（Luke Muehlhauser）：您花了很多年的时间研究AI安全问题，尤其是在医疗环境中，例如在您与Subrata Das的2000年书中，Safe and Sound: Artificial Intelligence in Hazardous Applications。What kinds of AI safety challenges have you focused on in the past decade or so?

约翰Fox: From my first research job, as a post-doc with AI founders Allen Newell and Herb Simon at CMU, I have been interested in computational theories of high level cognition. As a cognitive scientist I have been interested in theories that subsume a range of cognitive functions, from perception and reasoning to the uses of knowledge in autonomous decision-making. After I came back to the UK in 1975 I began to combine my theoretical interests with the practical goals of designing and deploying AI systems in medicine.

自从我们的书籍于2000年出版以来，我一直致力于通过设计和部署多种临床系统来测试其中的想法，并证明AI技术可以显着提高临床决策和过程管理的质量和安全性。亚博体育苹果app官方下载患者的安全是临床实践的基础，因此，除了可以改善人类绩效，安全性和道德规范的目标的目标外，始终是我研究议程的顶端。亚博体育官网亚博体育苹果app官方下载

卢克·穆尔豪瑟（Luke Muehlhauser）：Was it straightforward to address issues like safety and ethics in practice?

约翰Fox: While our concepts and technologies have proved to be clinically successful we have not achieved everything we hoped for. Our attempts to ensure, for example, that practical and commercial deployments of AI technologies should explicitly honor ethical principles and carry out active safety management have not yet achieved the traction that we need to achieve. I regard this as a serious cause for concern, and unfinished business in both scientific and engineering terms.

The next generation of large-scale knowledge based systems and software agents that we are now working on will be more intelligent and will have far more autonomous capabilities than current systems. The challenges for human safety and ethical use of AI that this implies are beginning to mirror those raised by the singularity hypothesis. We have much to learn from singularity researchers, and perhaps our experience in deploying autonomous agents in human healthcare will offer opportunities to ground some of the singularity debates as well.

Luke: You write that your “attempts to ensure… [that] commercial deployments of AI technologies should… carry out active safety management” have not yet received as much traction as you would like. Could you go into more detail on that? What did you try to accomplish on this front that didn’t get adopted by others, or wasn’t implemented?

约翰: Having worked in medical AI from the early ‘seventies I have always been keenly aware that while AI can help to mitigate the effects of human error there is a potential downside too. AI systems could be programmed incorrectly, or their knowledge could prescribe inappropriate practices, or they could have the effect of deskilling the human professionals who have the final responsibility for their patients. Despite well-known limitations of human cognition people remain far and away the most versatile and creative problem solvers on the planet.

In the early ‘nineties I had the opportunity to set up a project whose goal was to establish a rigorous framework for the design and implementation of AI systems for safety critical applications. Medicine was our practical focus but the RED project¹was aimed at the development of a general architecture for the design of autonomous agents that could be trusted to make decisions and carry out plans as reliably and safely as possible, certainly to be as competent and hence as trustworthy as human agents in comparable tasks. This is obviously a hard problem but we madesufficient progresson theoretical issues and design principles that I thought there was a good chance the techniques might be applicable in medical AI and maybe even more widely.

我以为AI就像医学一样，我们都认为医疗设备和制药公司有责任表明他们的产品有效且安全，然后才能获得商业用途的认证。我还认为，AI研究人员同样会认识到，我们对在安亚博体育官网全关键环境中受到不良工程或滥用影响的所有潜在影响的人有“护理义务”，但这很幼稚。基于AI研究的技术的商业工具迄今为止的重点是仅仅获得和保持客户和安全始终取得后排座位。亚博体育官网

回想起来，我应该预测，确保AI产品安全不会吸引商业供应商的热情。如果您将AI应用程序与药物进行比较，我们都知道必须对制药公司进行牢固的监管，以确保它们履行对客户和患者的护理义务。但是，证明药物是安全的，而且还冒着揭示您的新奇迹毒品甚至没有您声称的效果的风险！AI也一样。

我仍然会惊讶于软件开发人员的乐观性 - 他们似乎总是充满信心，最糟糕的情况不会发生，或者如果他们确实发生了，那么他们的管理是别人的责任。过去，这种技术过度自信导致了过去无数的灾难，令我惊讶的是它坚持不懈。

这有另一块,担忧roles and responsibilities of AI researchers. How many of us take the risks of AI seriously so that it forms a part of our day-to-day theoretical musings and influences our projects? MIRI has put one worst case scenario in front of us – the possibility that our creations might one day decide to obliterate us – but so far as I can tell the majority of working AI professionals either see safety issues as irrelevant to the pursuit of interesting scientific questions or, like the wider public, that the issues are just science fiction.

I think experience in medical AI trying to articulate and cope with human risk and safety may have a couple of important lessons for the wider AI community. First we have a duty of care that professional scientists cannot responsibly ignore. Second, the AI business will probably need to be regulated, in much the same way as the pharmaceutical business is. If these propositions are correct then the AI research community would be wise to engage with and lead on discussions around safety issues if it wants to ensure that the regulatory framework that we get is to our liking!

Luke：现在您写道：“过去，这种技术过度自信导致了无数的灾难……”您想到的“灾难”有哪些例子？

约翰: Psychologists have known for years that human decision-making is flawed, even if amazingly creative sometimes, and overconfidence is an important source of error in routine settings. A large part of the motivation for applying AI in medicine comes from the knowledge that, in the words of the Institute of Medicine, “To err is human” and overconfidence is an established cause of clinical mistakes.²

Over-confidence and its many relatives (complacency, optimism, arrogance and the like) have a huge influence on our personal successes and failures, and our collective futures. The outcomes of the US and UK’s recent adventures around the world can be easily identified as consequences of overconfidence, and it seems to me that the polarized positions about global warming and planetary catastrophe are both expressions of overconfidence, just in opposite directions.

Luke：走得更远……如果有一天我们可以设计AGIs, do you think we are likely to figure out how to make them safe?

约翰: History says that making any technology safe is not an easy business. It took quite a few boiler explosions before high-pressure steam engines got their iconic centrifugal governors. Ensuring that new medical treatments are safe as well as effective is famously difficult and expensive. I think we should assume that getting to the point where an AGI manufacturer could guarantee its products are safe will be a hard road, and it is possible that guarantees are not possible in principle. We are not even clear yet what it means to be “safe”, at least not in computational terms.

显而易见的是，入门级机器人产品（例如进行简单的家务机器人或正在试用医院使用的“护士机器人”）的机器人产品如此简单的行为曲目，以至于不难设计其软件控制器以使其不难在大多数可能的情况下安全运营。标准安全工程技术，例如榛³are probably up to the job I think, and where software failures simply cannot be tolerated software engineering techniques like formal specification and model-checking are available.

围绕更具挑战性的机器人应用（例如自动驾驶汽车和医疗机器人技术）也有很多乐观。Moustris等。⁴假设自动手术机器人正在出现，可以在各种角色中使用，以自动化的角色在诸如开心手术之类的复杂手术中自动化，他们希望它们成为标准的，并彻底改变手术的实践。但是，在这一点上，在我看来，具有重要认知曲目的外科机器人是可行的，并且在可预见的将来，人类外科医生将处于循环状态。

Luke：那么，人工智能可以从自然智能中学到什么？

作为一名从事医学工作的认知科学家，我的兴趣与从事AGIS的科学家共同扩展。医学是一个如此庞大的领域，可以安全地练习它，需要处理无数临床方案和互动的能力，即使在单个专家子领域工作也需要从其他子领域进行大量知识。如此之多，以至于现在众所周知，即使是经验丰富的临床曲目的非常有经验的人也会遭受明显的错误水平。⁵可以在医学上有所帮助的人工智能将需要极大的多功能性，这将需要对医学专业知识以及一系列认知能力（例如推理，决策，计划，沟通，反思，学习等）有一般的了解。

If human experts are not safe is it well possible to ensure that an AGI, however sophisticated, will be? I think that it is pretty clear that the range of techniques currently available for assuring system safety will be useful in making specialist AI systems reliable and minimizing the likelihood of errors in situations that their human designers can anticipate. However, AI systems with general intelligence will be expected to address scenarios and hazards that are beyond us to solve currently and often beyond designers even to anticipate. I am optimistic but at the moment I don’t see any convincing reason to believe that we have the techniques that would be sufficient to guarantee that a clinical super-intelligence is safe, let alone an AGI that might be deployed in many domains.

Luke: Thanks, John!

1. Rigorously Engineered Decisions↩
2. Overconfidence in major disasters:

   •D. Lucas.Understanding the Human Factor in Disasters.Interdisciplinary Science Reviews. Volume 17 Issue 2 (01 June 1992), pp. 185-190.
   •“核安全和安全。

   Psychology of overconfidence:

   •Overconfidence effect.
   •C。Riordan。Three Ways Overconfidence Can Make a Fool of YouForbes Leadership Forum.

   Overconfidence in medicine:

   •R. Hanson.过度自信消除了DOC优势。克服偏见，2007年。
   •E。Berner，M。Graber。过度自信是医学诊断错误的原因。The American Journal of Medicine. Volume 121, Issue 5, Supplement, Pages S2–S23, May 2008.
   •T. Ackerman.Doctors overconfident, study finds, even in hardest cases.Houston Chronicle, 2013.

   一般技术示例：

   •J. Vetter, A. Benlian, T. Hess.Overconfidence in IT Investment Decisions: Why Knowledge can be a Boon and Bane at the same Time.ICIS 2011诉讼。论文4. 2011年12月6日。↩

3. Hazard and operability study↩
4. Int J Med Robotics Comput Assist Surg 2011;7：375–39↩
5. A. Ford.Domestic Robotics – Leave it to Roll-Oh, our Fun loving Retrobot。道德与新兴技术学院，2014年。↩