达vid J. Atkinson on autonomous systems

卢克·穆罕沃斯：你的一个项目之一IHMCis “仁慈在自治系统信赖中的作用亚博体育苹果app官方下载“：

The exponential combinatorial complexity of the near-infinite number of states possible in autonomous systems voids the applicability of traditional verification and validation techniques for complex systems. New and robust methods for assessing the trustworthiness of autonomous systems are urgently required if we are to have justifiable confidence in such applications both pre-deployment and during operations… The major goal of the proposed research is to operationalize the concept of benevolence as it applies to the trustworthiness of an autonomous system…

Some common approaches for ensuring desirable behavior from AI systems include testing, formal methods, hybrid control, and simplex architectures. Where does your investigation of “benevolence” in autonomous systems fit into this landscape of models and methods?

达vid J. Atkinson: First let me point out that test, formal methods and such other techniques that you point out have little to do with ensuring desirable behavior and more to do with avoiding errors in behavior due to design or implementation flaws. These are not equivalent. Existing techniques improve reliability, but that is only one component of trust and it only concerned with behavior “as designed”. Furthermore, as your quote from my material points out, when it comes to the near-infinite state spaces of autonomy, those “common approaches” are inherently limited and cannot make the same strong claims regarding system behavior that they could with machines where the envelope of behavior could be completely known.

Therefore, I chose to look for answers regarding trust, trustability and trustworthiness in the operations phase of the autonomous system lifecycle because it is here, not in testing and evaluation, where the full complexity of autonomous behavior will manifest itself in response to the uncertainty, dynamics, and real-world complexity that is its major strength. My approach is to focus is on the applicability of human interpersonal trust to operation of autonomous systems. The principle reasons for this are 1) Autonomy is limited — humans make the decisions to rely upon an intelligent agent, subject to a variety of individual and situational factors and constraints, and; 2) Eons of evolution have created a reasonably good mechanism in humans for trust. It is fundamental to every human social transaction and it works. Beyond reliability, studies in multiple disciplines have shown that people want evidence of capability, predictability, openness (aka transparency), and safety before granting a measure of trustworthiness to a machine. Key questions revolve around the nature of that evidence, how it is provided, and what inferences can be reasonably made. People use both cognitive and affective mental processes for evaluating trustworthiness. My central claim is that if we can reverse engineer these mechanisms and make intelligent autonomous agents (IATs) that are compliant with the human trust process (two big ifs), then we will have created a new way for humans to have trusting relationships with they machines they rely upon. It will be a transition from seeing IATs as tools to treating them as partners.

仁慈是有趣的几个原因。首先，它是一个复杂的归属，建立了一个关于包括善意，能力，可预测性，缺乏隐藏议程，机构和其他信仰的另一个人的信仰结构，大部分可能在多种相互作用中发挥作用人类和IAT。其次，只是看着信仰列表使它成为一个艰难的问题，尽管我的同事认为人们将没有问题归咎于机器。第三，IAT的重要应用，也许体现为机器人，其中仁慈的独特心理对成功至关重要。例如，灾难救援。众所周知，灾民受害者具有独特的心理学，对这些唤起的心理，恐惧和心理生理影响。人类的第一响应者接受了对受害者心理学的特殊培训。其中一个原因是受害者，有理由害怕他们的生命，可能非常不愿意相信救援人员而没有信任，没有合作，救援可能变得非常困难。仁慈似乎是所需信任的一部分。 Today, we have no idea whatsoever whether a real disaster victim will trust and cooperate with a robot rescuer.

Circling back to your question about “fitting in to the landscape of models and methods”, the ultimate goal of my research is to formulate design requirements, interaction methods, operations concepts, guidelines and more that, if followed, will result in an IAT that can itself engender well-justified human trust.

Luke: As you say, formal methods and other techniques are of limited help in cases where we don’t know how to formulate comprehensive and desirable design requirements, as is often the case for autonomous systems operating in unknown, dynamic environments. What kinds of design requirements might your approach to trustworthy systems suggest? Would these be formal design requirements, or informal ones?

达vid：今天我们可以做些什么，我们可以在与未知的动态环境相关的具体要求中进行任何情况。我们的目标之一是缩小这些差距，因此可以研究精确的问题。挥动手和呻吟的事情是不确定性的一件事，另一件事完全是为了做到这一点。

Generally speaking, we are working towards formal specification of the traditional types of requirements: Functional, Performance, Design Constraint, and Interface (both Internal and External). By “formal”, I mean “complete according to best practices”. I do not mean “expressed in formal logic or according to a model-based language” — that is a step beyond. Requirements may be linked in numerous ways to each other forming a directed graph (hopefully with no cycles!). Relationships include Source, Required-By, Depends-upon and so forth.

For example, an attribution of benevolence by a human “Trustor” requires a belief (among numerous others) that the candidate “Trustee” (the autonomous system) has “no hidden ill will”. This is a very anthropomorphic concept that some might scoff at, but studies have demonstrated its importance to attribution of benevolence and so it is a factor we must reckon in designing trustworthy and trustable autonomous systems. But what does it even mean?

只是为了让你了解我们如何破坏这一点，这是一些派生要求。I should emphasize that it is very premature to make any claims about the quality or completeness of what we done with requirements engineering thus far — mostly that work is on the schedule for next year so I’ll only give you the types and titles: We will have as a matter of course a generic Level 1 Functional requirement to “Provide Information to Human User”. Derived from this is an Interface requirement something like “Volunteer Information: The Autonomous System shall initiate communication and provide information that is import to humans” (actually, that is two requirements). This in turn is elaborated by a number of Design Constraints such as “Disposition: The Autonomous System shall disclose any disposition that could result in harm to the interests of a human”. These Design Constraints are in turn linked to numerous other detailed requirements such as this Performance Requirement “Behavior – Protective: The Autonomous System shall recognize when its behavior could result in harm to the interests of a human”. A designer of an autonomous system will recognize a number of very hard problems in this simple example that need to be solved to effectively address the hypothetical need for a benevolent autonomous system.. Our focus in this project is on “what” needs to be done, not “how” to do it.

毫无疑问，此要求过程将产生大量问题，需要进一步研究。一些要求可能会发生冲突，并且任何特殊应用都必须使权衡研究能够优先考虑。尽管如此，我们的目标就是尽可能多地拼出。我们将区分从目标或目标的强制性要求。在可能的情况下，我们将对自治系统架构的特定元素分配要求，例如“目标选择机制”（可能会出现与优先级相关的问题并影响预测性并因此相信）。亚博体育苹果app官方下载对于各种要求，我们将提供一个基本的研究和经验数据或其他可以用于分析的讨论。那部分在我的脑海中是非常高的优先级。我遇到了太多次，所以不可能理解它们是如何衍生的。我还想采取风险驱动的要求，以便个人要求或要求组，可以与特定的风险相关联。这是必须通过进一步的应用程序特定分析来量化的另一个区域。 Finally, a good requirement has to be verifiable. There is considerable work to be done on this topic with respect to autonomous systems.

Requirements engineering is a lot of work of course, and the history of large scale system development is replete with horror stories of poor requirements. That’s why I want to express our trust-related requirements formally following requirements engineering standards the extent possible. From my previous experience at NASA, I know that the less work a project has to do, the more likely it is to to adopt existing requirements. So we are developing our trust-related requirements consciously with the goal of making them easy to understand and easy to reuse. Finally, given the scope of what is required it is likely that we will only be able to go just so far under the auspices of our current project to provide a vector for future work (hint hint to potential sponsors out there!)

Luke: From your description, this research project seems somewhat interdisciplinary, and the methodology seems less clear-cut than is the case with many other lines of research aimed at similar goals (e.g. some new project to model-check a particular software design for use in robotics). It’s almost “pre-paradigmatic,” in the Kuhnian sense. Do you agree? If not, are there are other research groups who are using this methodology to explore related problems?

达vid: Yes, the project is very interdisciplinary but with the primary ones being social and cognitive psychology, social robotics, and artificial intelligence we well as the rigor contributed by solid systems engineering. The relevant content within each of those disciplines can be quite broad. It is not a small undertaking. My hope is that I can plant some memes in each community to help bring them together on this topic of trust and these memes will foster further work. As far as methodology, we are pursuing both theoretical development and experimentation, and trying to be as rigorous as exploratory work of this nature will permit. We have to understand the previous psychological work on human interpersonal trust, and human factors studies on human-automation trust, to find those results that may have important implications for human trust of an intelligent, autonomous agent. The importance of agency is a good example.

We know from psychological studies that an attribution of “free-will”, or more narrowly, the ability to choose, is an important component of deciding whether someone else is benevolent or not. That is, if a person feels the other is compelled to help then the less likely they are to believe that other person is benevolent. Apart from philosophers, most people don’t think very deeply about this. They make a presumption of free-will and then look to see if there are reasons it is limited, for example, is the person just following orders, or required by their profession or social norms to act in a particular way? With machines, we start from the other side: people assume machines have no free-will because they believe machines are (just) programmed. However, there are一些研究这表明，当机器行为复杂（足够），并且有点不可预测，因为有许多可能的行动方案，人们开始归因于精神状态，包括选择的能力。我的亨希是这是对我们天生的需求的诱发回应，以解释他人在意向性框架中的行为。在某些时候，提供足够的功能，即那些进化设计的启发式机会努力帮助理解。为什么可能是，我会留给进化的社会生物学家。

Back to methodology now: This is a phenomena for which we can design a study involving humans and machines, with systematic variation of various factors to see what qualities a machine actually requires in order to evoke a human attribution of the ability to choose to help. This year we will be conducting just such a study. I hope to begin running participants this summer. We have been rigorous with experimental design, choice of what data to collect and the statistical methods we will use to analyze the results. While we will fudge a little bit on the robot implementation, using simulation in places and “wizard-of-oz” techniques for language interaction for example, the products of the study ought to be recognized as solid science by researchers in multiple disciplines if we do it right. In general, I think this a very hard goal to achieve because each discipline community has certain preferences and biases about what they like to see in methodology before they are convinced. There are a couple of other groups working on social robotics who use this approach, and a very few number of psychologists who are working on human-centered design of automation. I don’t want to start listing names because I’m sure there are others of whom I’m not yet aware. I do know that I am certainly not the first to confront this challenge. Multidisciplinary research of this type is always, in some sense, pre-paradigmatic because it is a struggle for understanding and legitimacy at the boundaries of separate disciplines, not the core. Artificial intelligence has always been multidisciplinary, a strength as well as a weakness as far as more general acceptance among related disciplines. I don’t worry too much about what other people think. I just do what I believe has to be done.

Luke:什么是你希望将一些具体的结果achieved during the next 5 years of this kind of research?

达vid: As a wise man once said, “It’s hard to make predictions, especially about the future.”

My hopes. This one is concrete to me, but perhaps not what you had in mind: I hope that the value of our approach will be convincingly demonstrated and other research groups and early-career researchers will join in. There is much to be explored and plenty of opportunity to make discoveries that can have a real impact.

平衡，似乎在许多潜在的应用领域，最大的挑战是不信任或缺乏信任，而是过度信任。大多数情况下，人们在大多数情况下都没有造成困难的信任机器，因为这有点涉及世代态度，这种情况可能会增加。有时这会导致过度依赖和自满，然后如果事情变得酸味，那么令人惊讶（和潜在的危险）条件。一个有趣但真实的例子是驾驶员在长长的一条直的高速公路上设置巡航控制，走出驾驶员座位，然后回来制作三明治。你可以猜出发生了什么。有效的团队合作要求每个团队成员都理解其他人的优势和局限性，而且不会发生过度依赖。智能化，自主队友需要同样的能力在这一基本的团队建设过程中有效地参与 - 这一过程需要建立相互熟悉程度的时间和经验。我们将为该解决方案做出贡献。

I believe our work will lead directly to an understanding of When, Why, What, and (some of) How a machine needs to interact with human teammates to inoculate against (and/or correct for) under- and over-reliance. This technology is key for solving what many people claim is a lack of transparency in intelligent systems. It will help “users” to better understand the competency of a machine (the most important quality) in a given context consisting of dynamic situational factors, tasks and goals. This will in turn increase predictability (another important quality) and thereby help mitigate concerns about risks and safety. Ultimately, a deep solution that is broadly applicable will require a higher degree of machine meta-reasoning and self-awareness than we can engineer today, but this is an area of active research where useful results ought to be appearing more and more frequently. (The field of cognitive (developmental) robotics is very exciting.) However, I do expect concrete and useful results for early applications in some semi-structured task domains. A few examples of domains containing “low hanging fruit” for applications are transportation (e.g., autonomy-assisted driving, long haul trucking), healthcare (patient monitoring, therapy assistance, assisted living), and some defense-related applications. My group is actively working towards all of these possibilities. I don’t want to leave you with the impression that creating effective applications will be easy because many hard basic research challenges remain, and we will undoubtedly discover others when we start to transition the technology into real-world applications. Nevertheless, I’m optimistic!

Luke: Thanks, David!