新论文：“ Cirl框架中的不可验证”

||文件

Incorrigibility in the CIRL Framework

MIRI assistant research fellow Ryan Carey has a new paper out discussing situations where good performance inCooperative Inverse Reinforcement Learning(CIRL) tasks fails to imply that software agents will assist or cooperate with programmers.

这paper, titled “Incorrigibility in the CIRL Framework，”提出了四种情况，其中cirl违反了四个条件符合条件定义Soares等。(2015)。Abstract:

价值学习系统有激励措施遵循关闭说明亚博体育苹果app官方下载，假设关闭指令提供了有关哪些操作导致有价值结果的信息（从技术意义上讲）。但是，此假设对于模拟错误指定的模型并不强（例如，在程序员错误的情况下）。我们通过提出一些有监督的POMDP方案来证明这一点，其中参数化奖励功能中的错误删除了遵循关闭命令的激励措施。这些困难与Soares等人讨论的困难相似。（2015年）在他们的有关科罗的论文中。

We argue that it is important to consider systems that follow shutdown commands under some weaker set of assumptions (e.g., that one small verified module is correctly implemented; as opposed to an entire prior probability distribution and/or parameterized reward function). We discuss some difficulties with simple ways to attempt to attain these sorts of guarantees in a value learning framework.

该论文是对Hadfield-Menell，Dragan，Abbeel和Russell的论文的回应，“开关游戏。” Hadfield-Menell et al. show that an AI system will be more responsive to human inputs when it is uncertain about its reward function and thinks that its human operator has more information about this reward function. Carey shows that the CIRL framework can be used to formalize the problem of corrigibility, and that the known assurances for CIRL systems, given in “The Off-Switch Game”, rely on strong assumptions about having an error-free CIRL system. With less idealized assumptions, a value learning agent may have beliefs that cause it to evade redirection from the human.

关闭按钮的目的是关闭AI系统亚博体育苹果app官方下载如果所有其他保证都失败了, e.g., in the event that the AI system is ignoring (for one reason or another) the instructions of the operators. If the designers of [the AI system]r已经完美地编程了系统，以至于先验和[奖励亚博体育苹果app官方下载功能]r完全没有错误，然后是Hadfield-Menell等人的定理。（2017年）确实适用。实际上，这意味着为了变得可验证，有必要拥有一个不确定所有可能重要的事物的AI系统。亚博体育苹果app官方下载问题在于，在所有可能的世界上执行贝叶斯推理，所有可能的价值函数都非常棘手。实际上，人类可能必须使用大量的启发式方法和近似值来实施系统的信仰系统和更新规则。亚博体育苹果app官方下载[…]

Soares等。（2015年）似乎想要一个可以作为最后一个度假胜地机制的关闭按钮，以便在观察并拒绝程序员建议的情况下关闭AI系统（程序员认为该系统正在发生故障）。亚博体育苹果app官方下载清楚地，一些part of the system must be working correctly in order for us to expect the shutdown button to work at all. However, it seems undesirable for the working of the button to depend on there being zero critical errors in the specification of the system’s prior, the specification of the reward function, the way it categorizes different types of actions, and so on. Instead, it is desirable to develop a shutdown module that is small and simple, with code that could ideally be rigorously verified, and which ideally works to shut the system down even in the event of large programmer errors in the specification of the rest of the system.

为了在价值学习框架中执行此操作，我们需要一个价值学习系统，该系统（i）能够将其操作覆盖，该模块被关闭命令的小型验证模块；亚博体育苹果app官方下载（ii）没有动力去除，损坏或忽略关闭模块；（iii）有一些小动力来保持其关闭模块；即使在广泛的情况下r，先验，一组可用动作等。

即使学习了公用事业功能，仍然需要对意外失败的额外防御。希望这可以通过对AI系统进行模块化来实现。亚博体育苹果app官方下载为此，我们将需要一个代理的模型，该模型将以符合其他系统组件的指定性能的方式进行可检修。亚博体育苹果app官方下载

Sign up to get updates on new MIRI technical results

Get notified every time a new technical paper is published.

August 2017 Newsletter

||Newsletters

research updates

General updates

News and links

July 2017 Newsletter

||Newsletters

一批重大Miri中期更新：迄今为止，我们收到了最大的捐款，从以太坊投资者那里获得了101万美元！我们的研亚博体育官网究优先事项也有所改变，反映了四位新的全职研究人员（Marcello Herreshoff，Sam Eisenstat，Tsvi Benson-Tilsen和Abram Demski）以及Patrick Lavictoire和Jessica Taylor的离开。

research updates

General updates

News and links

Updates to the research team, and a major donation

||News

We have several major announcements to make, covering new developments in the two months since our2017 strategy update:

1. 5月30日，我们收到了惊喜$1.01 million donationfrom anEthereumcryptocurrency investor. This is the single largest contribution we have received to date by a large margin, and will have a substantial effect on our plans over the coming year.

2.Two new full-time researchersare joining MIRI: Tsvi Benson-Tilsen and Abram Demski. This comes in the wake of Sam Eisenstat and Marcello Herreshoff’s addition to the team在五月。我们还开始与工程师进行试用，为我们的新板块合作software engineer job openings。

3。Two of our researchers have recently left: Patrick LaVictoire and Jessica Taylor, researchers previously heading work on our “高级机器学习系统的对齐亚博体育苹果app官方下载”研亚博体育官网究议程。

For more details, see below.

read more »

2017年6月通讯

||Newsletters


research updates

General updates

News and links

May 2017 Newsletter

||Newsletters

research updates

General updates

  • Ourstrategy update讨论了我们的AI预测和研究重点，新的外展目标，Miri/DeepMind合作以及其他亚博体育官网新闻的变化。
  • MIRI is hiring software engineers!If you’re a programmer who’s passionate about MIRI’s mission and wants to directly support our research efforts,在这里申请与我们一起审判。
  • MIRI Assistant Research Fellow Ryan Carey has taken on an additional联系有了存在生存风险的中心，也正在帮助编辑Informaticaon superintelligence.

News and links

2017 Updates and Strategy

||美里战略

In our last strategy update (2016年8月), Nate wrote that MIRI’s priorities were to make progress on our代理基础agenda and begin work on our new “高级机器学习系统的对齐亚博体育苹果app官方下载” agenda, to collaborate and communicate with other researchers, and to grow our research and ops teams.

Since then, senior staff at MIRI have reassessed their views on how far offartificial general intelligence(AGI) is and concluded that shorter timelines are more likely than they were previously thinking. A few lines of recent evidence point in this direction, such as:1

  • AI research is becoming more visibly exciting and资金充足。这表明，更多的顶尖人才（在下一代和当前一代）可能会将他们的注意力转移到AI上。
  • Agi吸引了更多的学术关注，这是最高AI组的既定目标DeepMind,OpenAI, andFAIR。In particular, many researchers seem more open to thinking about general intelligence now than they did a few years ago.
  • research groups associated with AGI are showing much clearer externalsignsof profitability.
  • AI成功Alphagoindicate that it’s easier to outperform top humans in domains like Go (without any new conceptual breakthroughs) than might have been expected.2这降低了我们对与其他领域中人类相抗衡的重大概念突破数量的估计。

这re’s no consensus among MIRI researchers on how long timelines are, and our aggregated estimate puts medium-to-high probability on scenarios in which the research community hasn’t developed AGI by, e.g., 2035. On average, however, research staff now assign moderately higher probability to AGI’s being developed before 2035 than we did a year or two ago. This has a few implications for our strategy:

1. Our relationships with current key players in AGI safety and capabilities play a larger role in our strategic thinking. Short-timeline scenarios reduce the expected number of important new players who will enter the space before we hit AGI, and increase how much influence current players are likely to have.

2. Our research priorities are somewhat different, since shorter timelines change what research paths are likely to pay out before we hit AGI, and also concentrate our probability mass more on scenarios where AGI shares various features in common with present-day machine learning systems.

这两个更新都代表我们已经流行的方向，原因是各种原因。3However, we’re moving in these two directions more quickly and confidently than we were last year. As an example, Nate is spending less time on staff management and other administrative duties than in the past (having handed these off to MIRI COO Malo Bourgon) and less time on broad communications work (having delegated a fair amount of this to me), allowing him to spend more time on object-level research, research prioritization work, and more targeted communications.4

I’ll lay out what these updates mean for our plans in more concrete detail below.

read more »

  1. Note that this list is far from exhaustive.↩
  2. 相对一般的算法（加上大量的计算）能够超越人类的表现，从无法赢得标准游戏中最糟糕的人类专业人士到dominating the very best professionalsin the space of a few months. The relevant development here wasn’t “AlphaGo represents a large conceptual advance over previously known techniques,” but rather “contemporary techniques run into surprisingly few obstacles when scaled to tasks as pattern-recognition-reliant and difficult (for humans) as professional Go”.↩
  3. 这publication of “Concrete Problems in AI Safety例如，去年，去年使我们减少了我们在基于广泛的AI社区的大规模宣传的时间，以花费更多的时间与我们在OpenAI，Google Brain，DeepMind和其他地方认识的研究人员建立更牢固的合作亚博体育官网。↩
  4. Nate continues to set MIRI’s organizational strategy, and is responsible for the ideas in this post.↩

Software Engineer Internship / Staff Openings

||News

这机器情报研究所亚博体育官网is looking for highly capable software engineers to directly support ourAI alignment亚博体育官网研究工作，重点是与机器学习有关的项目。我们正在寻找具有强大编程技能的工程师，他们对Miri的使命充满热情，并寻求具有挑战性和智力吸引力的工作。

While our goal is to hire full-time, we are initially looking for paid interns. Successful internships may then transition into staff positions.

关于实习计划

这start time for interns is flexible, but we’re aiming for May or June. We will likely run several batches of internships, so if you are interested but unable to start in the next few months, do still apply. The length of the internship is flexible, but we’re aiming for 2–3 months.

Examples of the kinds of work you’ll do during the internship:

  • replicate recent machine learning papers, and implement variations.
  • Learn about and implement machine learning tools (including results in the fields of deep learning, convex optimization, etc.).
  • run various coding experiments and projects, either independently or in small groups.
  • rapidly prototype, implement, and test AI alignment ideas related to machine learning (after demonstrating successes in the above points).

For MIRI, the benefit of this program is that it’s a great way to get to know you and assess you for a potential hire. For applicants, the benefits are that this is an excellent opportunity to get your hands dirty and level up your machine learning skills, and to get to the cutting edge of the AI safety field, with a potential to stay in a full-time engineering role after the internship concludes.

Our goal is to trial many more people than we expect to hire, so our threshold for keeping on engineers long-term as full staff will be higher than for accepting applicants to our internship.

这Ideal Candidate

Some qualities of the ideal candidate:

  • Extensive breadth and depth of programming skills. Machine learning experience is not required, though it is a plus.
  • 非常熟悉与AI一致性有关的基本思想。
  • Able to work independently with minimal supervision, and in team/group settings.
  • Willing to accept a below-market rate. Since MIRI is a non-profit, we can’t compete with the Big Names in the Bay Area.
  • 热衷于在Miri工作并帮助AI Alignment的领域。
  • 不寻找“通用”软件工程位置。

Working at MIRI

我们努力使在Miri工作成为有意义的经历。

  • 现代工作空间 - 我们中的许多人都有可调节的站台，并带有大型外部监视器。我们认为工作空间人体工程学很重要，并尝试操纵工作站以使其尽可能舒适。我们的办公室还提供免费的零食，饮料和餐点。
  • 灵活的工作时间,我们没有严格的办公时间, and we don’t limit employees’ vacation days. Our goal is to make rapid progress on our research agenda, and we would prefer that staff take a day off than that they extend tasks to fill an extra day.
  • 居住在湾区 - 美里办公室位于加利福尼亚州伯克利市中心。从我们的办公室，您可以步行30秒即可到达BART（湾区快速运输），这可以使您绕过湾区。步行3分钟即可到达加州大学伯克利分校校园；yabo体育官网还有30分钟的巴特乘车前往旧金山市中心。

EEO & Employment Eligibility

MIRI is an equal opportunity employer. We are committed to making employment decisions based on merit and value. This commitment includes complying with all federal, state, and local laws. We desire to maintain a work environment free of harassment or discrimination due to sex, race, religion, color, creed, national origin, sexual orientation, citizenship, physical or mental disability, marital status, familial status, ethnicity, ancestry, status as a victim of domestic violence, age, or any other status protected by federal, state, or local laws.

申请

如果有兴趣，点击此处申请。For questions or comments, emailengineering@www.hdjkn.com。

Update (December 2017): We’re now putting less emphasis on finding interns and looking for highly skilled engineers available for full-time work.此处更新的工作发布。