We’ve received several thoughtful questions in response to ourfundraising post to the Effective Altruism Forumand our newFAQ。From quant trader Maxwell Fritz:
我对Miri的球场的捕捉反应通常是，“是的，AI是一个真正的关注点。但我不知道Miri是否是合适的人才努力，或者他们对这个问题的方法是正确的人“。
Most of the FAQ and pitch tends to focus on the “does this matter” piece. It might be worth selling harder on the second component – if you agree AI matters, why MIRI?
At that point, there’s two different audiences – one that has the expertise in the field to make a reasoned assessment based on the quality of your existing work, and a second that doesn’t have a clue (me) and needs to see a lot of corroboration from unaffiliated, impressive sources (people in that first group).
The pitches tend to play up famous people who know their shit and corroborate AI as a concern – but should especially make it clear when those people believe in MIRI. That’s what matters for the “ok, why you?” question. And the natural follow up is if all of these megarich people are super on board with the concern of AI, and experts believe MIRI should lead the charge, why aren’t you just overflowing with money already?
And from mathematics grad student Tristan Tager:
我猜想“为什么Miri”，而不是“谁是Miri”或“为什么Ai”，是你们应该解决的最大的营销障碍。
对我来说,“为什么米里”分解成两个问题。The first and lesser question is, what can MIRI do? Why should I expect that the MIRI vision and the MIRI team are going to get things done? What exactly can I expect them to get done? Most importantly in addressing this question, what have they done already and why is it useful? The Technical Agenda is vague and mostly just refers to the list of papers. And the papers don’t help much — those who don’t know much about academia need something more accessible, and those who do know more about academia will be skeptical about MIRI’s self-publishing and lack of peer review.
但第二个和更大的问题是，Miri会让谷歌不会做什么？谷歌拥有大量资金，创意和远见人员，世界上最好的程序员，以及一系列成功的产品，包括一定程度的AI - 而且他们最近收购了几个AI业务并形成了AI道德委员会。似乎他们似乎直接接近了同样的大问题，而不是理论上，并有深口袋，热心，以及丰富的动手体验。
There are a number of good questions here. Later this week, Nate plans to post a response to Tristan’s last question:Why is MIRI currently better-positioned to work on this problem than AI groups in industry or academia?（Update February 17：链接这里。)
在这里，我想回复Tristan和Maxwell提出的其他几个问题：
- How can non-specialists assess MIRI’s research agenda and general competence?
- 我们可以使用什么样的成就作为Miri过去和未来成功的措施？
- 最后：If a lot of people take this cause seriously now, why is there still a funding gap?
一般注意事项
当我们为MIRI做出我们的案例时，我们通常专注于考虑AI对存在风险的重要点的理由（Nate的四个背景索赔, Eliezer’s五个论文) and for thinking that early theoretical progress is possible in this area (MIRI’s Approach）。
We focus on these big-picture arguments because the number of people working on this topic is still quite small. The risk scenario MIRI works on has only risen to national attention in the last 6-12 months; at the moment, MIRI is the only research organization I know of that is evenclaiming专业从事对准问题的早期技术研究。亚博体育官网
There are multiple opportunities to support technical research into long-term AI safety at the scale of “funding individual researcher X to work on discrete project Y.” Some recipients of the 2015 Future of Life Institute (FLI)grants落入这一类别，例如，斯图尔特罗素和保罗基督教。
However, there aren’t multiple opportunities in this area at the scale of “funding an AI group to pursue a large-scale or long-term program,” and there aren’t many direct opportunities to bring in entirely new people and grow and diversify the field. MIRI would love to have organizational competitors (and collaborators) in this space, but they don’t yet exist. We expect this to change eventually, and one of our expansion goals is to make this happen faster, by influencing other math and computer science research groups to take on more AI alignment work and byrecruiting highly qualified specialists from outside the existential risk communityto become career AI alignment researchers.
The upside of getting started early is that we have a chance to have a larger impact in a less crowded space. The downside is that there are fewer authoritative sources to appeal to when outsiders want to verify that our research agenda is on the right track.
For the most part, those authoritative sources will probably need to wait a year or two. Academia is slow, and at this stage many computer scientists have only been aware of this area for a few months. Our mission is seen as important by a growing number of leaders in science and industry, and our technical agenda is seen by a number of AI specialists as promising and deserving of more attention — hence its inclusion in the FLIresearch priorities document。But we don’t expect the current state of the evidence to be universally convincing to our most skeptical potential donors.
对于那些想要对Miri的产出进行详细的独立评估的人来说，我们的建议是等待更广泛的学术界的回应。（我们也在研究直接从独立研究人员直接征求关于我们的研究议程和早期结果的选项。）亚博体育官网
In the interim, however, “Why Now Matters“指出原因捐款米里可能哈ve a much larger impact now than they would several years down the line. For donors who are skeptical (or curious), but are not so skeptical that they require a fine-grained evaluation of our work by the scholarly community, I’ll summarize some of the big-picture reasons to think MIRI’s work is likely to be high-value.1
Influence on the AI conversation
In the absence of in-depth third-party evaluations of our work, interested non-specialists can look at our publication history and the prominence of our analyses in scholarly discussions of AI risk.
Miri最重要的成就分为三类：编写顶级优先权AI对齐问题;开始早期工作;让人们对我们的研究和使命感兴趣。亚博体育官网我们的去年审查discusses our recent progress in formalizing alignment problems, as well as our progress in getting the larger AI community interested in long-term AI safety. Ourpublications listgives a more detailed and long-term picture of our output.
MIRI co-founder and senior researcher Eliezer Yudkowsky and Future of Humanity Institute (FHI) founding director Nick Bostrom are responsible for much of the early development and popularization of ideas surrounding smarter-than-human AI risk. Eliezer’s ideas arecited prominentlyin the 2009 edition ofArtificial Intelligence: A Modern Approach（AI:MA), the leading textbook in the field of AI, and also in Bostrom’s 2014 bookSuperintelligence。
Credit for more recent success in popularizing long-term AI safety research is shared between MIRI and a number of actors: Nick Bostrom and FHI, Max Tegmark and FLI, Stuart Russell (co-author ofAI:MA), Jaan Tallinn, and others. Many people in this existential-risk-conscious cluster broadly support MIRI’s efforts and are in regular contact with us about our decisions. Bostrom, Tegmark, Russell, and Tallinn are all MIRI advisors, and Tallinn, a co-founder of FLI and of the Cambridge Centre for the Study of Existential Risk (CSER), cites MIRI as a key source for his views on AI risk.
Writing in early 2014, Russell and Tegmark, together with Stephen Hawking and Frank Wilczek,注意到The Huffington Postthat “little serious research is devoted to these issues [of long-term AI risk] outside small non-profit institutes such as the Cambridge Center for Existential Risk, the Future of Humanity Institute, the Machine Intelligence Research Institute, and the Future of Life Institute.”2在这些组织中，MIRI是目前专门从事设计安全，有益的令人智能的AI系统的技术障碍的。亚博体育苹果app官方下载FHI，CSER和FLI在更广泛的问题上做重要的工作，包括预测和战略工作，外联和调查其他全球风险。
Publications
特里斯坦提出了Miri的技术议程是自我发表的担忧。审查我们过去的出版物，Miri已经自我发布了许多结果。不止一次，我们有看到论文的经验rejectedwith comments that the results are interesting but the AI motivation is just too strange. We’ve begun submitting stripped-down papers and putting the full versions on arXiv, but figuring out the best way to get these results published took some trial and error.
Part of the underlying problem is that the AI field has been repeatedly burned by “冬天” when past generations over-promised and under-delivered. Members of the field are often uncomfortable looking too far ahead, and have historically been loathe to talk about general intelligence.
我们的方法正是相反的：我们直接关注试图识别一般推理的基本方面not well-understood，同时明确地避免了专注于当前的AI系统（无论我们的努力如何更拥挤，更有亚博体育官网可能发生）。亚博体育苹果app官方下载这意味着我们今天的工作往往缺乏直接的实际应用，同时也能引发一般情报的不受欢迎。
We’re getting more work published these days in part because the topic of smarter-than-human AI is no longer seen as academically illegitimate to the extent it was in the past, and in part because we’ve pivoted in recent years from being an organization that primarily worked on movement growth (via Singularity Summits, writings on rationality, etc.) and some forecasting research, to an organization that focuses solely on novel technical research.
我们的七纸technical agendawas initially self-published, but this was primarily in order to make it available early enough to be read and cited by attendees of the“AI的未来”会议in January. Since then, a较短的版本“朝向理想决策理论”的技术议程纸已被接受于AGI-15（完整版is on arXiv), and we’ve presented the full technical agenda paper “易燃” at AAAI-15, a leading academic conference in AI. The overview paper, “Aligning Superintelligence with Human Interests,” is forthcoming in a Springer anthology on the technological singularity.
The other four technical agenda papers aren’t going through peer review because they’re high-level overview papers that are long on explanation and background, but short on new results. We’ve been putting associated results through peer review instead. We published Vingean reflection results in the AGI-14 proceedings (“自我改善时空嵌入式智能自我引用问题”), and other results have been accepted to ITP 2015 (“Proof-Producing Reflection for HOL“). We have two peer-reviewed papers related to both logical uncertainty and realistic world-models: one we presented two weeks ago at AGI-15 (“Reflective Variants of Solomonoff Induction and AIXI”) and another we’re presenting at LORI-V later this year (“反思性奥卡尔：古典博弈论的基础”). We also presented relevant决策理论结果at AAAI-14.
The “Value Learning” paper is the only paper in the research agenda suite that hasn’t had associated work go through peer review yet. It’s the least technical part of the agenda, so it may be a little while before we have technical results to put through peer review on this topic.
（7月15日更新：“The Value Learning Problem” has now been peer reviewed and presented at the IJCAI 2016 Ethics for Artificial Intelligence workshop.)
通过出版突出的期刊和会议程序，我们希望获得更多研究员对我们的工作感兴趣。亚博体育官网有益的结果是，我们正在努力的基本研究问题会有更多的非Miri评估（和贡献）。亚博体育官网在更近的未来，我们还有一些博客帖子，旨在解释我们的研究议程的一些技术部分，例如Vingean反思。亚博体育官网
In all, we’ve published seven peer-reviewed papers since Nate and Benja came on in early 2014. In response to recent industry progress and new work by MIRI and the existential risk community, it’s become much easier to publish papers that wrestle directly with open AI alignment problems, and we expect it to become even easier over the next few years.
成功是什么样的？
Miri的成功部分意味着在我们身上交付夏季筹款人目标：越来越多的研究团队并参加额外的项目条亚博体育官网件，其中融资目标我们能够击中。我们今年的计划是专注于在精英场地生产一些高质量的出版物。如果我们的筹款人进展顺利，这应该会影响我们如何在该计划上执行的有效性以及我们如何快速生成和发布新结果。
A more direct measure of success is our ability to make progress on the specific technical problems we’ve chosen to focus on, as assessed by MIRI researchers and the larger AI community. In “MIRI’s Approach,” Nate distinguishes two types of problems: ones for which we know the answer in principle, but lack practical algorithms; and ones that are not yet well-specified enough for us to know how to construct an answer even in principle. At this point, large-scale progress for MIRI looks like moving important and neglected AI problems from the second category to the first category.
Maxwell raised one more question: if MIRI is receiving more mainstream attention and approval, why does it still have a funding gap?
Part of the answer, sketched out in “Why Now Matters,” is that we do think there’s a good chance that large grants or donations could close our funding gap in the coming years. However, large donors and grantmakers can be slow to act, and whether or not our funding gap is closed five or ten years down the line, it’s very valuable for us to be able to expand and diversify our activities now.
Elon Musk and the Open Philanthropy Project recently awarded $7M in grants to AI safety research, and MIRI’s core research program received一个大型项目补助金。This is a wonderful infusion of funding into the field, and means that many more academics will be able to start focusing on AI alignment research. However, given the large number of high-quality grant recipients, the FLI grants aren’t enough to make the most promising research opportunities funding-saturated. MIRI received the fourth-largest project grant, which amounts to $83,000 per year for three years.3.这是一个非常慷慨的补助金，它将大大加强我们支持研究人员和运行研讨会的努力，但它无处可去，以关闭我们的资金缺口。亚博体育官网
Since this is the first fundraiser we’ve run in 2015, it’s a bit early to ask why the newfound attention and approval our work has received this year hasn’t yet closed the gap. The FLI grants and our ongoing fundraiser are part of the mechanism by which the funding gap shrinks. It is shrinking, but the process isn’t instantaneous — and part of the process is making our case to new potential supporters. Our hope is that if we make our case for MIRI clearer to donors, we can close our funding gap faster and thereby have a bigger impact on the early scholarly conversation about AI safety.
We’re very grateful for all the support we’ve gotten so far — and in particular for the support we received before we had mainstream computer science publications, a fleshed-out research agenda, or a track record of impacting the discourse around the future of AI. The support we received early on was critical in getting us to where we are today, and as our potential as an organization becomes clearer through our accomplishments, we hope to continue to attract a wider pool of supporters and collaborators.
- 一如既往，你也可以shoot us more specific questionsthat aren’t addressed here.↩
- QuotingRussell，Tegmark，Hawking和Wilczek：
Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all.
So, facing possible futures of incalculable benefits and risks, the experts are surely doing everything possible to ensure the best outcome, right? Wrong. If a superior alien civilization sent us a text message saying, “We’ll arrive in a few decades,” would we just reply, “OK, call us when you get here — we’ll leave the lights on”? Probably not — but this is more or less what is happening with AI. Although we are facing potentially the best or worst thing ever to happen to humanity, little serious research is devoted to these issues outside small non-profit institutes such as the Cambridge Center for Existential Risk, the Future of Humanity Institute, the Machine Intelligence Research Institute, and the Future of Life Institute.↩
- We additionally received about $50,000 for theAI Impactsproject, and will receive some fraction of the funding from two other grants where our researchers are secondary investigators, “Inferring Human Values” and “适用正式验证对反思性推理。”↩
Did you like this post?You may enjoy our otheryabo app posts, including:
