MIRI researcher Evan Hubinger reviews “11 different proposals for building safe advanced AI under the current machine learning paradigm”, comparing them on outer alignment, inner alignment,training competitiveness, and performance competitiveness.

Other updates

• We keep being amazed by new shows of support ⁠— following ourlasttwoannouncements, MIRI has received a donation from another anonymous donor totaling ~$265,000 in euros, facilitated byEffective Giving UKand theEffective Altruism Foundation. Massive thanks to the donor for their generosity, and to both organizations for their stellar support for MIRI and other longtermist organizations!
• Hacker NewsdiscussesEliezer Yudkowsky'sThere's No Fire Alarm for AGI.
• MIRI researcher Buck Shlegeris talks aboutdeference and inside-view modelson the EA Forum.
• OpenAIunveils GPT-3, a massive 175-billionparameter language model that can figure out how to solve a variety of problems without task-specific training or fine-tuning. Gwern Branwen's pithysummary:
  

  GPT-3 is terrifying because it's a tiny model compared to what's possible, trained in the dumbest way possible on a single impoverished modality on tiny data, yet the first version already manifests crazy runtime meta-learning—and the scaling curvesstillare not bending!

  Further discussionby Branwenandby Rohin Shah.

• Stuart Russell gives this year's Turing Lecture online, discussing “证明地有益的人工智能”.

MIRI has received an anonymous donation of ~$275,000 in euros, facilitated byEffective Giving UK. Additionally, the Survival and Flourishing Fund, working with funders Jaan Tallinn and Jed McCaleb, has announced$340,000 in grants to MIRI. SFF is a new fund that istaking over much of BERI's grantmaking work.

To everyone involved in both decisions to support our research, thank you!

Other updates

• An Orthodox Case Against Utility Functions: MIRI researcher Abram Demski makes the case against utility functions that rely on a microphysical “view from nowhere”.
• Stuart Armstrong’s “If I were a well-intentioned AI…” sequence looks at alignment problems from the perspective of a well-intentioned but ignorant agent.
• AI Impacts' Asya Bergal summarizestakeaways from “safety-by-default” researchers.
• 阿加瓦尔和Norouzi报告improvements in offline RL.
• Daniel Kokotajlo’sThree Kinds of Competitivenessdistinguishes performance-competitive, cost-competitive, and date-competitive AI systems.
• TheStanford Encyclopedia of Philosophy's newEthics of AI and Roboticsarticle includes a discussion of existential risk from AGI.

MIRI has been awarded its largest grant to date— $7,703,750 split over two years from Open Philanthropy, in partnership with Ben Delo, co-founder of the cryptocurrency trading platform BitMEX!

We have also been awarded generous grants by the Berkeley Existential Risk Initiative ($300,000) and the Long-Term Future Fund ($100,000). Our thanks to everybody involved!

Other updates

• Buck Shlegeris of MIRI and Rohin Shah of CHAI discuss Rohin's 2018–2019 overview of technical AI alignment researchon the AI Alignment Podcast.
• From MIRI's Abram Demski:Thinking About Filtered Evidence Is (Very!) HardandBayesian Evolving-to-Extinction. And from Evan Hubinger:Synthesizing Amplification and Debate.
• OpenAI贝丝的巴恩斯,保罗global, Ouyang, and Geoffrey Irving:Progress on AI Safety via Debate.
• Zoom In: An Introduction to Circuits: OpenAI's Olah, Cammarata, Schubert, Goh, Petrov, and Carter argue, “Features are the fundamental unit of neural networks. They correspond to directions [in the space of neuron activations]. […] Features are connected by weights, forming circuits. […] Analogous features and circuits form across models and tasks.”
• DeepMind's Agent57appears to meet one of the AI benchmarks inAI Impacts' 2016 survey, “outperform professional game testers on all Atari games using no game specific knowledge”, earlier than NeurIPS/ICML authors predicted.
• From DeepMind Safety Research:Specification gaming: the flip side of AI ingenuity.

As the COVID-19 pandemic progresses, the LessWrong team has put togethera database of resourcesfor learning about the disease and staying updated, and 80,000 Hours has a new write-up onways to help in the fight against COVID-19. In my non-MIRI time, I've been keeping my own quick and informal notes on various sources' COVID-19 recommendationsin this Google Doc. Stay safe out there!

Updates

• My personal cruxes for working on AI safety:一个谈话记录from MIRI researcher Buck Shlegeris.
• Daniel Kokotajlo of AI Impacts discussesCortés, Pizarro, and Afonso as Precedents for Takeover.
• O'Keefe, Cihon, Garfinkel, Flynn, Leung, and Dafoe's “The Windfall Clause” proposes “an ex ante commitment by AI firms to donate a significant amount of any eventual extremely large profits” that result from “fundamental, economically transformative breakthroughs” like AGI.
• Microsoft announces the 17-billion-parameter language modelTuring-NLG.
• Oren Etzioni thinks AGI istoo far offto deserve much thought, and cites Andrew Ng's “overpopulation on Mars” metaphor approvingly — but he's also moving the debate in a very positive directionby listing specific observations that would make him change his mind.

Updates

• Colmreviews our 2019 fundraiser: taking into account matching, we received a total of $601,120 from 250+ donors. Our thanks again for all the support!
• Evan Hubinger'sExploring Safe Explorationclarifies points he raised inSafe Exploration and Corrigibility. The issues raised here are somewhat subtler than may be immediately apparent, since we tend to discuss things in ways that collapse the distinctions Evan is making.
• Logician Arthur Milchiorreviews the AIRCS workshop and MIRI's application processbased on his first-hand experience with both. See also后续讨论with MIRI and CFAR staff.
• Rohin Shah posts an in-depth2018–19 review of the field of AI alignment.
• From Shevlane and Dafoe's new “The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?”:

  [T]he existing conversation around AI has heavily borrowed concepts and conclusions from one particular field: vulnerability disclosure in computer security. We caution against AI researchers treating these lessons as immediately applicable. There are important differences between vulnerabilities in software and the types of vulnerabilities exploited by AI. […]

  补丁软件往往易于创建和can often be made in a matter of weeks. These patches fully resolve the vulnerability. The patch can be easily propagated: for downloaded software, the software is often automatically updated over the internet; for websites, the fix can take effect immediately.

  [… F]or certain technologies, there is no low-cost, straightforward, effective defence. [… C]onsider biological research that provides insight into the manufacture of pathogens, such as a novel virus. A subset of viruses are very difficult to vaccinate for (there is still no vaccination for HIV) or otherwise prepare against. This lowers the defensive benefit of publication, by blocking a main causal pathway by which publication leads to greater protection. This contrasts with the case where an effective treatment can be developed within a reasonable time period[.]

• Yann LeCun and Eliezer Yudkowskydiscuss the concept “AGI”.
• CFAR's Anna Salamon contrasts“reality-revealing” and “reality-masking” puzzles.
• Scott Alexanderreviews Stuart Russell'sHuman Compatible.

Links from the research team

MIRI researchers anonymously summarize and comment on recent posts and papers:

• ReACDT: a hack-y acausal decision theory— “Stuart Armstrong calls this decision theory a hack. I think it might be more elegant than he's letting on (i.e., a different formulation could look less hack-y), and is getting at something.”
• RePredictors exist: CDT going bonkers… forever— “I don't think Stuart Armstrong's example really adds much over some variants of Death in Damascus, but there's some good discussion of CDT vs. EDT stuff in the comments.”
• ReIs the term mesa optimizer too narrow?— “Matthew Barnett poses the important question, '[I]f even humans are not mesa optimizers, why should we expect mesa optimizers to be the primary real world examples of [malign generalization]?'”
• ReMalign generalization without internal search— “I think Matthew Barnett's question here is an important one. I lean toward the 'yes, this is a problem' camp—I don't think we can entirely eliminate malign generalization by eliminating internal search. But it is possible that this falls into other categories of misalignment (which we don't want to term 'inner alignment').”
• Re(A -> B) -> A in Causal DAGsandFormulating Reductive Agency in Causal Models— “I've wanted something like this for a while. Bayesian influence diagrams model agents non-reductively, by boldly asserting that some nodes are agentic. Can we make models which represent agents, without declaring a basic 'agent' type like that? John Wentworth offers an approach, representing agents via 'strange loops' across a use-mention boundary; and discusses how this might break down even further, with fully reductive agency. I'm not yet convinced that Wentworth has gotten it right, but it's exciting to see an attempt.”

Updates

• Our2019 fundraiserended Dec. 31. We'll have more to say in a few weeks in our fundraiser retrospective, but for now, a big thank you to the ~240 donors who together donated more than $526,000, including $67,484 in the first 20 seconds ofGiving Tuesday(not counting matching dollars, which have yet to be announced).
• Jan. 15 is the final day ofCFAR's annual fundraiser. CFAR alsorecently ran an AMAandhas posted their workshop participant handbook online.
• Understanding “Deep Double Descent”: MIRI researcher Evan Hubinger describes a fascinating phenomenon in ML, and an interesting case study in ML research aimed at deepening our understanding, and not just advancing capabilities. In a follow-up post, Evan also considerspossible implications for alignment research.
• Safe Exploration and Corrigibility: Evan notes an important (and alignment-relevant) way that notions of exploration in deep RL have shifted.
• “Learning Human Objectives by Evaluating Hypothetical Behavior”: UC Berkeley and DeepMind researchers “present a method for training reinforcement learning agents from human feedback in the presence of unknown unsafe states”.

Links from the research team

This continues my experiment from last month: having MIRI researchers anonymously pick out AI Alignment Forum posts to highlight and comment on.

• Re(When) is Truth-telling Favored in AI debate?— “A paper by Vojtěch Kovařík and Ryan Carey; it's good to see some progress on the debate model!”
• ReRecent Progress in the Theory of Neural Networks— “Noah MacAulay provides another interesting example of research attempting to explain what's going on with NNs.”
• ReWhen Goodharting is optimal— “I like Stuart Armstrong's post for the systematic examination of why we might be afraid of Goodharting. The example at the beginning is an interesting one, because it seems (to me at least) like the robot really should go back and forth (staying a long time at each side to minimize lost utility). But Stuart is right that this answer is, at least, quite difficult to justify.”
• ReSeeking Power is Instrumentally Convergent in MDPsandClarifying Power-Seeking and Instrumental Convergence— “It's nice to finally have a formal model of this, thanks to Alex Turner and Logan Smith. Instrumental convergence has been an informal part of the discussion for a long time.”
• ReCritiquing “What failure looks like”— “I thought that Grue Slinky's post was a good critical analysis of Paul Christiano's ‘going out with a whimper’ scenario, highlighting some of the problems it seems to have as a concrete AI risk scenario. In particular, I found the analogy given to the simplex algorithm persuasive in terms of showcasing how, despite the fact that many of our current most powerful tools already have massive differentials in how well they work on different problems, those values which are not served well by those tools don't seem to have lost out massively as a result. I still feel like there may be a real risk along the lines of ‘going out with a whimper’, but I think this post presents a real challenge to that scenario as it has been described so far.”
• ReCounterfactual Induction— “A proposal for logical counterfactuals by Alex Appel. This could use some more careful thought and critique; it's not yet clear exactly how much or little it accomplishes.”
• ReA dilemma for prosaic AI alignment— “Daniel Kokotajlo outlines key challenges for prosaic alignment: ‘[…] Now I think the problem is substantially harder than that: To be competitive prosaic AI safety schemes must deliberately create misaligned mesa-optimizers and then (hopefully) figure out how to align them so that they can be used in the scheme.’”