[Caveat lector: this is a very long, rambling meditation on concepts and coordination. It's not cut down for size or well-organized. That said, I had several insights while writing it.] 1. Case 1: Is an embryo a person? 1.1. A big blob of subquestions 1.2. Difficult questions produce uncertain, sticky, varied opinions 1.3. A multi-question blob does not have "an answer" 1.4. Coordination about X is fragile to not knowing what other people will think of X 1.5. It's hard to coordinate about big question blobs 1.6. Correction: It's hard to coordinate distributed judgements about big question blobs 1.7. So, to coordinate, people look for simple questions 2. Case 2: Should people be allowed to think freely? 2.1. The First Amendment and delegation 2.2. Erosion of free thought 2.3. Power vacuum 2.4. Simple concepts help coordination because they are anti-invidious 3. Case 3: How nice should you be? 3.1. Nice and kind 3.2. Burning goodness 3.3. The vi...

[This is a message I have, several times, wanted to send to commenters (mainly on public forums). I'm putting it here so I can link to it.] For whatever reason, in order for me to actively independently desire to continue this thread of discussion, I would probably need something from you. What I would need from you is something like more evidence that you're genuinely trying to understand what I'm saying in context, and some of the output from that process. This is not a dunk on you. This is me just saying: For the time being, I don't want to keep talking on this thread, though I'm still at least a little interested in the topic. By default I would just go silent. But, as a slight improvement over that, I'm letting you know that I might want to keep talking on this thread, if I had more sense that (and how) you were trying to understand what I'm saying / what I think / what we agree or disagree about / etc. I might have stated slightly more specifical...

[Epistemic status: !! 🚨 Drama Alert 🚨 !! discoursepoasting, LWslop] 1. Case 1: You only get six words 2. Case 2: Trees may be cool but how should concepts work in general?? 3. Case 3: The Bannination 4. The Pattern 5. Conclusion 1. Case 1: You only get six words In 2024, the MATS team published a post, originally titled " Talent Needs in Technical AI Safety ". I, a hero, made this comment and elaborated in the ensuing comment thread. The content isn't so important here—basically, I was objecting to a certain framing in the post, which tied into a general issue I had with the broader landscape of people nominally working on decreasing AGI X-risk. Now, I have not actually read this post. (I kinda skimmed it and read parts.) So I don't actually know what's in it. The post's description of itself, from the introduction: In the winter and spring of 2024, we conducted 31 interviews, ranging in length from 30 to 120 minutes, with key figures in AI ...

1. Introduction 2. Extended table of contents 3. Notice when you might be deferring 4. Just say that you might be deferring 5. Factor out and fortify endorsed deference 6. Respect the costs of independent investigation 7. Give stewardship, not authority 8. Take me to your leader 9. Recount your sources 10. Expose cruxes about deference 11. Distinguish your independent components 12. Retreat to non-deferential cruxes 13. Open problems in graceful deference 14. Acknowledgements 1. Introduction Last time, I asked about The problem of graceful deference . We have to defer to other people's judgements of fact and of value, because there are too many important questions to consider thoroughly ourselves. Is germline engineering moral? What should I work on to decrease existential risk? Do I really have to floss? Should I get vaccinated? How feasible is safe AGI? Pick one each month or year to start the long process of becoming an expert on; the rest you'll have to de...

1. Moral deference 2. Correlated failures 3. The open problem 1. Moral deference Sometimes when I bring up the subject of reprogenetics, people get uncomfortable. "So you want to do eugenics?", "This is going to lead to inequality.", "Parents are going to pressure their kids.". Each of these statements does point at legitimate concerns. But also, the person is uncomfortable, and they don't necessarily engage with counterpoints. And, even if they acknowledge that their stated concern doesn't make sense, they'll still be uncomfortable— until they think of another concern to state. This behavior is ambiguous—I don't know what underlies the behavior in any given case. E.g. it could be that they're intent on pushing against reprogenetics regardless of the arguments they say, or it could be that they have good and true intuitions that they haven't yet explicitized. And in any case, argument and explanation is usually best. Still,...

1. Introduction: human intelligence amplification and existential risk 2. Deference 3. Reversal test 4. Human empowerment is good 5. Abundance decreases X-risk 5.1. Abundance makes less motive to make AGI 5.2. Abundance makes a healthier society 6. Empowerment that democratizes is good 7. Big Good harder than Big Bad; requires more ideas 7.1. Good outcomes are narrow targets 7.2. Easier to hurt than to help 7.3. One-shot problems require thinking hard 7.4. We're still bottlenecked on ideas 8. HIA downsides are opportunities for HIA meta-level upsides 9. HIA downside insincerity 10. It's even plausibly better to have smarter capabilities research leaders 11. Proof by Intimidation 1. Introduction: human intelligence amplification and existential risk I've always taken it for granted that increasing the brainpower of humanity is a good thing. It's intuitively clear to me. But since I've started working on human intelligence amplification (HIA), s...