1. Introduction: human intelligence amplification and existential risk 2. Deference 3. Reversal test 4. Human empowerment is good 5. Abundance decreases X-risk 5.1. Abundance makes less motive to make AGI 5.2. Abundance makes a healthier society 6. Empowerment that democratizes is good 7. Big Good harder than Big Bad; requires more ideas 7.1. Good outcomes are narrow targets 7.2. Easier to hurt than to help 7.3. One-shot problems require thinking hard 7.4. We're still bottlenecked on ideas 8. HIA downsides are opportunities for HIA meta-level upsides 9. HIA downside insincerity 10. It's even plausibly better to have smarter capabilities research leaders 11. Proof by Intimidation 1. Introduction: human intelligence amplification and existential risk I've always taken it for granted that increasing the brainpower of humanity is a good thing. It's intuitively clear to me. But since I've started working on human intelligence amplification (HIA), s...

How can we make many humans who are very good at solving difficult problems?

Summary: adding artificial connections between distant areas of the brain might increase intelligence in two ways. The first way is by simply increasing connectivity in areas that perform abstract thinking; since evolution was clearly bottlenecked on connectivity, that might be valuable to the brain. The second way is by reprioritizing brainware according to our values in our current environment. Prosthetic connectivity seems bottlenecked on a bunch of nitty-gritty (bio)engineering work that's on the mainstream BCI pathway.

AGI is likely to come this century. Say you have a plan $X$ that would prevent AGI from destroying the world. How beneficial is it to set plan $X$ in motion sooner by $n$ years? A very short answer: it reduces the probability of AGI ruin by something like $n/2$ or $n/3$ percent. Which is a lot. A slightly longer answer: it reduces the probability of AGI ruin by roughly $fn$, where $f$ is the probability per year of AGI ruin around the $n$-year interval of time between when plan $X$ would have been completed with intervention and without intervention. So if $X$ would take a very long time either way, or if the chances of AGI ruin are very spread out through time, then the intervention doesn't matter that much; otherwise the intervention probably makes a noticeable difference. A fuller answer: This post assumes that the arrival of AGI ruin, and the workings of $X$, are independent. It also assumes that the probability of AGI ruin without $X$ is 1, so results should be scaled down...

$\newcommand{\Var}{\mathrm{Var}}$ $\newcommand{\second}{2\text{nd}}$ $\newcommand{\kth}{k\text{-th}}$ $\newcommand{\R}{\mathbb{R}}$ $\newcommand{\Ltwo}[1]{\|#1\|_2}$ $\newcommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}$ If you put in work to select additive components of some random variable, how far out can you get in the distribution of that variable? This post will focus on normally distributed variables, which is handy since the sum of many individually small random variables is roughly normally distributed by the Central Limit Theorem . (Note: In places this post is long-ish and discursive (and explains an error I made) because it's trying to get a mathematical understanding of selection that can inform mathematical intuitions about more complicated kinds of selection. If you just want a summary of the numerical situation, look at the tables and graphs.) Code for tables and diagrams are in this Github repository . Thanks to Sam Eisenstat for many ...

There are downside risks to selecting genomes of future human children for traits like health, intelligence, lifespan, mental health, and so on. This essay maps out some of these, starting with these intuitions: Unnaturalness. Objectification. Transgression. Misalignment.

Previous work: Gwern , Anon This is all speculation from a lay perspective. The idea is to make an embryo with a genome selected chromosome by chromosome from some input genomes.

Previous work: Gwern My lay understanding is that it's not known how to sequence gametes without destroying them. If we could non-destructively sequence gametes, it would be easier to e.g. screen for genetic illnesses. Instead of fertilizing a few eggs, which are expensive to acquire, and then sequencing the resulting embryos, we could produce many sperms, find acceptable sperms, and then use those to fertilize the few available eggs. This post describes a way to sequence gametes non-destructively, assuming that it's possible to sequence gametes destructively. I lack lots of basic biological knowledge, so I can't verify that this idea makes sense, would work, or would be feasible or efficient. I hope others who are more informed can check and use the idea. I'll focus on sperm for clarity; I don't know whether / how this might extend to ova.