AGI is likely to come this century. Say you have a plan $X$ that would prevent AGI from destroying the world. How beneficial is it to set plan $X$ in motion sooner by $n$ years? A very short answer: it reduces the probability of AGI ruin by something like $n/2$ or $n/3$ percent. Which is a lot. A slightly longer answer: it reduces the probability of AGI ruin by roughly $fn$, where $f$ is the probability per year of AGI ruin around the $n$-year interval of time between when plan $X$ would have been completed with intervention and without intervention. So if $X$ would take a very long time either way, or if the chances of AGI ruin are very spread out through time, then the intervention doesn't matter that much; otherwise the intervention probably makes a noticeable difference. A fuller answer: This post assumes that the arrival of AGI ruin, and the workings of $X$, are independent. It also assumes that the probability of AGI ruin without $X$ is 1, so results should be scaled down...

I don't know how to define control or even point at it except as a word-cloud, so it's probably wanting to be refactored. The point of talking about control is to lay part of the groundwork for understanding what determines what directions a mind ends up pushing the world in. Control is something like what's happening when values or drives are making themselves felt as values or drives. ("Influence" = "in-flow" might be a better term than "control".)

A high-level confusion that I have that seems to be on the way towards understanding alignment, is the relationship between values and understanding. This essay gestures at the idea of structure in general (mainly by listing examples).

A gemini model is a kind of model that's especially relevant for minds modeling minds.

In trying to understand minds-in-general, we sometimes ask questions that talk about "big" things (taking "big" to ambiguously mean any of large, complex, abstract, vague, important, touches many things, applies to many contexts, "high-level"). E.g.: What is it for a mind to have thoughts or to care about stuff? How does care and thought relate? What is it to believe a proposition? Why do agents use abstractions? These "big" things such as thought, caring, propositions, beliefs, agents, abstractions, and so on, have to be analyzed and re-understood in clearer terms in order to get anywhere useful. When others make statements about these things, I'm pulled to pause their flow of thoughts and instead try to get clear on meanings. In part, that pull is because the more your thoughts use descriptions that aren't founded on words with clear meaning, the more leeway is given to your words to point at different things in different in...

$\newcommand{\Z}{\mathbb{Z}}$ What's a thing, in general? Minds deal with things, so this question comes up in trying to understand minds. Minds think about things, speak of things, manipulate things, care about things, create things, and maybe are made of things.

$\newcommand{\Var}{\mathrm{Var}}$ $\newcommand{\second}{2\text{nd}}$ $\newcommand{\kth}{k\text{-th}}$ $\newcommand{\R}{\mathbb{R}}$ $\newcommand{\Ltwo}[1]{\|#1\|_2}$ $\newcommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}$ If you put in work to select additive components of some random variable, how far out can you get in the distribution of that variable? This post will focus on normally distributed variables, which is handy since the sum of many individually small random variables is roughly normally distributed by the Central Limit Theorem . (Note: In places this post is long-ish and discursive (and explains an error I made) because it's trying to get a mathematical understanding of selection that can inform mathematical intuitions about more complicated kinds of selection. If you just want a summary of the numerical situation, look at the tables and graphs.) Code for tables and diagrams are in this Github repository . Thanks to Sam Eisenstat for many ...

There are downside risks to selecting genomes of future human children for traits like health, intelligence, lifespan, mental health, and so on. This essay maps out some of these, starting with these intuitions: Unnaturalness. Objectification. Transgression. Misalignment.