Some behavior seems like it's just making things possible, without actually doing much of anything, while other behavior seems to actually do something. Is there a principled, or a useful, distinction between possibilizing and actualizing? Is it possible to possibilize a large effect on the world without actualizing large effects on the world?

$\require{AMScd}$ Thought and action in pursuit of convergent instrumental subgoals do not automatically reveal why those subgoals are being pursued——towards what supergoals——because many other agents with different supergoals would also pursue those subgoals, maybe with overlapping thought and action. In particular, an agent's ultimate ends don't have to be revealed by its pursuit of convergent subgoals. It might might therefore be easy to covertly pursue some ultimate goal by mostly pursuing generally useful subgoals of other supergoals. By the inspection paradox for the convergence of subgoals, it might be easy to think and act almost comprehensively like a non-threatening agent would think and act, while going most of the way towards achieving some other more ambitious goal.

[Epistemic status: just a guess / hypothesis.] Suppose Alice is anti-immigration and has political power. She politically pushes for laws against immigration, government spending towards capacity to prevent immigration like walls and guards, policies to deport illegal immigrants, and so on. Alice also pushes against policies to cope with whatever the current de facto status quo is, e.g. to alleviate harms done by whatever is already going on, or at least doesn't push for such alleviation policies. Those policies would alleviate pressure to change, and Alice wants change; they'd make the status quo less bad, and Alice doesn't like the status quo. And, those policies being passed would constitute, relative to Alice's desired anti-immigration stance, a symbolic victory for the side opposing Alice; it would "say", in the language of politics, that "we are okay with the status quo, we're organizing to make something like the status quo work well". Pl...

What are an agent's values? An answer to this question might be a good description of the agent's external behavior and internal workings, without showing how one could modify the agent's workings or origins so that the agent pushes the world in a specific different direction.

1. Shell game 2. Perpetual motion machines 3. Shell games in alignment Example: hiding the generator of large effects Example: hiding the generator of novel understanding Other? 1. Shell game Here's the classic shell game: Youtube Screenshot from that video. The little ball is a phantom: when you look for it under a specific shell, it's not there, it's under a different shell.

Are there unbounded modes of thinking that are systemically, radically distinct from each other in relevant ways?

Summary: adding artificial connections between distant areas of the brain might increase intelligence in two ways. The first way is by simply increasing connectivity in areas that perform abstract thinking; since evolution was clearly bottlenecked on connectivity, that might be valuable to the brain. The second way is by reprioritizing brainware according to our values in our current environment. Prosthetic connectivity seems bottlenecked on a bunch of nitty-gritty (bio)engineering work that's on the mainstream BCI pathway.

Humans are born with some elements of their minds, and without many other elements, some of which they'll acquire as their life unfolds. In particular, the elements that we pretheoretically call "values"——aesthetic preferences, goals, life goals, squad goals, aspirations, needs, wants, yearnings, drives, cravings, principles, morals, ethics, senses of importance, and so on——are for the most part acquired or at least unfolded, rather than being explicitly present in a newborn. How does this happen? What generates these mental elements? Hypothesis: a human derives many of zer values by imputing coherent agency to zer past behavior, and then adopting the goals of that fictitious agency as actively influential criteria for future action.

[This post is labeled בבל, meaning it's especially experimental. See: בבל disclaimer ] A subgoal sticks its head into eternity.

Counting-down coherence is the coherence of a mind viewed as the absence of deviation downward in capability from ideal, perfectly efficient agency: the utility left on the table, the waste, the exploitability. Counting-up coherence is the coherence of a mind viewed as the deviation upward in capability from a rock: the elements of the mind, and how they combine to perform tasks.

To have large relevant effects on the world, a mind has to understand a lot about the world. The mind has to have a lot of the structure of the cosmos (the entirety of the world, in any aspect or abstraction) highly accessible to itself for use in skillful action. To understand a lot about the world, the mind has to gain a lot of understanding that it didn't have previously. When a mind gains understanding, that's a change in the mind. Does that change have to include a change to the values of the mind?

Suppose we want to observe the thoughts of a mind in order to detect whether it's making its way towards a plan to harm us, and ideally also to direct the mind so that it pursues specific aims. To this end, we might hope that the mind and its thinking are organized in a way we can come to understand in the way that we understand ourselves and our thinking . We might hope that when the mind considers plans that involve something, e.g. plans that involve the coffee cup, it does so using a concept alike to our concept [[coffee cup]]. When the mind recognizes, predicts, imagines, simulates, manipulates, designs, combines things with, describes, studies, associates things with, summarizes, remembers, compares things with, deduces things about, makes hypotheses about, or is otherwise mentally involved with the coffee cup, maybe it always does so in a way that is fully comprehendable in fixed terms that are similar to the terms in which we understand ourselves when we do those activities...

Sometimes people defer to other people, e.g. by believing what they say, by following orders, or by adopting intents or stances. In many cases it makes sense to defer, since other people know more than you about many things, and it's useful to share eyes and ears, and coordination and specialization are valuable, and one can "inquisitively defer" to opinions by taking them as challenges to investigate further by trying them out for oneself. But there are major issues with deferring, among which are: Deferral-based opinions don't contain the detailed content that generated the opinions, and therefore can't direct action effectively or update on new evidence correctly. Acting based on deferral-based opinions is discouraging because it's especially not the case that the whole of you can see why the action is good. Acting based on deferral-based opinions to some extent removes the "meaning" of learning new information; if you're just going t...

I don't know how to define control or even point at it except as a word-cloud, so it's probably wanting to be refactored. The point of talking about control is to lay part of the groundwork for understanding what determines what directions a mind ends up pushing the world in. Control is something like what's happening when values or drives are making themselves felt as values or drives. ("Influence" = "in-flow" might be a better term than "control".)

A high-level confusion that I have that seems to be on the way towards understanding alignment, is the relationship between values and understanding. This essay gestures at the idea of structure in general (mainly by listing examples).

A gemini model is a kind of model that's especially relevant for minds modeling minds.

In trying to understand minds-in-general, we sometimes ask questions that talk about "big" things (taking "big" to ambiguously mean any of large, complex, abstract, vague, important, touches many things, applies to many contexts, "high-level"). E.g.: What is it for a mind to have thoughts or to care about stuff? How does care and thought relate? What is it to believe a proposition? Why do agents use abstractions? These "big" things such as thought, caring, propositions, beliefs, agents, abstractions, and so on, have to be analyzed and re-understood in clearer terms in order to get anywhere useful. When others make statements about these things, I'm pulled to pause their flow of thoughts and instead try to get clear on meanings. In part, that pull is because the more your thoughts use descriptions that aren't founded on words with clear meaning, the more leeway is given to your words to point at different things in different in...

$\newcommand{\Z}{\mathbb{Z}}$ What's a thing, in general? Minds deal with things, so this question comes up in trying to understand minds. Minds think about things, speak of things, manipulate things, care about things, create things, and maybe are made of things.

$\newcommand{\Var}{\mathrm{Var}}$ $\newcommand{\second}{2\text{nd}}$ $\newcommand{\kth}{k\text{-th}}$ $\newcommand{\R}{\mathbb{R}}$ $\newcommand{\Ltwo}[1]{\|#1\|_2}$ $\newcommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}$ If you put in work to select additive components of some random variable, how far out can you get in the distribution of that variable? This post will focus on normally distributed variables, which is handy since the sum of many individually small random variables is roughly normally distributed by the Central Limit Theorem . (Note: In places this post is long-ish and discursive (and explains an error I made) because it's trying to get a mathematical understanding of selection that can inform mathematical intuitions about more complicated kinds of selection. If you just want a summary of the numerical situation, look at the tables and graphs.) Code for tables and diagrams are in this Github repository . Thanks to Sam Eisenstat for many ...

There are downside risks to selecting genomes of future human children for traits like health, intelligence, lifespan, mental health, and so on. This essay maps out some of these, starting with these intuitions: Unnaturalness. Objectification. Transgression. Misalignment.

TL;DR: To keep track of your thoughts about a question that has multiple parameters, you can use multi-dimensional spreadsheets. If you use vim, vim-multisheets is a basic implementation of multi-dimensional spreadsheets; code here . Tracking beliefs and then automatically unfolding some of their consequences can point towards questions you haven't answered and point to contradictions in your beliefs.

Aliyah walks toward one side of a half-meter-wide ditch. Without missing a beat, she plants her left foot at the rounded lip of the ditch and then swings her right leg forward over the ditch to plant her right foot at the opposite lip, bobbling her upper body forward in dynamic equilibrium between the upward-centerward forces from her proceeding alternating legs, in a smooth motion reprising her past steps over solid ground. Boaz, following Aliyah, walks toward the half-meter-wide ditch. He starts to step across, but pauses halfway through, poised over the ditch. To amuse himself, he reverses by pushing back on the opposite lip, starts forward again, goes almost all the way, then reverses again. With just a bit of effort he can statically occupy a near approximation of any position and stance that Aliyah passed through during her step, dwelling for a time in that moment that was for Aliyah a non-extended point in a dynamic process. Carmel walks toward a meter-wide ditch. A push from ...

Very hot water stops itch. Itching due to an immune reaction--bug bite, allergies, poison ivy / poison oak--is caused by histamine. Heat makes histamine release quickly. So if you have an annoying itchy spot, go to the sink and run the water very hot. Don't burn yourself, but it should be really hot; hot enough that you can be okay with it running over your skin, but only by easing the cold water down gradually so the water gets hotter and hotter. If the itch feels weirdly intense, even pleasurable, like you're scratching all of it all at once, you're doing it right. Once the histamine is released, it should stay non-itchy for a while, like an hour or two or three. If the itch is in an inconvenient spot, e.g. face or neck, use a hot water bottle or in a pinch, ziploc bags. Works even for terrible poison ivy / poison oak rashes; it's way more effective than topical steroids for itch relief. When you can't find something and then you find it, after you're don...

An old woman twists a rag over a bucket and the dampness of the rag lessens. She's sitting in the livingroom of an old house on a chair in front of a low table. It's not clear how old she is. The rag is still damp, but less so now that she's twisted it, and at least mold probably won't bloom under the rag if she leaves it there on the low table spread in a rumpled square, so there she puts it, and now her hands are twisting another rag, steadily centered over the bucket. There are many rags on the table. They're all wet, and it's not clear why they're wet. Maybe water vapor collects on them during foggy nights with the windows left open, or maybe the roof of the old house is letting rain water pass through the part of itself that flies directly over the low table. It's not clear why the rags are wet and it's also not clear how old the old woman is. Sometimes she's as young as five, and often she's sixteen or twenty-nine, and it would not be e...

[CW: discussion of creepy bugs and animal suffering, some links show those things. IANAB] What's the difference between parasites and predators? I'll re-ask this question in different forms as we go. I invite you to figure out the answers for yourself. I'll give an answer, but it may be wrong and it's definitely not the whole story.

To understand a complex system like a mind or an ecosystem, we have to understand a tangled web of objects, features, processes, relations, correlations, clusters, constraints, causes, and so on. It helps to find underlying explanations and generating process, and to find deep reasons that explain why many relationships are the way they are. To find those underlying explanations, it helps to know relationships between relationships: which relationships cause, explain, or constrain other relationships. When staring at an assembly of relationships and asking which relationships explain other relationships, it can seem like we're at a loss for where to go; sometimes there are clearly relationships, but there's no clear way to extract any further order, because there's no basis on which to say that one relationship explains another——the relationships are just there, and that's it. By way of example/analogy, suppose we look at a list of facts like [3×4=12, 12/4=3, 4=12/3, 1...

To explore better possibilities for nurturing new minds, and to care about the problem in the first place, it helps to remember what's wrong with what we do to new minds. John Taylor Gatto speaks about this from experience and insight: Seven Lessons Taught in School, 1991 (If you're going to read the following, at least read the seven lessons (part I) of that essay.) Here's another list of harms caused by schooling. 1. You aren't a mind, and don't bother trying to behave like one. Children naturally attend to things until they're done with them: From Maria Montessori, My System of Education, 1915: IPFS pdf link A little girl, about three years of age, was deeply absorbed in the work of placing wooden blocks and cylinders in a frame for that purpose. The expression of her face was that of such intense attention, that it was almost a revelation to me. Never before had I seen a child look with such "fixedness" upon an object, and my conviction abo...

[Context: In the seven years I've been living in the Bay Area, I've attended two Rationalist Solstices. Both times I basically hated the experience. I don't really know why, but in thinking about it I think I've understood something worth saying about rituals and symbolism. (I don't think what follows is a crux for me about Rationalist Solstice (I don't even know if it applies); I'd guess that has more to do with people gathering in large groups in general seeming harmful, distortive, or hostile or something.)] On Friday evening--Erev Shabbat (= evening of the day of rest)--Jews light candles and say a brachah. I was told by adults that this is done to symbolize a separation between the week and the holy Shabbat. On Saturday evening, when Shabbat is over, Jews do Havdalah (= separating): they light another candle; smell spices; and say more brachot. I was told that the warmth and light of the candle serve to remember to us the warmth and light of Shabbat a...