[This post is labeled בבל, meaning it's especially experimental. See: בבל disclaimer ] A subgoal sticks its head into eternity.

Counting-down coherence is the coherence of a mind viewed as the absence of deviation downward in capability from ideal, perfectly efficient agency: the utility left on the table, the waste, the exploitability. Counting-up coherence is the coherence of a mind viewed as the deviation upward in capability from a rock: the elements of the mind, and how they combine to perform tasks.

To have large relevant effects on the world, a mind has to understand a lot about the world. The mind has to have a lot of the structure of the cosmos (the entirety of the world, in any aspect or abstraction) highly accessible to itself for use in skillful action. To understand a lot about the world, the mind has to gain a lot of understanding that it didn't have previously. When a mind gains understanding, that's a change in the mind. Does that change have to include a change to the values of the mind?

Suppose we want to observe the thoughts of a mind in order to detect whether it's making its way towards a plan to harm us, and ideally also to direct the mind so that it pursues specific aims. To this end, we might hope that the mind and its thinking are organized in a way we can come to understand in the way that we understand ourselves and our thinking . We might hope that when the mind considers plans that involve something, e.g. plans that involve the coffee cup, it does so using a concept alike to our concept [[coffee cup]]. When the mind recognizes, predicts, imagines, simulates, manipulates, designs, combines things with, describes, studies, associates things with, summarizes, remembers, compares things with, deduces things about, makes hypotheses about, or is otherwise mentally involved with the coffee cup, maybe it always does so in a way that is fully comprehendable in fixed terms that are similar to the terms in which we understand ourselves when we do those activities...