I.

The idea is: you have a Judgment Machine. System 1, intuition, whatever you want to call it. Note: this idea is significantly oversimplified (in Kahneman’s book too), read “The Rationality Quotient” for more details. But it’ll do.

The important part is that you don’t have access to its internal workings. It lets you just look at things and boom – you know the right thing to do.

Or the wrong thing to do! You’ve listened to it so many times and got burned so many times. Blurted out things you shouldn’t have said. Kept wearing that pair of jeans you like, even though you’ve been told by enough people that it makes you look like a homeless person.

You notice it happening time and again, and you get tired of it. “Oh, my system 1 keeps overpowering my system 2! I gotta learn how to control it. Shut it down. I’m not against laughter and joy, and I will not abandon my values – but I won’t let system 1 tell me how to achieve those values. Nuh-uh.”

So now, whenever system 1 tells you “you could try X”, you get suspicious. You treat it like a child who never grew up. “I don’t like this guy, true – but I won’t fire him just because I don’t like him! That’s the child-in-me speaking.”

II.

Except that – it’s not a child. It grew up with you. All your mistakes were happening right before its eyes. It has read all the same books that you’ve read, and liked the same posts you liked. Scott Alexander, Yudkowsky, Kegan, The Last Psychiatrist, whatever, you scrolled them right past its eyes.

Yeah, sometimes it still says “this guy has mustache, I don’t like him”. True.

But sometimes it also says “hey, this sounds awfully like a class of disastrous situations described in ten or so different posts I’ve read in the past two years” — which is quite a bit more useful, and sometimes life-changing. By routinely dismissing system 1’s input as “child’s babbling”, you lose access to a huge library of associations that it has amassed over the years. At some point it will bite you back.