Rationality From AI to Zombies
by Eliezer Yudkowsky
Buy for free or more!
Released under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported license
⃝cc BY-NC-SA 3.0
http://creativecommons.org/licenses/by-nc-sa/3.0/
Contents v
Preface
Biases: An Introduction xxiii
Book I Map and Territory
A Predictably Wrong
. 1 What Do I Mean By “Rationality”? 7
. 2 Feeling Rational 12
. 3 WhyTruth?And… 15
. 4 . . . What’s a Bias, Again? 19
. 5 Availability 23
. 6 Burdensome Details 26
. 7 Planning Fallacy 30
. 8 Illusion of Transparency: Why No One Understands You 34
. 9 Expecting Short Inferential Distances 37
. 10 The Lens That Sees Its Own Flaws 40
B Fake Beliefs
. 11 Making Beliefs Pay Rent (in Anticipated Experiences) 45
. 12 A Fable of Science and Politics 49
. 13 Belief in Belief 54
. 14 Bayesian Judo 59
. 15 Pretending to be Wise 61
. 16 Religion’s Claim to be Non-Disprovable 65
. 17 Professing and Cheering 69
. 18 Belief as Attire 72
. 19 Applause Lights 74
C Noticing Confusion
. 20 Focus Your Uncertainty 81
. 21 What Is Evidence? 84
. 22 Scientific Evidence, Legal Evidence, Rational Evidence 88
. 23 How Much Evidence Does It Take? 91
. 24 Einstein’s Arrogance 95
. 25 Occam’s Razor 98
. 26 Your Strength as a Rationalist 103
. 27 Absence of Evidence Is Evidence of Absence 106
. 28 Conservation of Expected Evidence 109
. 29 Hindsight Devalues Science 112
D Mysterious Answers
. 30 Fake Explanations 117
. 31 Guessing the Teacher’s Password 120
. 32 Science as Attire 124
. 33 Fake Causality 127
. 34 Semantic Stopsigns 132
. 35 Mysterious Answers to Mysterious Questions 136
. 36 The Futility of Emergence 140
. 37 Say Not “Complexity” 143
. 38 Positive Bias: Look into the Dark 147
. 39 Lawful Uncertainty 150
. 40 My Wild and Reckless Youth 154
. 41 Failing to Learn from History 158
. 42 Making History Available 160
. 43 Explain/Worship/Ignore? 163
. 44 “Science” as Curiosity-Stopper 165
. 45 Truly Part of You 168
Interlude: The Simple Truth 173
Book II How to Actually Change Your Mind
Rationality: An Introduction 199
E Overly Convenient Excuses
. 46 The Proper Use of Humility 211
. 47 The Third Alternative 216
. 48 Lotteries: A Waste of Hope 219
. 49 New Improved Lottery 221
. 50 But There’s Still a Chance, Right? 224
. 51 The Fallacy of Gray 227
. 52 Absolute Authority 232
. 53 HowtoConvinceMeThat2+2=3 239
. 54 Infinite Certainty 242
. 55 0 And 1 Are Not Probabilities 246
. 56 Your Rationality Is My Business 250
F Politics and Rationality
. 57 Politics is the Mind-Killer 255
. 58 Policy Debates Should Not Appear One-Sided 257
. 59 The Scales of Justice, the Notebook of Rationality 261
. 60 Correspondence Bias 264
. 61 Are Your Enemies Innately Evil? 267
. 62 Reversed Stupidity Is Not Intelligence 270
. 63 Argument Screens Off Authority 274
. 64 Hug the Query 280
. 65 Rationality and the English Language 282
. 66 Human Evil and Muddled Thinking 286
G Against Rationalization
. 67 Knowing About Biases Can Hurt People 293
. 68 Update Yourself Incrementally 296
. 69 One Argument Against An Army 299
. 70 The Bottom Line 302
. 71 What Evidence Filtered Evidence? 305
. 72 Rationalization 309
. 73 A Rational Argument 312
. 74 Avoiding Your Belief ’s Real Weak Points 315
. 75 Motivated Stopping and Motivated Continuation 320
. 76 Fake Justification 323
. 77 Is That Your True Rejection? 326
. 78 Entangled Truths, Contagious Lies 330
. 79 Of Lies and Black Swan Blowups 334
. 80 Dark Side Epistemology 335
H Against Doublethink
. 81 Singlethink 343
. 82 Doublethink (Choosing to be Biased) 345
. 83 No, Really, I’ve Deceived Myself 349
. 84 Belief in Self-Deception 351
. 85 Moore’s Paradox 356
. 86 Don’t Believe You’ll Self-Deceive 359
. I Seeing with Fresh Eyes
. 87 Anchoring and Adjustment 365
. 88 Priming and Contamination 368 Do
. 89 We Believe Everything We’re Told? 371
. 90 Cached Thoughts 374
. 91 The “Outside the Box” Box 377
. 92 Original Seeing 380
. 93 Stranger than History 383
. 94 The Logical Fallacy of Generalization from Fictional Evidence 385
. 95 The Virtue of Narrowness 391
. 96 How to Seem (and Be) Deep 395
. 97 We Change Our Minds Less Often Than We Think 399
. 98 Hold Off On Proposing Solutions 401
. 99 The Genetic Fallacy 404
J Death Spirals
. 100 The Affect Heuristic 409
. 101 Evaluability (and Cheap Holiday Shopping) 413
. 102 Unbounded Scales, Huge Jury Awards, and Futurism 418
. 103 The Halo Effect 422
. 104 Superhero Bias 426
. 105 Mere Messiahs 430
. 106 Affective Death Spirals 434
. 107 Resist the Happy Death Spiral 437
. 108 Uncritical Supercriticality 443
. 109 Evaporative Cooling of Group Beliefs 447
. 110 When None Dare Urge Restraint 451
. 111 The Robbers Cave Experiment 454
. 112 Every Cause Wants to Be a Cult 458
. 113 Guardians of the Truth 461
. 114 Guardians of the Gene Pool 466
. 115 Guardians of Ayn Rand 468
. 116 Two Cult Koans 473
. 117 Asch’s Conformity Experiment 476
. 118 On Expressing Your Concerns 480
. 119 Lonely Dissent 483
. 120 Cultish Countercultishness 487
K Letting Go
. 121 The Importance of Saying “Oops” 497
. 122 The Crackpot Offer 500
. 123 Just Lose Hope Already 503
. 124 The Proper Use of Doubt 505
. 125 You Can Face Reality 508
. 126 The Meditation on Curiosity 509
. 127 No One Can Exempt You From Rationality’s Laws 513
. 128 Leave a Line of Retreat 516
. 129 Crisis of Faith 520
. 130 The Ritual 528
Book III The Machine in the Ghost
Minds: An Introduction 539
Interlude: The Power of Intelligence 547
L The Simple Math of Evolution
. 131 An Alien God 553
. 132 The Wonder of Evolution 561
. 133 Evolutions Are Stupid (But Work Anyway) 565
. 134 No Evolutions for Corporations or Nanodevices 570
. 135 Evolving to Extinction 576
. 136 The Tragedy of Group Selectionism 582
. 137 Fake Optimization Criteria 588
. 138 Adaptation-Executers, Not Fitness-Maximizers 591
. 139 Evolutionary Psychology 595
. 140 An Especially Elegant Evolutionary Psychology Experiment 600
. 141 Superstimuli and the Collapse of Western Civilization 604
. 142 Thou Art Godshatter 608
M Fragile Purposes
. 143 Belief in Intelligence 617
. 144 Humans in Funny Suits 621
. 145 Optimization and the Intelligence Explosion 627
. 146 Ghosts in the Machine 634
. 147 Artificial Addition 639
. 148 Terminal Values and Instrumental Values 645
. 149 Leaky Generalizations 654
. 150 The Hidden Complexity of Wishes 658
. 151 Anthropomorphic Optimism 665
. 152 Lost Purposes 669
N A Human’s Guide to Words
. 153 The Parable of the Dagger 677
. 154 The Parable of Hemlock 679
. 155 Words as Hidden Inferences 683
. 156 Extensions and Intensions 686
. 157 Similarity Clusters 690
. 158 Typicality and Asymmetrical Similarity 692
. 159 The Cluster Structure of Thingspace 695
. 160 Disguised Queries 699
. 161 Neural Categories 704
. 162 How An Algorithm Feels From Inside 710
. 163 Disputing Definitions 716
. 164 Feel the Meaning 721
. 165 The Argument from Common Usage 725
. 166 Empty Labels 730
. 167 Taboo Your Words 733
. 168 Replace the Symbol with the Substance 737
. 169 Fallacies of Compression 742
. 170 Categorizing Has Consequences 746
. 171 Sneaking in Connotations 749
. 172 Arguing “By Definition” 753
. 173 Where to Draw the Boundary? 757
. 174 Entropy, and Short Codes 761
. 175 Mutual Information, and Density in Thingspace 765
. 176 Superexponential Conceptspace, and Simple Words 772
. 177 Conditional Independence, and Naive Bayes 779
. 178 Words as Mental Paintbrush Handles 787
. 179 Variable Question Fallacies 790
. 180 37 Ways That Words Can Be Wrong 793
Interlude: An Intuitive Explanation of Bayes’s Theorem 803
Book IV Mere Reality
The World: An Introduction 834
O Lawful Truth
. 181 Universal Fire 843
. 182 Universal Law 847
. 183 Is Reality Ugly? 850
. 184 Beautiful Probability 855
. 185 Outside the Laboratory 861
. 186 The Second Law of Thermodynamics, and Engines of Cognition 867
. 187 Perpetual Motion Beliefs 875
. 188 Searching for Bayes-Structure 879
P Reductionism 101
. 189 Dissolving the Question 887
. 190 Wrong Questions 892
. 191 Righting a Wrong Question 895
. 192 Mind Projection Fallacy 899
. 193 Probability is in the Mind 901
. 194 The Quotation is Not the Referent 906
. 195 Qualitatively Confused 909
. 196 Think Like Reality 913
. 197 Chaotic Inversion 916
. 198 Reductionism 918
. 199 Explaining vs. Explaining Away 923
. 200 Fake Reductionism 928
. 201 Savannah Poets 931
Q Joy in the Merely Real
. 202 Joy in the Merely Real 939
. 203 Joy in Discovery 942
. 204 Bind Yourself to Reality 946
. 205 If You Demand Magic, Magic Won’t Help 949
. 206 Mundane Magic 953
. 207 The Beauty of Settled Science 957
. 208 Amazing Breakthrough Day: April 1st 959
. 209 Is Humanism a Religion Substitute? 962
. 210 Scarcity 965
. 211 The Sacred Mundane 968
. 212 To Spread Science, Keep It Secret 972
. 213 Initiation Ceremony 976
R Physicalism 201
. 214 Hand vs. Fingers 983
. 215 Angry Atoms 986
. 216 Heat vs. Motion 991
. 217 Brain Breakthrough! It’s Made of Neurons! 996
. 218 When Anthropomorphism Became Stupid 998
. 219 A Priori 1002
. 220 Reductive Reference 1006
. 221 Zombies! Zombies? 1012
. 222 Zombie Responses 1032
. 223 The Generalized Anti-Zombie Principle 1040
. 224 GAZP vs. GLUT 1050
. 225 Belief in the Implied Invisible 1058
. 226 Zombies: The Movie 1063
. 227 Excluding the Supernatural 1068
. 228 Psychic Powers 1075
S Quantum Physics and Many Worlds
. 229 Quantum Explanations 1081
. 230 Configurations and Amplitude 1087
. 231 Joint Configurations 1098
. 232 Distinct Configurations 1103
. 233 Collapse Postulates 1111
. 234 Decoherence is Simple 1115
. 235 Decoherence is Falsifiable and Testable 1125
. 236 Privileging the Hypothesis 1134
. 237 Living in Many Worlds 1139
. 238 Quantum Non-Realism 1145
. 239 If Many-Worlds Had Come First 1154
. 240 Where Philosophy Meets Science 1163
. 241 Thou Art Physics 1168
. 242 Many Worlds, One Best Guess 1173
T Science and Rationality
. 243 The Failures of Eld Science 1187
. 244 The Dilemma: Science or Bayes? 1196
. 245 Science Doesn’t Trust Your Rationality 1201
. 246 When Science Can’t Help 1205
. 247 Science Isn’t Strict Enough 1210
. 248 Do Scientists Already Know This Stuff ? 1215
. 249 No Safe Defense, Not Even Science 1221
. 250 Changing the Definition of Science 1227
. 251 Faster Than Science 1229
. 252 Einstein’s Speed 1233
. 253 That Alien Message 1241
. 254 My Childhood Role Model 1250
. 255 Einstein’s Superpowers 1256
. 256 Class Project 1261
Interlude: A Technical Explanation of Technical Explanation 1267
Book V Mere Goodness
Ends: An Introduction 1321
U Fake Preferences
. 257 Not for the Sake of Happiness (Alone) 1329
. 258 Fake Selfishness 1333
. 259 Fake Morality 1335
. 260 Fake Utility Functions 1338
. 261 Detached Lever Fallacy 1342
. 262 Dreams of AI Design 1348
. 263 The Design Space of Minds-in-General 1353
V Value Theory
. 264 Where Recursive Justification Hits Bottom 1359
. 265 My Kind of Reflection 1368
. 266 No Universally Compelling Arguments 1372
. 267 Created Already in Motion 1377
. 268 Sorting Pebbles into Correct Heaps 1380
. 269 2-Place and 1-Place Words 1384
. 270 What Would You Do Without Morality? 1389
. 271 Changing Your Metaethics 1391
. 272 Could Anything Be Right? 1395
. 273 Morality as Fixed Computation 1400
. 274 Magical Categories 1405
. 275 The True Prisoner’s Dilemma 1414
. 276 Sympathetic Minds 1419
. 277 High Challenge 1424
. 278 Serious Stories 1428
. 279 Value is Fragile 1437
. 280 The Gift We Give to Tomorrow 1443
W Quantified Humanism
. 281 Scope Insensitivity 1453
. 282 One Life Against the World 1456
. 283 The Allais Paradox 1459
. 284 Zut Allais! 1462
. 285 Feeling Moral 1469
. 286 The “Intuitions” Behind “Utilitarianism” 1472
. 287 Ends Don’t Justify Means (Among Humans) 1480
. 288 Ethical Injunctions 1485
. 289 Something to Protect 1493
. 290 When (Not) to Use Probabilities 1499
. 291 Newcomb’s Problem and Regret of Rationality 1505
Interlude: The Twelve Virtues of Rationality 1516
Book VI Becoming Stronger
Beginnings: An Introduction 1527
X Yudkowsky’s Coming of Age
. 292 My Childhood Death Spiral 1535
. 293 My Best and Worst Mistake 1539
. 294 Raised in Technophilia 1544
. 295 A Prodigy of Refutation 1550
. 296 The Sheer Folly of Callow Youth 1554
. 297 That Tiny Note of Discord 1561
. 298 Fighting a Rearguard Action Against the Truth 1567
. 299 My Naturalistic Awakening 1571
. 300 The Level Above Mine 1576
. 301 The Magnitude of His Own Folly 1580
. 302 Beyond the Reach of God 1586
. 303 My Bayesian Enlightenment 1595
Y Challenging the Difficult
. 304 Tsuyoku Naritai! (I Want to Become Stronger) 1605
. 305 Tsuyoku vs. the Egalitarian Instinct 1608
. 306 Trying to Try 1610
. 307 Use the Try Harder, Luke 1614
. 308 On Doing the Impossible 1617
. 309 Make an Extraordinary Effort 1624
. 310 Shut Up and Do the Impossible! 1629
. 311 Final Words 1639
Z The Craft and the Community
. 312 Raising the Sanity Waterline 1651
. 313 A Sense That More Is Possible 1655
. 314 Epistemic Viciousness 1660
. 315 Schools Proliferating Without Evidence 1663
. 316 Three Levels of Rationality Verification 1666
. 317 Why Our Kind Can’t Cooperate 1670
. 318 Tolerate Tolerance 1678
. 319 Your Price for Joining 1681
. 320 Can Humanism Match Religion’s Output? 1686
. 321 Church vs. Taskforce 1691
. 322 Rationality: Common Interest of Many Causes 1696
. 323 Helpless Individuals 1700
. 324 Money: The Unit of Caring 1703
. 325 Purchase Fuzzies and Utilons Separately 1707
. 326 Bystander Apathy 1712
. 327 Collective Apathy and the Internet 1716
. 328 Incremental Progress and the Valley 1719
. 329 Bayesians vs. Barbarians 1724
. 330 Beware of Other-Optimizing 1731
. 331 Practical Advice Backed by Deep Theories 1736
. 332 The Sin of Underconfidence 1740
. 333 Go Forth and Create the Art! 1746
Bibliography
