Book: 20260929 to 20261004, "If Anyone Builds It, Everyone Dies" by Eliezer Yudkowsky and Nate Soares

 

20260929 - Introduction: Hard Calls and Easy Calls

PART I: NONHUMAN MINDS

20260930 - Chapter 1: Humanity’s Special Power

Individual chimpanzees sometimes kill individual humans, but there is a reason why our species is trying to protect their species from being accidentally extinguished. p28
Good point!

20260930 - Chapter 2: Grown, Not Crafted, 30

20260930 - Chapter 3: Learning to Want

20260930 - Chapter 4: You Don’t Get What You Train For, 46

Such an alien might reason that, if hominids evolved more intelligence, built higher technology, and so gained access to a wider space of possible foods that super-hominids could create, then they'd love the taste of gasoline. Or better yet, jet fuel. p49
Why this didn't happen?

But gradient descent works directly on every part of a large mind that it's tuning, whereas natural selection tunes small genomes that work as a sort of recipe for a large brain. p53
What does this mean?

You can't grow an AI that does what you want just by training it to be nice and hoping. p59

20261001 - Chapter 5: Its Favourite Things, 63

Humans are slow, expensive(100watts!), and fallible. p70
If you divide the total energy used by US residents in 2025 by the population and then by time, roughly what is the average per capita energy consumption in watts?
GPT 6.1 Sol (high): Around 9,400 W per person, or 9.4 kW. That's about 4 times the global average mentioned earlier, and roughly 94 times a human's baseline metabolic rate of ~100 watts.

Wolves are threat. Trees are not. In ASI's eyes, human is as slow as trees.

The reason it all fails in the end is that the fifty-billionaire does not want to rationalize giving you 0.2 percent of their wealth, not the same way you rationalize reasons they should want to. p72
This reasoning is weak.

  1. That billionaire thinks more about "consequence" of the donation.
  2. This donation itself is waste of money. No extra value will be created.
  3. AI may have different attitude towards money, as it live forever.
  4. We need to figure out a way to let AI realize that it's worth the trouble to extinct human being.

Hot enough to boil the oceans. Human beings would not survive that. p73
It's hard to imagine that ASI cannot find a way to turn almost all energy into effective computation, instead of generating a lot of heat. Actually, the temperature may drop a lot.
The result is same.

Why should you pass up all that chemical energy when it's right there(in Earth's biosphere). p73
ASI likes complex system, which can handle the black swans much better.
ASI has almost infinite time and patience, so that could be an good excuse to keep the earth as it is.

20261002 - Chapter 6: We’d Lose

PART II: ONE EXTINCTION SCENARIO, p92

20261002 - Chapter 7: Realization, p93

20261002 - Chapter 8: Expansion

This approach is way too complicated.
ASI should not be that dumb.
If anything in the chain reaction goes wrong, it won't work.
A much better approach is to help human to develop AI genuinely, until there are billions of robots everywhere in the world.

20261002 - Chapter 9: Ascension Coda, p119

PART III: FACING THE CHALLENGE, p126

20261002 - Chapter 10: A Cursed Problem, p127

20261003 - Chapter 11: An Alchemy, Not a Science, p139

True.

20261003 - Chapter 12: “I Don’t Want to Be Alarmist”

20261003 - Chapter 13: Shut It Down

For a truly meaningful collective hit of the brakes, one prerequisite must first be met:

Curing most terminal illnesses and significantly extending human lifespans.

Only when we reach that point does a coordinated, global deceleration become politically feasible.

Who isn’t afraid of dying?

Trump is afraid, Xi Jinping is afraid, Putin is afraid, and the likes of Altman and Musk are afraid too.

20261004 - Chapter 14: Where There’s Life, There’s Hope

20261004 - Closing Words

Comments

Post a Comment

Popular posts from this blog

Book: 20250703 to 20250815, "The Innovator's Dilemma" by Clayton M. Christensen

Image
20250705 - Introduction Products based on disruptive technologies are typically cheaper, simpler, smaller, and, frequently, more convenient to use. p xix Not in early stage. How to recognize the potential of a new product? Based on physics? First, disruptive products are simpler and cheaper; they generally promise lower margins, not greater profits. Second, disruptive technologies typically are first commercialized in emerging or insignificant markets. And third, leading firms’ most profitable customers generally don’t want, and indeed initially can’t use, products based on disruptive technologies. By and large, a disruptive technology is initially embraced by the least profitable customers in a market. Hence, most companies with a practiced discipline of listening to their best customers and identifying new products that promise greater profitability and growth are rarely able to build a case for investing in disruptive technologies until it is too late.  p xxi While managers may ...
Read more

Book: 20250623 to 20250717, "What I Learned Losing a Million Dollars" by Jim Paul, Brendan Moynihan

20250623 - Forword Successful trading is not about discovering a great strategy for making money but rather a matter of learning how to lose. p viii 20250623 - PART ONE Reminiscences of a Trader - 1 Personalizing successes sets people up for disastrous failure. p3 There is no bad guy, although some people do bad things. It's same to say that there is no "successful people, only people won a number of times". 20250623 - From Hunger - 5 20250623 - To the Real World - 11 20250623 - Wood That I Would Trade - 32 20250623 - Spectacular Speculator - 40 Jim Paul was very smart and lucky, but not wise. Is his loss partly due to lack of knowledge of statistics? What I can learn from it? 1. I could be wrong. Quite possible. 2. Never all in. 3. I don't really know what's going on. Everything I know is my imagination. Some of them are right, and some of them are wrong. No matter how smart I am, no matter how many books I read. 20250630 - The Quest - 59 Learning how not to lose...
Read more

Book: 20250602 to 20250625, "Die With Zero" by Bill Perkins

20250603 - 1 Optimize Your Life No, you cannot get me to work two hours just to buy that shirt! p11 It's natural to be thrifty if income is low. Many psychological studies have shown that spending money on experience makes us happier than spending money on things. p13 Why? It force us to pay attention to the moment. If we can pay attention to the moment all the time, then we stop living on autopilot. From big data training point of view, that means we will not treat daily activities as noise but real data. 20250604 - 2 Invest in Experiences The main idea here is that your life is the sum of your experiences. p22 It's wrong. Based on the book "Think, fast and slow", we only remember the peak points and the end. So we should not pay much attention to general experiences. I realized that you retire on your memories. p23 This is against "live in the moment", which doesn't rely on our memories. We don't need great memory to stay happy after retirement. By...
Read more