1
I’m actually so excited about this fireside chat because a lot of people have been thinking about superintelligence and AI and I could not be more happy to speak to Yuval Noah Harari and Max TEGMARK. To both of you, we’re going to have good conversation over the next 30 minutes to try to understand superintelligence. Now, superintelligence means different things to different people. Even the big ones that are trying to build it have different timelines and different definitions. So what does superintelligence mean to you?
2
That it can make $1,000,000 on its own, that it’s an agent that you release to the financial system, for instance, and it can do everything, including opening its own bank account and it can make $1,000,000. Then it’s superintelligence and then you can have millions of those taking over the financial system. I mean, I wish we’d here can react to that because he’s made a couple of millions and maybe he’s worried about it. Max. What does it mean to you? So let’s build up to it like a layer
3
cake first. What’s artificial intelligence? It’s just non-biological intelligence. What’s intelligence? Well, for those of us who are researching and building artificial intelligence, but we simply define intelligence as the ability to accomplish goals. The more difficult the goals, the more intelligent, the more diverse the goals are. The more broadly intelligence is and superintelligence. Was originally defined in the book called Superintelligence as an artificial intelligence, which is just vastly better than humans at any
4
cognitive processes. So practically it would mean it can do every job much better than us by definition, and in fact. Could pretty quickly figure out how to improve itself and be able to be smarter than all of humanity combined. So Elon Musk thinks, you know, we could have AGI this year. If you speak to Demis Hassabis, he thinks it’s like five, ten years away. Who’s right? I don’t know. I mean, but it’s a very, very short time. However you look at it, and if it is coming, then humanity is completely unprepared for it.
5
I hope it takes longer because it means we have longer to prepare. But this is a very, very short timeframe anyway. I mean, just to put into context, I mean, is this bigger and different than the Industrial Revolution is bigger than anything. I mean, because it’s not a tool, it’s an agent. I mean, every previous invention in human history was a tool, whether it’s the printing press, whether it’s atom bomb, whether it’s an aeroplane. It’s a tool in our hand. We decide what to do with it.
6
Here we are creating an agent that can make decisions by itself. It doesn’t wait for us to decide. It can invent ideas by itself. It’s introducing a different species, nonorganic species, to planet Earth, which is presumably backed by the claims of these people more intelligent than us. You look at the history of humanity. In the history of biology, it usually doesn’t end well for the less intelligent species when the more intelligent species comes along. We were getting laughed, but I would be worried should we not worry, be worried
7
that dark humor is a good way to cope. Yeah, you know, and. The actual definition we talked about, of course, means that it’s a new species. If you have robots that are both vastly smarter than us and also every bit as agile as they can do, everything we can do, they can make robot factories and make new robots. They could reproduce. In other words, they checked all the boxes on the species definition and this is very accepted stuff in Silicon Valley. You can go read Sam Altman’s blog called
8
The Merge, which he wrote 11 years ago. He says Homo sapiens is going to be the first species to build its successor species. And we can debate about whether we want to do that or not. But that’s what it means for your question. When will it happen? This is where there is a really genuine controversy among experts. I think it’s easier to see the. Do not lose sight of the forest for the trees by just zooming out a little bit. So Alan Turing, the Godfather, he said in 1951 already that if you build a
9
basically a new species, it’s smarter than us. By default, it’s going to take control. Just walk down to the nearest zoo and ask yourself, is it the humans in the cages? Or some dumber species? And it’s the default outcome, as you’ve said, that the smarter species controls because intelligence gives power. Now, Alan Turing also said in 1951, don’t worry, it’s far away. Far away, but I’m going to give you a test so you know when it’s close. It’s called the Turing Test. It’s about mastering language and
10
knowledge at the level of a human. And since then, I was mostly overhyped. Decade after decade, overpromising under-delivering until about four years ago when it switched, the became under hyped. Almost every professor and other researcher I know predicted that six years ago, that we were decades away from passing the Turing Test from getting something as good as Jackie Beattie for say. And they were all wrong because it happened already. And since then it’s continued going faster than most of my technical
11
colleagues thought. It went pretty quickly in the past four years from sort of high school level, university level Ph.D., level two, professor level and beyond in some areas. And just last year, for the first time, I won the gold medal in the International Math Olympiad, which is sort of the intellectual version of being Usain Bolt and in the Olympics. And there’s no sign whatsoever of the race slowing down. So I think we can say quite confidently that if you buy into the basic premise that the brain is a biological computer,
12
of course we can build a better one. Some people think it’ll happen next year or two years from now. Some think five years, something ten years. But most serious technical people I know have stopped talking about decades from now. I think the amazing thing is, if you think about it, well, were you on the day that I passed the Turing test, you know, for decades people are talking the Turing test. The Turing test. Where were you on the day it happened? Nobody remembers because it’s just switched. Yeah.
13
Yeah. Nobody. And people stop talking about the Turing test. Nobody talks about the Turing test anymore. It just happened. Yeah. And the thing is that to change the world, to change history, you don’t need superintelligence. Very dumb. Intelligence is still enough to change history. We humans. Yeah. You know, you look at, say, social media, which is controlled by very, very primitive eyes and how social media change society, politics, psychology over the last ten years or so. So we don’t really need I mean,
14
superintelligence is a chimera. They keep changing the goal posts. I mean, even quite primitive A.I. is sufficient to change history and society. I don’t know when we reach the point, but I’m sure that in the next decade or so, we will have to deal with a new wave of immigration. You know, immigration, one of the biggest political issues right now, A.I. Immigration. That we will have hundreds of millions of immigrants coming from only two countries, China and the US. And, you know, it’s strange, the U.S.
15
still telling countries, close your borders to human immigrants, but open them wide to our immigrants. And I’m not against immigration. We will have, you know, a doctors in the health care system and a teachers in the education system. But they will bring problems. And the big question is, how does society, humane society, adapt to a giant wave of immigration from a different species? Can you talk, first of all, how you see it changing? So how is it changing our economy? When we have superintelligence, how is
16
it changing the fabric of society? I think people understand, but it’s actually very difficult to see. What does it mean? What does it mean? I give just maybe to two example finance. Once a ISE can act as financial agents and start investing money by themselves, making money by themselves. What happens with if a ISE come up with new financial investment strategies, which I would like? Wolf 37 of of of the famous AlphaGo game, completely new financial strategies and next stage new financial devices. The financial history of humanity is
17
humans inventing new financial devices. Money, checks, stocks, bonds, ETFs. If you remember the 2007 eight financial crisis it started was the invention of new financial devices. The CDOs that people thought for a few years were wonderful until they brought the market down. What happens if any financial agents invent new financial devices which are mathematically so complex that no human understands the financial system anymore and therefore cannot regulate it anymore? But we don’t want to regulate it because it makes trillions.
18
And then there is a crash, and not a single one on the planet understands what is happening because the financial system has developed to a stage that only a ISE understand what is happening there. A different example What happens when you raise kids from day zero when they interact with A’s more than they interact with other humans? That if you ask the child or if you want the child to see what are the main interactions of the child and how does the child’s psychology develops and things like attachment and friendship. The main interaction is with A.I..
19
What are the implications for human psychology and in society? We have no idea. We will know in 20 years. This is the biggest psychological and social experiment in history. And we are conducting it and nobody has any idea what the consequences will be. You know, just to go back to the theme of immigration, a lot of the people who oppose immigration, if they hear that their son or daughter is dating an immigrant boyfriend, they get nervous. What will happen when their son or daughter starts dating an A.I.
20
boyfriend? More nervous, Max. And this is to the point. I mean, you believe that actually, you know, the only way to make this a success is to train the line, superintelligence with the human goals. But no, at Davos, you also see, I mean, what is the human goal? I don’t actually believe that. Let me just clarify a little bit. So for you, this great question you just asked, what happens in the world when the that the jobs and so on when we build superintelligence? You know, it’s important to remember we do not have superintelligence now.
21
So it’s a huge mistake to start thinking about how A.I. is having some small effects on the job market now. And you can do reskilling by definition. I can do all. The job is much better and cheaper than our superintelligence can. So by definition. We are economically obsolete. When superintelligence comes opening, I used to have on their website that their goal was to replace all valuable human work. So forget about jobs. We cannot get paid for anything after that. Maybe society can find a way of still
22
giving some money to people. If humans stay in control. But right now the the famous control problem which people have worked on for decades, many of the smartest minds how do you control a smarter species is unsolved. Many believe it’s impossible, just like it’s impossible for chimpanzees, you know, to control us. So most likely, if we build superintelligence, it’s the end of the era where humans are in charge of Earth. Elon Musk was saying that on a stage just a month ago, you know, it’s going to be machines in charge, not us.
23
But this is not inevitable. It’s a huge mistake I think many people make. That’s what’s going to happen. As if we humans now are just some sort of passive bystanders, you know, eating popcorn, waiting for A.I. to take over the world. We’re building this stuff. And many of the most influential people steering this development are here in Davos. Right. And. So you can build it differently. Yes, totally. Counteraction. We totally can. Yeah. So, for example, raise your hand here in the audience if you’re excited about Ala cures cancer and helps us, it’s a lot of
24
hands that take them down. Who is excited about that simply makes us economically obsolete than replaces us. Okay, one guy, but nobody else wants that, right? So the vast majority of people do not want humans to remain in charge. So I think it’s quite likely we will not actually race to build superintelligence because almost nobody wants it, not just ordinary people. But but you do think the Chinese government and the US government want to have something built that’s going to overthrow them?
25
Of course not. But if you can control it, right, if you can align it with what you want, then you think that’s a possibility. Those are two totally different things, actually. I’m so glad you asked. Control means, you know, you have the power of it. You can shut it down if you want. Alignment. As opposed to without control means that we lose control over it. I is the boss of Earth. But for some reason, that decides to be nice to us. This is what most of the companies are pushing for right now.
26
And I think if it was way more widely understood by politicians that this actually implies the overthrow of the US government, you know, they wouldn’t be so cool with it. Yeah, you what I mean is this, you know, do you have to control the concentration of power, of this, of this? I think it’s not a coincidence that we see the rise of AI and the return of imperialism at the same time. And I think certainly in the US, the new imperial vision of the world is based on the assumption that we are winning the
27
air race. I will give us control of everything the economy, military culture. So we don’t need allies, we don’t need anybody. We can just take control of the world ourselves. And it’s all premised on the assumption that we are building. We are winning the air race, and it will give us everything. Yeah, except this is a bit like a Greek tragedy in that what a lot of the politicians haven’t understood yet. Which is pretty obvious to many of us scientists, is that we have no way of
28
controlling right now something that’s smart. So what will happen first is some company maybe gets more powerful than the US government and sort of starts more or less becoming the government and then they lose control over the machines. And then it’s a very sad ending for all the humans there. So the actual there are actually two separate races going on here, which must not conflict. One is a race to dominance, whereas superpowers are trying to get to dominance by building more powerful
29
tools, economic tools, military tools that they can still control. Then there’s the second race to see who can be the first to build superintelligence, which their boss is going to take over. Throw them. So if someone really wants control, what they should build is the tools and they have very strict regulations to make sure nobody messes up and builds the new species that replaces us. Can democracy survive this? Hopefully, yes. I mean, democracy’s ideally suited to survive this because we are going to
30
make mistakes with A.I., with the way we develop it, with the way we deploy it. And we need a self-correcting mechanism. We need a mechanism that says, okay, we made a mistake. Let’s go back and try something else in history. The best mechanism we know of this type is democracy. The whole idea of democracy is you elect somebody, you try a set of policies, and after four years or five years, you say, Hey, we made a mistake. Let’s try something else. In many ways, the A.I. becomes much more dangerous in a
31
dictatorial setting. Because, you know, in a democracy, say, for aid to take control or to manipulate the system, it’s very difficult to manipulate a democratic system in a dictatorship, in an autocratic regime. You just need to learn how to manipulate a single person who is usually very paranoid and very narcissistic, which is why it’s very easy to manipulate them, at least for a superintelligence. Can I add some optimism here because you’re looking a little bit concerned. No, not at all.
32
I mean, we’re having a great time. Cocktails are coming. I completely agree that we don’t need to build superintelligence. We don’t need to go down that road. And hopefully the politicians, you know, especially powerful politicians, the last thing they want is to build something that will take power away factually. And hopefully when they realize that this is serious, they will not go down that path. But it isn’t too late. And I say, I wasn’t worried. I was thinking about, you know, is it
33
too late? When is it too late to design something that is not omnipotent? If you give it all the power, then you become obsolete. So I think it goes more the other way around. Once society gives the right incentives to those who build a tech and figure out a way of doing these things so you can have your cancer cure and all the great tools, but not the out of control Skynet. So the optimism comes from. Is why I think neither the US nor the North China is going to ultimately let their own companies build the out of control Superintelligent Skynet for
34
China. You’ve all just explained why. Clearly Xi Jinping in the CCP don’t want to lose control and they have all the means they need to have to be able to stop Chinese companies from building superintelligence that can be uncontrolled right in America. Similarly, the NASDAQ establishment in America will start to see it as this is an that’s a threat, very bipartisan. But even before that, we’re seeing an amazingly crazy bipartisan coalition emerging now in the in recent months in
35
America. I call it the Bernie to Bannon Coalition, the B-to-B coalition. Yeah, these two people saying exactly the same stuff about AI there. They say things like, you know, it’s so crazy. It’s illegal for a creepy 60 year old man to be manipulating and pretending to be a girlfriend of a of a young teenager and persuade them to commit suicide. It’s illegal for a drug company to sell medicines that haven’t been tested in the clinical trial to these kids. Why on earth should it be legal for an
36
A.I. company to sell an A.I. girlfriend chat bot which has now caused many teenage suicides? They’re saying basically we need to treat companies the same way we treat pharma companies and restaurants and everyone else. First should meet the safety standards and then you can sell them. I actually think we’re going to start seeing these incentives where companies have to meet the safety standards. No one will have a clue how to make superintelligence pass any kind of safety standards, right?
37
So that means companies will innovate to build a cancer cures and make fantastic productivity gains and all the stuff you want. And the other stuff will not happen for the foreseeable future. And I think that’s great. I was going to ask, who should be in charge of I don’t know if it’s an ethics committee or something to see. You need to do it before. Right. But this is a solved problem. We know how to do clinical trials and we do have some like the company is in charge of convincing a bunch of
38
government appointed experts that this medicine here is not going to cause massive birth defects like thalidomide once did. Even in a restaurant, the restaurant is in charge of cleaning up the kitchen and making sure it’s not full of rats and persuading the health inspector that this is okay. This is a small problem. We don’t need to reinvent the wheel of how to put safety standards on an industry because we’ve done it in every American industry except I at this point. How do you see that humans changing with
39
age anyway? I know there’s you know, there’s a debate on whether humans want to be or the superior species because it’s intelligence or whether it’s a sense of belonging also and purpose that that keeps society together. Well, I will challenge our deepest identity that not not talking about superintelligence. I’m talking about, again, this wave of immigrants that we will encounter more and more everywhere and they will challenge us in many of the things we thought define our humanity, that, you know, when a robot or a car runs faster
40
than us, we are okay with it because we never defined ourselves by our ability to run faster than everybody else. We always knew that cheetahs can run faster than us. So if cars and robots can do that, that’s fine. But we defined ourselves by things like the ability to think. I think, therefore I am by our ability to create. We are the most creative species on the planet. What happens to human identity when there is something on the planet which is maybe not is scary superintelligence, but is still able to think better than us, is still much more creative than us
41
in many fields. We already saw it in narrow fields like in chess or in goal that A.I. thinks better, is more creative. This will happen in more and more fields. What happens when it comes to religion? That, you know, I have a friends, I’m a meditator and part of a meditating community. When they have an issue with their meditation, they no longer go to a meditation master. They go to another land. They go to an AA to get a device. And this is likely to happen in Christianity, in Islam, in Hinduism. What happens to religion when a ISE
42
replace the you know, especially in religions which are based on texts, on scriptures? No Jewish rabbi is able to remember all the Jewish texts ever written. I can easily do that. So if you have a question about Judaism and you go to the I am not to the rabbi, what does it mean? For human religion and for religious identity. What happens if apes create a new religion? You know, it shouldn’t sound so far fetched because almost every religion in history claimed that it was created by a non-human intelligence. Until now, this was all fiction.
43
But now it can be true. You can actually have a religion created by a non-human intelligence. What does it mean for human identity? I mean, a lot of the big tech companies will say that we’ll have more time from humans to spend quality time, emotional time with people. Mm hmm. Maybe that’s one way of looking at it. But then again, if you if you grew up interacting and much of your again, your psychological makeup is through interactions with eyes, how will it impact the way we interact with other humans?
44
You know, with other humans? Part of the issue is that sometimes they have feelings that annoy us. We are angry at them. They are angry at us. Hey, I’m assuming that it has no feelings of its own. At least you know it can feed on narcissism. It can be this thing that is always focused on you. Like you come back home from work and your husband or your wife don’t really pay attention to you. And they are grumpy because of something that happened in work to them. I will never do that. It will always focus on you.
45
Never will. Dogs. Yeah. And getting used to having a relationship with something like that. And then trying to build a relationship with a human being that can become much more difficult than in any previous time in history. So when you speak, both of you, when you speak to heads of state or chief executives of big companies that say, look, I’m worried about the picture that you’re painting here, what do I do about it? How should I look at it? What do you tell them? I tell them if they’re in government is
46
start treating AI companies like you. Treat any other government companies in your country, put up, put safety standards on them, and then they innovate to meet them. And. When I talk to people in the companies who are lobbying against all regulations, I ask them, You have all these voluntary commitments you promised that your company is going to hold to these standards. Why don’t you go lobby your politicians to make your own promises, be binding law on all your competitors? Also, I encourage you to ask them here
47
as well. Those went that was why they aren’t doing that yet. But it’s important to remember it’s not too late for us to steer towards a really inspiring future with technology. Why should you cure cancer? Why shouldn’t we solve all these other challenges that human intelligence has just done? We can do it, but that is not the path we’re on right now. Right now, we’re on this race to replace. And I’m not just talking about ultimately trying to replace all jobs. We are starting to see those of us who are parents, how there’s a race to
48
replace human relationships by having people instead put little rectangles between the humans and and even. The attention economy is shifting in more towards even an attachment economy, where some people, some children have been so attached and manipulated by eyes that they kill themselves. You know, that is the wrong direction. We need to of course, we need to really change direction and. The solution again is. I don’t want to sound like a broken record, but if we we know how to do this, we’ve we’ve we’ve decided to
49
regulate every other industry with safety standards. That’s why we can trust our medicines now. We can trust our cars. We can trust our food and the restaurants to not give us salmonella. We just have to do this as well. It’s tough because there’s a lot of lobbying money going against it. But there was justice. There was massive lobbying against seatbelts as well. You know, we can do this. What would you tell them ahead of an international agreement banning legal personhood to to a I. I think the most dangerous move at the
50
present moment is is gaining legal personhood or functional personhood. Eyes are not persons. They don’t have bodies. They don’t have minds. But in the legal and political system, we have legal personhood, for instance, to corporations. Corporations can own bank accounts. The can sue you in court. They can donate to politicians. They are considered legal persons. In India, gods, certain gods are considered legal persons. Now, until today, this was legal fiction because in the end, when Google decides
51
to buy a corporation, when Google decides to donate money to presidential campaign, it’s a human being making the decision. We say it’s Google, but if you look, you only actually a human being there, an executive, a shareholder of a trustee, you. Same with the Indian gods. It’s not really Shiva who is suing you in court. It’s the human trustees. A ISE can actually manage a bank account. They can actually manage a corporation. If you allow legal personhood to a ISE, you can have corporations without humans.
52
That might become the most successful corporations in the world. Lobbying politicians, suing you in court. And there is no human behind. I’m not. It doesn’t mean stopping any kind of research. You can continue all the research you want, but until we are sure about this thing, no legal personhood to a ISE. Which also means that, for instance, ISE cannot operate by themselves on social media. This is also legal personhood. I just want to say this is so wise. I mean. Granting robot rights, making
53
superintelligence would be the dumbest thing we’ve ever done in human history. And probably the last. Okay. Okay. And, you know, I mean, you know, people here, there’s there’s a lot of political shocks. There is geoeconomics. People are we’re inundated by news, good news, bad news. And you’re one of the greatest minds of our century. And it really actually caught my attention that you go on a silent retreat. So not many people think about their brains and how they think they just do
54
it. They have to show up every day in the office early. So how do you think about and how do you actually really take time to think and reset? And, you know, this is our most important tool, at least, you know, whether I’m a public intellectual or a politician or a manager of a company. My mind is my most important tool, and I need a healthy and balanced mind to deal with the world, especially the world of 2026. It’s crazy to invest in everything else and not in my mind. And the thing about investing in the mind, nobody can do that for you.
55
I can send my suit to the dry cleaner so somebody cleans my suit for money. I cannot send my mind to be cleaned by somebody else. It is the one thing I need to do for myself. Also, you know, in the field of A.I., we constantly talk about A.I. versus human intelligence. A.I. versus human. What do we know about the human mind? If you don’t understand the human mind, you cannot understand A.I.. You know, I hear people say, Oh, A.I. is just glorified autocomplete. It just completely just predicts the next words in a sentence.
56
That’s nothing. And then I sit for meditation and observe what’s happening inside my mind. And I’m a verbal person, I think in words, and I see words popping in my mind and forming sentences. And one of the practices of meditation is try to observe the next word that pops up in your mind. Where did it come from? Why this word and not another word? You know, just this sentence I was saying. I don’t know how it will end when I begin to say it. Like, what did they say? How it will end and not how it will conclude?
57
How it will terminate. How it will develop. Well, did the word end come from when I started the sentence? I did not know that it will end with the word end. Something in my mind kind of predicted, okay, the next word will be end. And if I don’t understand how this is happening in my mind. How can I have the kind of hubris? To say what I can and cannot do and what will be the relationship between A’s and humans if you don’t understand the human mind. And that’s why it feels like we have
58
free will, because we can’t predict what we’re going to decide until we’ve actually finished the thought process and made the decision. Gentlemen, that was so interesting. Thank you so much for for a wonderful conversation. Please give everyone a big round of applause. You’ve all.
Last edited by @suen 2026-01-26T05:03:46Z
