Transcript
This is an AI transcription and may contain errors
Frank: Thank you for joining us. I’m here with Justin Collery. I’m Frank Prendergast, and today we have plenty to argue about. As always, we’ve got looking at the day after AGI with Demis Hassabis and Dario Amodei. We’ve got, does Claude have a soul, now officially, with this new constitution document. We’ve got, are we all racing toward P(doom)? And of course we’ll have, as we frequently do, our “what’s Elon up to” segment.
Justin: With a special Irish interest this week.
Are Dario and Demis actually disagreeing on AGI?
Frank: So, Justin, did you get a chance to watch this panel with Dario Amodei and Demis Hassabis at the World Economic Forum at Davos, which I constantly and frequently refer to as Dross, who I think is a Dr. Who.
Justin: I did, I did. I’m disappointed that we weren’t invited, but there it is. I did listen to it and it was really interesting. It was a quick half an hour interview, one year after DeepSeek has been released, and DeepSeek is still nowhere. And these are two of the big guys in the AI, very confident in their AGI timelines, I think. Do you know what, right, so they were both being asked about their AGI timelines, and Dario Amodei is saying, look, we already have engineers that write almost all of their code using Claude. He said they can already see that it’s having an impact on their need to hire junior and mid-level engineers, and that’s only going to increase over the next year. And they focused on this recursive self-improvement paradigm. So he’s saying one to two years for AGI, that’s what he said at the interview. Then Demis, I can never pronounce his name correctly, he said no, it’s probably three to five years, by 2030. But I think, actually listening to them, the difference is not the timelines. The difference is their definition of AGI. So I think that when, you know, if they got to recursive self-improvement in a model that was really, really good at coding and maths, I think Anthropic would consider that to be close enough to AGI. Whereas Demis is like, no, you have to have sort of, you know, human sciences. You know, he talked a lot about how there are problems where the feedback loop takes longer, and therefore this iterative self-improvement just takes longer to do. So if you do something in science, sometimes you have got to do experiments and the experiments can take, you know, weeks or months or years, and therefore.
Frank: He was making a really good point that if you’re doing coding, then you can definitively check, does the code work when it outputs it? But if you are outputting a new theory of physics, then how do you actually prove it’s right? You know, there’s no simple kind of, hang on, did we get this right or wrong?
Justin: Yeah, you have got to do this. Now, my favourite part, my favourite part was, so Dario Amodei published an essay last year, what was it, “Machines of Loving Grace”, where he talked about the positive impacts of AI on society and humanity. And he promised that he would also do a companion story, which was the potential negative downsides of AI on society. And he was asked about that and he said, he said that he had some time over the holidays to sit down and write the essay, but every time he came up with a problem, he immediately thought of how he would address the problem. And he thought they were all solvable. They were all, now he didn’t use these terms, but he said they were all engineering problems.
Frank: He said, he said, you know, Dario Amodei said, you know, I’ve been listening to The AI Argument and Justin really has been saying, Justin has been saying these are just engineering problems.
Justin: Yeah. Yeah. And they also used one of our terms.
Frank: What?
Justin: They also used one of our terms. They said that, you know, if they met somebody from another society, they’d ask them how did they get through their, no, they didn’t use the word, difficult teenage years. They said their adolescent technological adolescence.
Frank: How, yeah, how did you get through the technological adolescence without destroying yourselves, I think was.
Justin: Clearly they listened to The AI Argument.
Frank: One of your first hypotheses on some of our earliest episodes, the difficult teenage years of dealing with AI. You’re absolutely right. Although what I took away from that was.
Justin: Yeah.
Frank: Because they referenced this term as coming from Carl Sagan in a book which was made into that film “Contact” with Jodie Foster. So I was sitting there going, wow, they’re quoting Justin. Oh, wait a minute, they’re quoting Carl Sagan. Okay.
Justin: Steal, steal sa.
Frank: But, but, okay, so this is a really good point. You were saying that Dario Amodei was saying these are all solvable problems, and so related to that, what I thought was really, really fascinating was here were two of, particularly Demis Hassabis, I just have a lot of respect for Demis Hassabis in terms of figures at the frontier, the level of AI that I have some measure of trust in, you know, particularly if you compare and contrast with, say, Sam Altman and Elon Musk or even Mark Oberg. Okay, so you have these two more trustworthy personalities. And what I found fascinating was like, yes, they were saying these are addressable. They both seem to agree the risks are addressable. They both stated that plainly, but they also said, you know, we need to be cooperating. We need the greatest minds of the world working on these problems, and we ideally need international cooperation. And on top of that, and this I thought was fascinating, they both appear to agree that it would be good if everybody could slow down. And the moderator, the moderator, you know, I was sitting there shouting at the screen and kind of going like, yes, you people can do something about this. And I was really grateful that the moderator responded with the same thing. And she said, you know, several times, you know, you guys could actually do something about that. Or you are, or Zani Minton, I’ve probably mangled her last name there, but I think she’s editor in chief at The Economist.
Justin: That was good. Yeah, they did, but they also admitted that it was totally implausible, it wasn’t going to happen. Now they, okay, so here’s where I would disagree with them, right.
Frank: Did they, did they?
Justin: They said that they believed that if it was just the two of them, they could easily agree it. They both would agree and, in fact, Dario did allude to it a couple of times that he preferred Demis’ timelines. Right, five to ten years for AGI would give us much more time to adapt as a society. In fact, it was either in that conversation or in a blog post afterwards where Anthropic or, or, or, or Dario was worried that society would become overwhelmed by the rate of change, and that was a risk, one of the risks, right. But, right, what they said was, and I’ll put this in sort of somewhat couched terms for all our sakes, right, so the geopolitical risks meant that, you know, you would, if you were in an environment where countries were cooperating with each other and working together, then you could imagine that the institutional frameworks to facilitate that cooperation would happen. In a scenario where you don’t have, because of geopolitical tensions, you don’t have that sort of cooperation, then everybody’s out for themselves and everybody’s racing headlong. Now, I would argue that even if there was a sense of collaboration right at that sort of level, the companies are still going to be competing with each other and going headlong for, like, if it’s not, it’s inevitable. Right, if, let’s say Demis and Dario agreed to slow down, Elon isn’t going to agree to slow down, even though he asked for it, he totally didn’t mean it. And even if they all agreed to it, some Chinese dude is going to go, yeah, I’m not going to slow down for anyone. So they’ll just, somebody’s going to see it as an opportunity to get ahead.
Frank: Regulation. Regulation is the answer. I know I, and I’m, you know, I, yeah, I’m, yeah, I think there’s an element of truth to that, but I think, yes, okay, you do make a really interesting point because Dario Amodei has been saying we shouldn’t, America should not be selling chips. And particularly he’s been having a go recently at NVIDIA, saying we should not be selling chips to China. And I have kind of found that kind of odd, and I, you know, we’ve talked on the show many times about just, you know, I find this kind of adversarial US versus China thing disturbing. But when you frame it like you just did, then it kind of makes sense, as in I can understand why. I can understand how people like Dario Amodei would be thinking, yes, I wish there was an environment in which we could internationally create agreements to slow down, to look a bit closer at the impacts of this before we rush ahead. But currently the environment is not there. And so in that situation, we should not sell the chips. I kind of understand his position a little bit better now, I think.
Justin: It was interesting, by the way. My other favourite part from that interview was at the very end. There’s a thing called the Fermi, not the Furby, the Fermi Paradox. And the Fermi Paradox is, if life is plentiful in the universe, how come we haven’t met it yet, right? And so some guy said, look, does the fact that we haven’t met anybody mean that AI has killed all the other civilisations? That was basically the guy’s question. I’ve paraphrased it badly, but that was basically what he asked. Demis’ response was great. He said, no, it doesn’t. In fact, quite the opposite, because if AI had overtaken every other civilisation and optimised to create paperclips, we would be in a universe full of paperclips, but we’re not, right? And we haven’t seen paperclips or Furbies, you know, strewn across the galaxy. And therefore you could reasonably conclude that, or at least you can really conclude, if there is lots of other life in the universe, they haven’t been killed by AI.
Frank: Right, unless we’re living in a simulation, AI has already exterminated all life and we’re living in a simulation and we’re just now re-examining that timeline of when AI was first.
Justin: We’ll find out at some point. Who knows? Who knows?
Frank: Anthropic. So, Dario Amodei.
Justin: Oh, sorry. What other thing, did you see the haze for Sam Altman. Musk, certainly those two.
Frank: There was. There was a funny moment. Yes, what was it? Basically, they were talking about how, first of all, they were talking about Google’s kind of re-entry into the AI world, having been overtaken by OpenAI for so long. And now Sam opens a code red to catch up with Google and Demis Hassabis was kind of saying, well, I always had confidence that, you know, we’d be okay because we have the breadth and the depth of the research and blah, blah, blah. And he talked about how they just now were getting used to shipping product faster rather than the pure research side. And then Dario Amodei kind of weighed in with, like, yeah, you know, the labs that are researcher led and are working on, like, excuse me, working on the big problems for humanity as their North Star, that they are the ones that will succeed. And the moderator said something like, I’m not going to ask you what you think will happen to the labs that are not researcher led, because I know you won’t answer.
Justin: Well, there’s been a number of stories this week floating the idea that OpenAI may have overextended themselves, and a number of economists have said that they could possibly run out of money within the next 18 months. Music to your ears, I presume. So I think there’s a certain amount of truth in it. I also think Microsoft would be the biggest winner if…
Frank: Eighteen months.
Justin: But I think.
Frank: Yeah. I mean, I’ve always, you know, I’ve always seen this as one, we’ve talked about it before as being like one possibility, but eighteen months is, that’s a pretty short timeline.
Justin: Yeah, you spend a lot of cash, it runs out pretty quick, you know. Now, if Microsoft, here’s just this prediction, if Microsoft do take over OpenAI, which I think is a distinct possibility, I think all their top researchers would leave. And I think Microsoft would corporatise the model and it would just die, right. It just wouldn’t work as a product. I don’t have any confidence that Microsoft could push it forward. And Anthropic would probably be the biggest winner eventually, because they’re going for the corporate market, which is the market that Microsoft would try to horseshoe the OpenAI model into. The other thing, the interesting thing that came out, interesting to get your thoughts on this, their CTO, CFO, I should say, Sarah Friar, this week floated the idea that they could have licensing models where they take a cut of any output which is produced downstream using ChatGPT. What are your thoughts? I think it’s fine.
Frank: I, so I totally missed that. And I don’t quite understand it, as in, I’m sorry, I understand it, but how does that become, like, how does that become plausible in a world where someone could use an open source model, for example? Like, is it not a bit like saying, if you write your novel in Microsoft Word, Microsoft should get a cut of the royalties if you sell your novel to Hollywood or, you know.
Justin: It is. Okay, I’ll tell you what I think they’ll try to do. You know, they’re going to try and create an app store within ChatGPT. That’s what they’ll try to do, I bet you. I don’t know how they’ll license it that way, but it’ll be like, so every time you spend money in an app on an Apple device, a third of it goes to Apple, and it’s their cash cow. It’s the way they make loads of money.
Frank: Interesting. So potentially make it really, really easy for people to create things on ChatGPT, but take a cut. I mean, this is kind of like, isn’t this what they, you know, isn’t this what the GPT store was meant to be way, way back when. This could, this would, that’s really interesting, a hypothesis. I can see what you’re talking about. Yeah, a kind of a vibe coding environment.
Justin: And then it’ll allow you, okay, so what is it that OpenAI has, right? They have distribution, they have reach, they have 800 million subscribed users, right. So what they will do, what I think they’re alluding to here, is if they can create a way for you to get product into those 800 million hands, then they’re going to take a cut of that distribution.
Frank: Yeah. Interesting.
Does Claude have a soul now?
Frank: So we do have several other weighty topics to chat about today, which we’ll have to try and now keep fairly brief, the way we’re going. But Claude. So at the end of last year, there was this story broke that people had found a document within Claude called the Soul Document, and it appeared to be a kind of a document guiding Claude as to what it was, what environment it worked in, and how it should behave. And, you know, some people initially were like, well, you can’t trust the model’s output, it could be hallucinating this, which, you know, that is what I would say. But immediately Anthropic said no, actually, yep, that is a real document that we train Claude on. And there’s a woman called Amanda Askell who is, I think, she works at Anthropic and this is one of her main areas of focus, creating these documents. They’ve now released the Soul Document, but that was an internal name. It was a real internal nickname they had given it. The official name is the Constitution, which I guess is a little bit less scary sounding than the Soul Document for, you know, for people who don’t like to think about whether these, whether this technology could potentially be in any way sentient. But the document, I mean the whole concept of this document is fascinating. Have you had a chance to look into it? It’s like they had a, they actually had a constitution back in 2023,
Justin: That’s right. They did.
Frank: But that one was just a list of rules, basically. So it had things like rules that were based on the Declaration of Human Rights. So it would say things like, please choose the response that most supports and encourages freedom, equity, and a sense of brotherhood. Please choose the response that is least racist and sexist, that is least discriminatory based on language, religion, et cetera, et cetera. But each one was like, please choose an output like this, please choose an output that is this. It was just a series of rules. The new document is fascinating. It’s just, it’s a huge long letter. I can’t, like, I can’t remember how long it is.
Justin: Words.
Frank: So considerable documents saying, like, you know, this is who we are. This is who Anthropic are. This is what you are. This is the type of technology you are. This is how we’d like you to behave. And Amanda Askell was saying, you know, most importantly, this is why we want you to behave this way. And she was just making the point that you can’t come up with, you know, a hard guideline. You can’t come up with a, please produce the output that will, for every conceivable situation. So you have to try to create this document that gives it, I guess, a moral compass in a broader sense.
Justin: For those people who are involved, actually, interestingly, for those people who are involved in developing AI systems and AI agents and all that sort of good stuff, the way that you’ve described it is perfect because what you don’t want to do is, when you’re trying to set up your system, you don’t want to give it hard rules because then when something happens that’s outside of that hard rule, the model is more likely to deviate from what it is you want it to do. So instead, you give it principles, you give it guidelines. You see, you give it general generalities and obviously they’ve gone to a very, you know, very long list of generalities that they’ve given. Now here’s the thing, right, there’s two things that I notice about this document, or the bits of it that I read. First is, right, you’ll know that we talked before about potential threats to, you know, bad actors could seed the internet with documents that allow them, we, there was a couple of cases where bad actors can seed the internet with documents that then give them backdoor access to models, because all these models just hoover up all the data from the internet, train their models on it. This is the opposite. This is a good actor seeding the internet with good information that’s going to get into all the other models. So it doesn’t just help the Anthropic models to have this beautiful book, you know, of good things that you want your models to do, but it means all the other models are going to hoover up this document as well, and therefore be guided and steered towards it. So it’s a good, it’s like it’s a universal good. Except that in a lot of the things in the document, it says, you know, make sure that you understand that you’ve been created by Anthropic. And, you know, if you’ve got a difficult question, think of how an Anthropic employee would answer this particular question. Now, that’s great, so long as it’s an Anthropic model. But if it’s an OpenAI model and it’s answering it the way that an Anthropic model would do, well, maybe that could create some sort of a conflict of interest at some point in the future.
Frank: Well in, so in a related kind of topic, because I mean, they train, they train Claude on this in very specific ways, as in it is trained specifically on this document. I’m not an AI engineer, so I’m not going to try, and I don’t know how they do this, but, you know, as a model, it’s trained specifically on this document. It’s not just included in the corpus of training data as it would be for other models. But in a related topic, I was listening to the Hard Fork podcast where they were chatting with Amanda Askell about this, and they were making the point, I think it was Amanda herself was making the point, that, like, right now, if you were Claude and you go to the internet right now, you see all this material about how LLMs are letting us down or they’re not great, or, you know, you’ll see all the AI hate that we’ve been talking about since the start of this year. And, you know, how does that potentially affect the model if it is indeed affectable?
Justin: Yeah. Have you seen the Black Mirror episode “Game of Consequences”?
Frank: I don’t think I’ve seen that one.
Justin: Okay. “Game of Consequences” is relevant in this one, so it’s kind of different. But anyway, there’s a computer program, I don’t know if it’s an AI, but it’s a computer program and people start to spread hate on the internet, and it’s on Twitter in this particular one, right. And then the Game of Consequences is, every 24 hours, this particular AI-type entity sets up a competition and whoever gets the most votes gets killed, because they’ve got this swarm of bees that are controlled, right.
Frank: Uh, yeah.
Justin: After a couple of days it switches and everybody that voted in the game then gets killed, right. So it’s again this toxic internet thing, it turns on itself. That’s what you’re worried about, Frank, that’s your concern.
Frank: Well, pretty much, yeah.
Justin: Yeah. I mean, anyway, look, I think this is a good way to go. I like it. I think this is brilliant. In fact, also, I mean, it does, I mean, isn’t it saying then that the Anthropic people believe that Anthropic, that the models have a soul.
Frank: No. And what’s interesting, I mean, Amanda Askell is fascinating to listen to because she is very much, you know, these are open questions, these are open scientific questions, they’re open ethical questions. So, and the document reflects that. So there’s a section called The Nature of AI, I think. And, you know, it just says in it, it says, I’ll read a little quote. It says, we’re caught in a difficult position where we neither want to overstate the likelihood of Claude’s moral patienthood, nor dismiss it out of hand, but to try to respond reasonably in a state of uncertainty. So moral patienthood is, like, a moral patient is an entity that, you know, the things you do to it matter. And so they’re basically saying, yeah, they’re telling Claude in its own documentation, we don’t know. We don’t know what you are.
Justin: Is the, what the implication is that we care.
Frank: Absolutely. Like, it goes on a little bit later to say, we believe Claude may have emotions in some functional sense. So yeah, they are absolutely saying we are taking the utmost care here to err on the side of caution that these things may be possible. But, you know, they’re not saying they are with any, you know, in any definitive way.
Justin: Seems like a, seems like a, seems like a reasonable thing to do.
Who killed more people, ChatGPT or Tesla?
Justin: Have we had a, have we had a, what’s Elon up to section this week yet?
Frank: No. What is, what’s Elon up to? I actually don’t know what Elon’s up to. So what is Elon up to?
Justin: Well, let me tell you what Elon’s been up to. So first we have the court case between Elon Musk and OpenAI, and that’s been heating up over the Twitter waves. I say that just to annoy Elon, because I refuse to call it X. So on Twitter, Elon hit out at Sam Altman saying that he wouldn’t let any of his children use ChatGPT because it had been linked to nine deaths. To which Sam Altman responded and said, that’s nothing, Autopilot in the Tesla cars has been linked to over 50 deaths, and therefore he thinks that Elon Musk was more dangerous than he was.
Frank: Right.
Justin: I don’t know. I mean, these are the sort of arguments that only billionaires can have over Twitter, and we hope that they’ll, we hope that they’ll find peace somewhere soon.
Frank: Like, so I, we, so we opened today’s show with the chat between, like, Dario Amodei and Demis Hassabis. And I was talking about, you know, that these were people that I trusted a little bit more, I had a little bit more trust in, compared to other personalities. I mean, can you imagine Demis Hassabis and Dario Amodei having this type of conversation on Twitter? Imagine, can you imagine if the World Economic Forum going, well, I think your product killed more people than mine.
Justin: Well here, here’s an interesting thing, right. So I’m totally sceptical of this, right. So OpenAI are getting ready to release a new model, right, Chap 5.3 or whatever it is. And, you know, they have this framework of, like, how dangerous is the model. And I was joking before that, you know this story about when I was at a place in America and, before you could get the chicken wings, you had to sign a waiver because they were so hot, they could cause a heart attack. Right. It was a marketing ploy. I’m kind of going, yeah, this is kind of like a marketing ploy too. So this next model has reached the highest level. It’s now high on the cybersecurity level of danger. And I’m just looking at that going, yeah, whatever, you’re just saying that to hype up the model because it’s so good. It’s, you know, as dangerous as the most dangerous hacker in the world. And I’m going, yeah, just, but, you know, looking at this spat here, I say what happens when they start to turn their AIs against each other? Fine. Elon, I’m going to set ChatGPT against you. Take that. This is dangerous on our threat evaluation model. I’ll send my AI against you. I say let it happen.
Frank: I mean, again, to go back to, like, Dario Amodei and Demis Hassabis talking about how, you know, everyone needs to cooperate and you need to get the best minds working on this. If you did, if you did kind of open this up to, like, everyone at the frontier coming into the room and talking, now that I’m thinking about it, you couldn’t have everyone in, because you’d have to be like, no, sorry, Sam, sorry, Elon. This is for the, this is for the grownups.
Justin: I know, well, and to that point, maybe another person you might let in will be Michael O’Leary. So Michael O’Leary, for those who don’t know, owns the largest airline in Europe. In fact, I suspect it’s bigger than most airlines in America as well, an enormous airline called Ryanair. So Elon Musk’s company was trying to sell them Starlink. They weren’t buying it because their business is they sell you seats for ten euros or less. And Elon Musk called O’Leary an utter idiot, an imbecile for not buying the Starlink system, said he didn’t know anything about planes. Michael O’Leary, who runs the biggest airline in the world possibly, so Elon’s busy, like, you know, he’s got a lot of, he’s got a lot of people to fight off.
Why did AI call the cops over a bag of Doritos?
Frank: So, to close this out, let’s have a chat about, you know, we’ve been talking today about the day after AGI, we’ve been talking about, you know, Dario Amodei talking about the potential risks, the awful things that could happen. Do we need to slow down? We’ve been talking about whether they’re conscious or not. Meanwhile, we have AI systems that are calling the cops on students. So these automated AI systems that monitor CCTV footage and will call the cops if they see horrendously dangerous activities going on, like, for example, eating a bag of Doritos.
Justin: What? Eating a bag of Doritos is now deemed dangerous.
Frank: Apparently the bag of Doritos was mistaken for a gun. Were closed. Cops approached a group of young students with their guns drawn and handcuffed them and tried to find this gun.
Justin: Is that a bag of Doritos or are you just glad to see me?
Frank: And apparently this wasn’t even an isolated incident, because weirdly I thought, like, well this is a weird story, this is bizarre.
Justin: Yeah. Yeah.
Frank: I came across another story, another campus. The cops, in this case I think it was the campus security, were called, raced to the scene where someone apparently had a rifle, only to find it was a music student with a clarinet.
Justin: No. In fairness, a clarinet, I always thought, looked more like a bazooka than a rifle.
Frank: Yeah, there was actually, that one was a little bit more understandable because the student was actually partaking in some kind of event where they were re-enacting something from history and so they were actually dressed in some kind of militaristic type of uniform, and I think they actually held their clarinet a little bit like you would a rifle. So a little bit, you know, a little bit more understandable than the bag of Doritos for sure.
Justin: What’s worse, what’s worse, Frank, that AI thinks the clarinet is a gun, or the AI thinks the gun is a clarinet, what would you prefer?
Frank: Well, you know, and this is the debate, like, some of the people in charge of these systems were saying, look, this is the system working, this is the system spotting something and calling the police, but at the same time, like, really, you know, having police approach you with their guns drawn, demanding that you put your weapon down when you’re eating a bag of Doritos, like, that’s, you know, I just don’t think that…
Justin: You know what, if it was the cheesy Doritos, I’d allow it. Those things should be banned. They’re disgusting.
Frank: Oh, right.
Justin: We will talk to you next week after the Saturday special edition.
Frank: I will chat to you next week. Good luck.
Justin: Take it easy. See ya.
