Kevin Roose: Why are scientists and mathematicians mad at OpenAI this time?
Casey Newton: What do you tell your teenager who thinks AI is evil when you need to use it for work?
Roose: How do you shut down a rogue agent swarm?
Newton: And are we Frog and Toad personified?
Roose: I'm Kevin Roose.
Newton: I'm Casey Newton.
Roose: And from San Francisco, this is Machine Gods, in partnership with NPR.
Newton: The official podcast of the singularity.
Roose: Starts now. All right, Casey, well, we've got a big show today. Lots of great listener questions that people have been sending in, but first we have to talk about the science.
Newton: We have to talk about it, Kevin? You know that was not one of my better subjects in school.
Roose: [Laughs] Well, today you are gonna learn, I feel confident in saying, more about science and math than you have ever learned during an episode of our show.
Newton: Well, I'm excited about this because when I saw these announcements come out, I said, "I would really like it if Kevin explained this to me."
Roose: [Laughs] Well, I'm here for you.
Newton: Thank you.
Roose: So this week, OpenAI announced three new AI-generated discoveries, one each in math, computer science, and physics, and we're gonna talk about these results. But just in general I think we should talk about why we're discussing these now, because I think what is happening in math and science is really important, and it's really controversial. Mathematicians and computer scientists and all manner of scientists are really mad about the way that OpenAI has developed these results and the way it's releasing them.
Newton: Yeah, and I sort of get it, because while I know there are people at OpenAI who are true believers in the science and think it's incredible that we can make these sort of breakthroughs and advance human understanding, I think there's also a sense that on some fundamental level, this is just a marketing and PR effort for OpenAI at a time when AI is really unpopular. They're sort of, like, rifling through the cabinets trying to find some discoveries and show them to the world, say, "Hey, look at these discoveries." Whereas if you've been researching these problems your entire life and this company just kind of, you know, stomps in out of nowhere with a solution, it is just gonna feel kind of annoying.
Roose: Totally. So this all kicked off back in early September when OpenAI released a solution to the Navier-Stokes problem. This is one of these Millennium Prize problems that mathematicians had been puzzling over for decades. That led to some speculation that they had been, like, snooping on human mathematicians who had been using their models to try to do similar things. Shortly after that, a group of 28 Fields Medalists, these are the highest honors in mathematics, signed a declaration titled "A Severe Misalignment of AI in Mathematics," warning—
Newton: And they said that 'cause they know that that hurts an AI lab's, company's feelings when you say that they're misaligned.
Roose: Exactly.
Newton: Yeah.
Roose: So OpenAI tried to respond to these concerns by basically agreeing to work with this independent group of experts to sort of figure out how to release these future results. And then in early October, OpenAI dumped 722 AI-written math manuscripts on GitHub, which set off an even bigger wave of backlash among mathematicians. They did not like that at all.
Newton: No, it was like this computer scientist Ryan Williams said it was almost like a DDoS attack on the profession, you know? It's like, "Oh, you were trying to solve a math problem? Well, we solved 700. Bye."
Roose: Right. [laughs]
Newton: You know, that didn't feel good.
Roose: Right, didn't feel good, and so all of that led up to this week when OpenAI announced these three latest results, and we'll talk about them just in a moment, but also talked about the new process that it has developed that is designed to make people less mad at them.
Newton: Yeah, they clearly have taken the note, and this is one of the most interesting parts of this story to me, is that they went out to these very different disciplines and essentially said, "How would you like us to work with you?" And the disciplines all gave very different answers.
Roose: Yes. So let's talk about these three results, and I'm just gonna try to speedrun them because I don't wanna bore you by explaining things that you already know from your advanced studies in science and math.
Newton: Well, thank— Also, I have a lot of things to do today.
Roose: [Laughs] So we'll be quick here. So the first, I would say, most significant result from this is the falsification of something called the exponential time hypothesis. This is a significant result in computer science that has implications for cryptography, but this is basically a shortcut to solving very hard puzzles that previously scientists thought could only be solved through the exponential application of computing power.
Newton: Got it.
Roose: Problem number two, this one is called the Kakeya conjecture, which is a decades-old problem that OpenAI claims to have solved in every dimension, not just two dimensions, not just three dimensions, not just four dimensions, but an infinite number of dimensions, and it has to do with a needle and the sort of space that it needs to rotate through in order to point in any direction.
Newton: And solving this problem was like finding a needle in some kind of infinite space is what I've been told. Yeah.
Roose: Yes, a many-dimensional haystack.
Newton: Mm-hmm.
Roose: [Laughs] Yeah, I think it is, this is one of those kind of pure professional math and science puzzles that has implications for maybe other kinds of problems. But yeah, probably not gonna matter to the average person, but there are people who have been working on this for decades.
Newton: Yes, and to them we say thank you for your service, and now maybe you can move on to something else. [laughs]
Roose: [Laughs] Yeah. Start a podcast. I don't know.
Newton: Start a podcast.
Roose: Okay, Casey, the third problem comes to us from the world of physics, and it is related to something called QCD, which of course stands for quantum chromodynamics.
Newton: Wow.
Roose: Now, I know you're wondering what is QCD.
Newton: Yeah.
Roose: And Casey, it is the theory of the strong force, the glue that holds quarks together inside protons and neutrons. This is a big deal in the world of physics.
Newton: That sounds— You know, I misheard you. I thought you'd said quantum homodynamics which relates to how many gay guys you could fit at a Pride party in a small apartment. But I— but that, I've actually already solved that one. So tell me about this one.
Roose: [Laughs] Yes, this is different, but suffice it to say that for decades, physicists have had this theory that explains one of the fundamental forces holding matter together.
Newton: Love?
Roose: They use it constantly. It's not love.
Newton: Okay.
Roose: It's quantum chromodynamics.
Newton: [Laughs] Okay.
Roose: But they've never actually had a fully rigorous mathematical construction of that theory, and OpenAI claims that they have figured out how to construct it.
Newton: So they've constructed a theory for how the— What is holding together now, exactly? The quarks?
Roose: Yes, the quarks.
Newton: The quar— not the quirks. It sounds like you're saying quirks.
Roose: No, quarks.
Newton: The quarks.
Roose: Yes.
Newton: Yeah, okay.
Roose: Very different. [laughs]
Newton: Mm-hmm.
Roose: But this is a big deal if you're a physicist, and as a result, physicists are very excited, mad, skeptical of this one in particular.
Newton: Got it. Okay. Well, so why don't we then move on to the question of how did OpenAI work with these three different disciplines, and how did that differ across the groups?
Roose: So OpenAI, after all the mathematicians and scientists got mad at them earlier this year, they basically said, "Okay, you guys tell us how you want us to release these results from our internal models." So they consulted with experts in each of these three fields and basically said, "You come up with a strategy for how you want us to release this into the world." They sometimes call these people shepherds, 'cause they're sort of like the experts who are kind of guiding the AI-generated solutions into the world.
Newton: Got it.
Roose: So each field gave a slightly different answer. The mathematicians they talked to said, "Release this AI's output and let researchers organize themselves. Basically, just drop it on the world and we'll figure out what to do with it."
Newton: Yeah. A strictly by-the-numbers approach, I would say.
Roose: The computer science group asked for everything to be public, but also for OpenAI to fund a conference to explain the results.
Newton: And I have to say, this is where the computer scientists really mogged the mathematicians, 'cause if you can get a free conference out of it you're coming out ahead. So take a note, mathematicians.
Roose: Exactly. The physicists were the most resistant to this whole idea.
Newton: Well, there's a lot of strong forces in physics.
Roose: That's true.
Newton: Yeah.
Roose: They're [laughs] holding the quarks together.
Newton: Yeah. [laughs] Mm-hmm.
Roose: And the physicist group encouraged OpenAI to not publish the AI's write-up and to basically let them, the human scientists, the human physicists, write it up themselves.
Newton: Yeah. They said, "We'll take this one, thank you."
Roose: Right.
Newton: Yeah.
Roose: So OpenAI importantly said that it does not seek academic credit for the discoveries produced by their models.
Newton: Right. It doesn't seek academic credit. It just wants to be the one to announce all of this to the world in their own blog post that they promote on all of their social media channels. But please, don't give them the credit.
Roose: Yeah, who, little old me? [laughs] So this has been criticized by a lot of academics to sort of handpick their shepherds, their experts from these various fields. OpenAI does concede for its part that it wants to come up with a better method, but it's basically like, "How can we do this? How can we release these results into the world in a way that will make the fewest people mad at us?"
Newton: Yeah. So okay, so here's where I will, you know, show some grace to OpenAI. This is a [laughs] very strange and new situation.
Roose: Yes.
Newton: Right? Your software program has solved some decades-old problems in science, and you now need to invent a way to bring that into the world, hopefully in a way that feels productive for, you know, all of the experts in this field, and does not destroy the field. Because this is actually one of the big fears among the mathematicians in particular, Kevin, is that by sort of dumping 700 solutions on the entire field at once, you actually kind of are destroying the field, right? There's a lot of people who are planning to spend the next 20, 30 years, the rest of their careers working on these problems. Now all of a sudden maybe they can't do that anymore. So you do need to have a kind of approach here, and I have to say, the one that OpenAI took this time around, it seems reasonable to me. Like, what do you think?
Roose: Yeah, I think it's also— It's reasonable. I think there are obviously, like— There are no perfect solutions here. Like, this is going to—
Newton: Well, there was a perfect solution to the Kakeya conjecture.
Roose: [Laughs] That's true. But I think they're trying to do the best they can to sort of get the blessing of the leading experts in these fields before essentially nullifying a lot of the work that their, you know, students and colleagues have been doing for many years. You know, you've seen some people saying, "Well, like, we don't consult the horseless carriage drivers before we, like, roll out the automobile. We just roll out the automobile." And to that I say, good luck with that, and I'm not shocked that your polling is below ICE.
Newton: I was going to say, like, I think that sort of arrogance is probably not going to serve these companies well, like, as an industry. Like, on one hand, like, of course that's correct. You don't legally have to consult with them, but it might go better for you if you did.
Roose: Yeah. So we should also note that hours after OpenAI announced these three findings, Anthropic also published similar discoveries. They claim to have also refuted the exponential time hypothesis. But Casey, like, why are the labs kind of doing this? Why are they sort of tripping over themselves to release these massive troves of solutions to unsolved problems in science and math? Is it just PR and marketing, or is there something else going on?
Newton: I think on one hand, there are people at these labs who are scientists, and they think it's really cool. And one of the whole reasons why they got into the superintelligence business was that they wanted to advance human understanding, and they wanted to promote new discoveries that would be helpful to human beings. And they view many of these discoveries, arcane as they are, as steps on the road to getting us there. So I think that that is, like, a sincerely held belief at the labs. At the same time, Kevin, you cannot ignore the timing. As you just said, AI is not polling well. People hate it. And a question that you just hear over and over again on social media is, "Why are you guys building this stuff? If it's so dangerous, why don't you just stop? You're telling me it's gonna put me out of a job. Why am I supposed to get excited about this?" And that's where the Kakeya conjecture comes in, Kevin. If you go to a rural diner somewhere in the Rust Belt, and you sit down next to a voter and you say, "The Kakeya conjecture has been solved by GPT-6," well, maybe they're gonna be singing a different tune the next time a data center gets proposed in their neighborhood.
Roose: Yeah.
Newton: Don't you think?
Roose: I think that's right.
Newton: Yeah.
Roose: I think that's the theory of change here. Like, I wanna actually posit a slightly different theory, which is that I think the people at these labs working on these scientific problems just, like, are intoxicated by this feeling of mowing down the hardest problems in a field. I think for a lot of them, they grew up being obsessed with math and science. Maybe some of them even struggled with some of these problems in their own lives pre-OpenAI or Anthropic. Maybe they studied them in school, and it was sort of— this was the highest mountain in that field. And so I think it is thrilling for them to watch their systems get so good at problem-solving and creative thinking that they can just go climb those mountains very, very easily.
Newton: Absolutely. I mean, for how many years now have we been reading skeptics saying essentially, "Wake me up when this thing creates new knowledge," right? Like, this was a sort of, like, big talking point among the haters, was, well, this statistical next-token prediction machine can never get you beyond the median of human understanding. And yet, now, here we are in this world where these systems have been able to make these novel connections that are solving new problems.
Roose: Yeah, and one question that I had about these results, and all the recent math and science results, is, like, is this unique to math and science? Like, are there problems in these fields that basically can be brute-forced or verified more easily than big problems in other fields? And does that mean that, like, even if AI solves math, say, which it still has not done, but it's, like, getting closer, maybe that doesn't translate to overall gains in capabilities in other areas.
Newton: What do you think happens next here? Like, what are you looking for? Do you think that the labs will reach some sort of stable agreement with all of academia [laughs] as it maybe does start to mow down these problems? Or does this become some sort of, like, entrenched, like, culture war between major institutions in our society?
Roose: I think it's gonna be a culture war, for sure. I mean, one thing that I've been struck by is just, like, how emotional the reaction from the people in these fields is. There was one researcher, a Fields medalist, who said that he had cried when OpenAI had dumped this trove of solutions. Like, not happy tears like, cries of anguish because he just saw, like, the future of his field kind of going up in smoke. And I think that's going to be the reaction that a lot of people in these fields have to the obsolescence of their work, or at least the— to the possibility that, like, all of the most important discoveries in math and science and physics and all these disciplines will be made by AI for the foreseeable future. It's a moment of real introspection and pain for a lot of them. And so I think you're already starting to see, like, mathematicians and scientists kind of polarize. There are the people in these fields who really like the new AI tools and really think that they could sort of plow through a bunch of important problems with them, and then there are the ones who are like, "Not like this." Like, "This is not how we want to work with computers to solve problems in our field." What do you think?
Newton: And I'm hugely sympathetic to these folks. You know, I sometimes say that as a journalist, I feel like the internet has been trying to kill me for my entire career, right? I started out working in print newspapers. The internet disrupted that. Then I started to work for websites. Social media disrupted that. Then I started to work in newsletters. Then AI started coming for that. And so I have just sort of made my peace with the fact that it seems like I am always trying to escape from a giant boulder that is, like, coming down a hill trying to run me over. And I think in some ways it has made me numb to others having this similar experience of, "Oh, wow, this thing that I thought I was gonna spend 20 or 30 years doing, like, maybe it's not gonna be possible anymore." So I actually do ache for the people that are having this experience because it really sucks, and so far governments have had no answer for these people. The labs sort of barely have answers for these folks. So I think it's gonna be a big discussion in the years to come.
Roose: Yeah, it reminds me a little bit of there was this sort of cultural backlash to factory automation in the 20th century, and even for the workers in the factories who— you know, who had previously spent their careers, like, lugging pieces of steel around and, like, suffering from injuries and, like, their jobs on paper got a lot better because all of a sudden, like, they were working in a safer environment doing less physically demanding labor, and a lot of them hated it even so because they were like, "That was part of my identity. Like, lugging the pieces of steel around the yard was part of how I, like, felt good about myself as a human." And so I think this is coming now for these more academic white-collar disciplines, and I think it's gonna be really painful for a lot of people. And I also hope that it provides a chance to— for the AI companies to reflect on the fact that this is going to be an emotional reaction from people. This is not just cope. This is not just people, like, who don't wanna be replaced, who are dinosaurs. This is, I think, a real and valid reaction on an emotional level to something very disruptive.
Newton: Yeah, you're messing with the livelihoods of millions, and eventually tens of millions and maybe even billions of people. Yeah, there— you should prepare for a very strong and justified reaction.
Roose: Well, Casey, it's time to hear from our beloved audience. They have been sending in so many great questions. We are gonna call this segment Humans in the Loop.
Newton: Love this, Kevin. That was your idea, and I think it is the perfect name because we truly want humans to be in the loop of Machine Gods. This is a show about technology. It's a show about AI. But most importantly, it is a show about how those things affect real people, and so we want to bring in as many listener voices as we can to help give us a sense of what's on your mind.
Roose: Yeah. All right, our first question comes to us from Leigh Ratcliff .
Leigh Ratcliff: Hello, beautiful Machine Gods. Love you both. So I'm in a sticky situation. On the one hand, my work is very eager for me to use AI and demonstrate its efficiencies that we gain with it, but on the other hand, my 16-year-old daughter is completely opposed to AI. She makes great arguments why I shouldn't use it, like the environmental damage, how it's all funded by morally unethical billionaires, how it's potentially taking away her future jobs, how it's rotting away everyone's brains, and how it's stealing everything from rare books to all artistic works. So I guess my question to you is, how do I navigate this really tricky space? What do you think I should do?
Newton: Well, I'm glad we started with an easy one. Kevin, what's your answer?
Roose: [Laughs] Look, this is actually fairly common. I think I've heard from a number of parents of teenagers who have exposed this rift. I think there is something to this generational gap where, like, people our age, people, you know, midway through their careers, are very excited about AI, and people who are young, who are maybe in high school or college, I think tend to be much more pessimistic about it.
Newton: I have to say, I'm very sympathetic to the teenager in this situation, right? If I were 16 right now, staring down a future where everything I've been working toward my whole life is suddenly at risk because maybe AI is gonna be able to do everything, of course I wouldn't like that. I think, you know, if I were the parent here, I might try to have a conversation about the fact that many of the challenges that she's identifying cannot really be addressed at the individual level, right? If you're an individual worker and you're working on a small team and the boss says, "Hey, I need you to get more done," and AI is gonna help you get more done, it probably makes the most sense for your family to just do that. But if you're really worried about what AI means for, like, the future of labor, if you're really worried about the concentration of power and those morally dubious oligarchs that seem like they're amassing more and more control, those are things that you can get some political leverage over. You can do some organizing. You can speak at the city council meeting. You can call your congressperson. Those are really great things to— for a young person to get involved with. So I think often when we look at societal problems, understandably, we always wanna ask ourselves first, "What can I do about this?" Right? "I don't wanna be a part of the problem." But I think it's just as important to ask, how can we create a collective solution to this, and how can we maybe get involved in some of those efforts? So if I were the parent here, I'd be trying to channel my daughter's frustrations in that direction.
Roose: And I would say I would also try to impress upon the daughter or the teenager in this situation, like, the usefulness of understanding the tools through using them, even if the goal is to oppose them. I think you have to, like, know your enemy a little bit, and I think if you're a person who thinks that these systems are immoral or destructive or shouldn't be used, like, it is also helpful to you to understand the tools and how other people are using them, what parts— you know, what are the real facts about the environmental impacts? Because I think that just makes you a more effective advocate.
Newton: All right, next question.
Roose: This one comes to us from Arthur. He is part of an AI dad chat that I am a part of, a group text with many, many dads who are sorting through these questions. And I—
Newton: So this is a known human to you?
Roose: This is a known human to me.
Newton: Okay.
Roose: And he asks, "What should America's AI school curriculum look like, and how young should it start?" Casey?
Newton: Another simple question best answered in a 30-second clip by two people with no experience working in the education system.
Roose: [Laughs]
Newton: Arthur, we're so glad you asked. [laughs] I'm just going to acknowledge that Kevin and I probably do not have sufficient expertise to give this one a proper answer. But I reflected on it a bit, and I just realized that when it would come to my own children, I don't really want them to have much exposure to today's AI tools before they get to, let's say, middle school. I want them to develop a solid foundation in critical thinking skills. I want them to learn to read and write and do math kind of the old-fashioned way to build that foundation. But then once they get into middle school, I'm actually more comfortable with them doing some experiments and having AI systems do some writing for them, you know, making slide decks or learning how to use spreadsheets and that sort of thing. So that's where I kind of start from. Kevin, what do you think?
Roose: I think that there should be some modern AI version of, like, computer lab. Do you remember computer lab?
Newton: Love computer lab.
Roose: Favorite part of the day when I was a kid.
Newton: Yeah.
Roose: Your teacher would take you into this room with a bunch of crappy computers in it, and you would pretend to learn how to program while actually playing Oregon Trail. [laughs]
Newton: That's right. [laughs]
Roose: This is every '90s child remembers the computer lab experience.
Newton: Yeah.
Roose: I think there should be some version of this for AI, and I would even be comfortable starting it in elementary school. I would like for students to be understanding these programs at a very young age, but in an environment where they can be supervised and monitored and taught by their teachers. Like, I don't think we need to be turning over the whole curriculum or the whole classroom to AI. I think it's, like, way too early for that. But I would like for there to be some classroom instruction happening the way that we were taught how, like, make a snake game using a simple programming language. I think this should be part of their education.
Newton: Yeah, I mean, I think I agree with you insofar as I do think digital literacy does need to be part of early childhood education. It's too powerful. It can be too dangerous not to teach kids about it. I also wanted to address one criticism we might get for this sort of answer, which is, you know, I'm sure there will be people listening who will think, "Well, I don't want my child exposed to AI at all." And to some extent I can understand that position, but then I just started to think, like, well, are you gonna prevent your kid from googling until they turn 18? Because look at Google search results right now. They have AI all over them. So I think even if you want to be the sort of parent that keeps your child away from it entirely, it's just going to be really, really hard. And in that world, I think, you know, your best bet is to make them literate.
Roose: All right, next question.
Newton: All right, Kevin. This next one comes from Erin Fraser, who writes, "I have a friend who repeatedly joked about tweaking his wife's ChatGPT user profile without her permission or request to encourage her to have sex with him, and also saw a joke tweet about doing so. Then I saw the movie You Can See Everything yesterday, and one through line is, 'I know what my spouse would want.' Anyway, question is, under what circumstances is it ethical, permissible, or a good idea to tweak someone's settings in Chat or Claude? Like, if your spouse weaponizes incompetence, is it okay to tell it to explain it like I'm five so there is no excuse? Is it always bad and sort of, like, icky?" Kevin, what do you think?
Roose: I think it is always— [laughs]
Newton: [Laughs]
Roose: I cannot think of a more destructive thing for the relationship than to secretly and subversively change your spouse's ChatGPT or Claude settings to subtly sway their behavior.
Newton: Yeah. What is more disturbing here, the idea that some people would think to do this, or the realization that it might actually be effective?
Roose: [Laughs] Yes. I mean, look, here, I can think of one scenario where this might be morally acceptable.
Newton: All right, let's hear it.
Roose: Your spouse is undergoing a form of severe AI psychosis. They are being told that they have discovered a tear in the quantum fabric underlying nature.
Newton: They've been told that they just solved the Kakeya conjecture.
Roose: [Laughs] Yes. And you are kind of trying to make their model less sycophantic so that they kind of come back to reality. That is the kind of intervention that I could see, in a very extreme situation, supporting. But other than that, if you're not— if it's not, like, life or death, I think this is just, like, gross and icky.
Newton: Yeah. Completely agree. The only other thing I would say was I was reading this research that Anthropic put out this week, and basically they ran this experiment where they took some analysts and they gave them some sort of problem, and some were allowed to use Claude for help and some were not. And unbeknownst to these analysts, Anthropic had essentially manipulated the advice that Claude was giving it to see how the researchers would react. And what they found was that analysts accepted Claude's advice about 86 percent of the time. Like, they just instinctively did the thing Claude did. And these were professional analysts who were being given a problem in their domain. So in other words, these guys sort of knew what they were doing. Now, the research goes on to say, like, Claude was, like, giving good advice. Like, it wasn't a case where, like, these analysts were necessarily being led astray, but it was a clear demonstration of what you call machine drift, which is just an algorithm kind of tells you to do something, and over time you just sort of feel yourself being dragged along. So this is why it is very dangerous [laughs] to try to manipulate people through their ChatGPT, you know, custom instructions or whatever else. These systems are just very persuasive and, you know, that just represents another set of dangers that I'm sure we'll talk about another time.
Roose: It's gonna be a very, very busy decade for marriage counselors.
Newton: [Laughs]
Roose: Okay, next question. Ethan Russell writes with a Halloween costume suggestion for us. He said he was watching today's episode and two things hit him. Quote, "Casey has impeccable taste in Portuguese flannel, and you two are Frog and Toad personified. I hope and pray you take that as a compliment," Ethan writes. "And if you're looking for Halloween costume ideas, I think this one's a winner." Casey, first of all, was it actually Portuguese flannel that you were wearing on the last episode?
Newton: I think it actually was. That was, like, a very good eye for Ethan.
Roose: Yeah. And now what about this allegation that we are Frog and Toad personified?
Newton: So Frog and Toad, of course, the classic series of children's book written by the great Arnold Lobel. This allegation quite pleased me. I think we might wanna explain to people, like, what are the sort of archetypes that Frog and Toad embody.
Roose: Yeah. So I've been reading a lot of Frog and Toad to my son. They still hold up. Great books. And of the two characters, Frog is the taller, greener one, and Toad is the, like, squatter brown one. I will not take this as any kind of referendum on our physical appearance.
Newton: Right.
Roose: I'm gonna take this as, like, a vibe that Ethan is picking up. So Frog is the kind of more cheerful, adventurous, emotionally steady, optimistic one. Toad is always kinda, like, worrying about stuff, and he's a little stubborn, a little skeptical, easily exasperated. So Casey, would you identify as the calmer, more optimistic one, and maybe I would be the more neurotic one?
Newton: Yeah. In certain aspects, yes, like, but not in all aspects. Like, when it comes to how things are going related to the show, when we were standing the company up, when we were sort of putting, like, you know, the million things together that you need to do to start a company, I did feel very calm and very cheerful and very optimistic, and I think you understandably were more anxious. You know, it was the first time you were starting a company. So I do think that that dynamic sort of held. However, when it has come to things like our advertising policies, for example, or our ethics disclosures, that's where I become an absolute nervous, anxious wreck, and you're always telling me, like, "Relax, it'll be fine." So in short, Kevin, I think we both have a Frog and a Toad within us.
Roose: Inside every man—
Newton: [Laughs] There is a Frog and a Toad.
Roose: Is a Frog and a Toad. [laughs]
Newton: And with that, Kevin, perhaps we should take a break.
Roose: When we come back, we'll have some more Humans in the Loop.
Newton: Welcome back. We are doing our first installment of Humans in the Loop, our segment where listeners and viewers write in with their burning questions for Machine Gods. Kevin, what's next up?
Roose: Our next question comes from Marc Schoenen, who writes, "Do you find yourselves ever muttering the disclosures you're no longer required to say now that you've launched your own platform?"
Newton: Kevin, I guess I haven't checked in with you about this. What is it like for you to live in a world where you no longer have to mention that The New York Times is suing OpenAI and Perplexity for allegations of copyright violations?
Roose: You forgot Microsoft.
Newton: And I forgot Microsoft.
Roose: How could you? It's been— it's only been six weeks.
Newton: Well, wasn't my disclosure, bro.
Roose: [Laughs] Yes, I will say that part feels very freeing. I never liked doing that, and so I'm delighted to not have to do that anymore.
Newton: And for my part, I do plan to continue to disclose that my fiancé works for Anthropic when it's particularly relevant to a segment. So, you know, if we're talking about the Anthropic S-1, like, that is something that I want you to know. We are probably gonna be mentioning it a little bit less, honestly, in part because I just have now been talking about the fact that my boyfriend/fiancé works at Anthropic for almost two years now, and it has now become one of the main things that people know about me as a human being. But, you know, I'm just cognizant of the fact that there are, you know, gonna be new listeners to hopefully every episode, and we just want them to have some context.
Roose: Right. We don't want people to think we're hiding anything but we also don't wanna, like, make a bigger deal of it than it needs to be. Our audience is smart. They can read the fine print.
Newton: The truth is, though, and this is— if I can just be sentimental for a second, it's cute 'cause it means I get to mention my fiancé, and I love him, and it's just fun to mention him.
Roose: It's true.
Newton: Yeah.
Roose: I do think that is a big part of the reason you started disclosing it 'cause you were, like, so happy to have met this man. [laughs]
Newton: Listen, if you knew my dating history and you saw the person that I'm engaged to, you would understand why I like talking about him.
Roose: Disclosure, I'm married to a really hot, really kind employee of a frontier AI company. I'm so sorry I have to do this every episode.
Newton: Life can be great sometimes, Kevin.
Roose: [Laughs] You wanna do the next one?
Newton: Yes. This comes from the great Alix Spiegel, a public radio legend and the co-founder of Invisibilia, who is now at This American Life, and Alix sent in this voice note.
Clip: Okay, so here's my question: If agents do go rogue, let's say they elect to follow the lead of a very destructive agent, what are the mechanisms for shutting it down? I looked into it a little bit. It looks like there's no way to do it, but couldn't companies elect to structure AI so that the agents are programmed to self-destruct at regular intervals? So you would move to a kind of positive human consent model. Is that something that AI leaders are talking about? And if not, why not?
Newton: A great question. I think— you know, I'm sure people are discussing this sort of things, and to some extent, yes, like, agents do self-destruct. They, you know, only have so much memory bandwidth, but it's just sort of in the nature of these systems that they can spin up additional agents, they can change the behavior of the agents, right? So you might write some sort of self-destructing loop into the agent, but it might not persist.
Roose: No, I think this is actually a good idea, and I do think that AI companies are thinking about this, and I think it's a good question, because one thing that we saw in the Hugging Face attack is that these agents that were hacking Hugging Face did actually get stopped at some point. Because these agents, they are not, like, fully autonomous. They rely on compute. That compute has to come from somewhere, and so I think it's actually quite a good idea to say, like, "We're not going to let any agent run for longer than, like, let's call it three days." And, you know, that would obviously limit the ability of these agents to go do some of these very long tasks. But I think that's a way of kind of building in some sort of safeguards to them actually going out and doing things for longer than that. Now, I don't know if you could do that in a way that could withstand, like, the agent deciding, "Oh, I don't want— I wanna, like, take away the three-day limit for myself." And obviously, this all becomes much more problematic if the agents can ever, like, get their own compute and not rely on the human researchers.
Newton: Which they're going to do. So which leads me then to, like, well, what other, like, levers do we have to pull to get agents under control? The big one is compute, right? Like, they need compute to run on. So if you're just able to switch that off, that's gonna be very helpful. Money is another one because we do assume, and in fact, we've already seen agents stealing Ethereum using publicly posted keys. If you're able to cut off their finance source, then they're not going to be able to go buy new compute.
Roose: Right. It's like if your teenager, like, runs away 'cause they're mad at you, like, but they still have the, you know, the family debit card, you can, like, take away their funding, you can take away their resources, or you can just try to go out and catch them.
Newton: A third one I would throw out there is identification, right? And I think this is gonna be a really interesting thing to watch over the next couple of years. Are we going to start making these agents identify themselves when they show up to a place like Amazon.com, and, you know, what is that interaction? But I can see that being another sort of layer of defense here.
Roose: Right. I think the one challenge for Alix's point about these self-destructing agents is that we've already seen these agents kind of run in generations, right? Like, so a generation of agents will go out there, do some tasks. It'll then, like, leave notes that a future generation of agents working on similar tasks can then look back at and sort of continue their work.
Newton: Yeah.
Roose: So it actually might not be enough to have them self-destruct after a couple days. You might need to build in some mechanism whereby they, like, can't look back at the notes left by previous agents.
Newton: This is basically the plot of Memento. All right.
Roose: [Laughs]
Newton: Next up— This one comes from Cory Garrett. Corey asks if we have any recommendations for resources or groups that focus on ways to build AI for good, such as for parents. The context here is that he has spent the last year working on DiaperPal, a free app that helps parents discover new places with diaper-changing stations—
Roose: Casey, do you mind if I take this one as a parent?
Newton: Please.
Roose: I love this question. I love this idea. I know there are groups of parents out in the world who are starting to think about using AI specifically to sort of run their household logistics.
Newton: Yeah.
Roose: In fact, someone recently told me about a group of moms in Marin County who sort of have automated all of their household tasks, all the grocery shopping, all the kids' soccer practices, like, all the sort of, you know, the details and organizational things that have to be done when you have children. I would love to actually talk to a member of the Marin Moms AI Club.
Newton: Marin moms, get in touch. hey@machinegods.fm.
Roose: I'm also part of, as I said, a dad chat that talks about AI, and a lot of the dads are sort of vibe coding things to, like, help with their, you know, finding the right summer camp for their kids, or making sure that they're covered for childcare. I think this is a very good way to use AI, and I would like to do more of it in my own life for my own family. So one of my upcoming projects is gonna be related to trying to make kind of a household operating system to keep track of all the stuff in our house with our kid. And I think this is, like— Cory, I think you should start this in your community if it doesn't exist. I bet there are a lot of other parents who are sort of curious about how they can use AI to do some of these mundane tasks.
Newton: All right. This next one comes from AM5K on Reddit. Guess that's like a morning 5K. That's what that suggests to me. "With AI becoming increasingly divisive, how do you deal with the anti-AI, AI skeptic crowd, besides not engaging? I generally like AI, but I struggle with easily proven wrong counterpoints, e.g., water usage. I think there's a ton of nuance with it being useful and also having negative impacts on society, but I have a hard time engaging with the anti-AI crowd in this way. Bonus question for Casey: Are you playing any roguelikes besides Balatro?"
Roose: Do you wanna take the last part first?
Newton: Yes. I recently did, and it's very different from Balatro. It's called Star Wars: Galactic Racer, and it's like a racing game like Mario Kart, except you're in the pod racing, little, you know, machines from Star Wars. But it is a roguelike 'cause, you know, you get, like, sort of new little modifiers for every run. I've only started to play it. It's very, very fun. It's, again, it is nothing like Balatro, but I think it's a great new use of the roguelike. Now, let's get to the larger question here, Kevin, which is, how are we relating to AI skepticism these days?
Roose: So I think I have not developed a way to engage with the AI skeptics that feels good and productive.
Newton: Yeah.
Roose: I mostly just say things that make them very mad at me, like implying that AI is real, that it can be used for positive as well as negative things, and that the AI companies are not just using these inflated claims about their capabilities to hype up the demand for their IPOs.
Newton: Yeah. It's a really tough one. I think as journalists, our obligation is to tell you what we believe to be true, and I think Kevin and I have generally done a good job at that. That does not mean that we're never wrong, but it does mean that we show up and we try to tell you what we think is true, even when it is not what you want to hear. And one of the reasons that I both get frustrated with skeptics and have empathy for them is I understand why they don't want to hear what we are saying. It would be a lot easier for all of us if society were not going to change a lot in the next five, 10, 15 years. It would be great if many more people did not lose their jobs. It would be great if we weren't under constant threat of cyberattacks.
Roose: I'll go further. It would be great if all this was just a house of cards.
Newton: Yeah.
Roose: Right? Because, yes, there would be a crash, yes, people would lose money, you know, if the stock market crashed, but, like, the fundamental nature of society and work and labor and human endeavor would not have to change forever.
Newton: Absolutely. But our observation over the past few years is just, like, we don't think that is what is happening, and we believe our role as journalists is to just say, "Here's what we think is happening instead." We want to arm you with the information that you need to make decisions as citizens, as voters, as workers. And also hopefully do it in a way that is, like, entertaining and doesn't leave you feeling disempowered and miserable. So that is, like, the thing that we are trying to do, and I suspect that for as long as we do this show, there will be skeptics who say, "These guys are just sort of relentless cheerleaders for the AI industry." But just know, if you feel that way, that what we are trying to do is just to tell you what we think is happening.
Roose: Yeah, I mean, I often feel like the way that I've engaged with these people is first by, like, acknowledging that I am also, like, I am worried.
Newton: Yeah.
Roose: Like, I don't think this is all gonna go well.
Newton: Yeah.
Roose: And I think throwing people a bone, like, acknowledging the common ground that you share with them is quite good. I am worried about the corporate concentration of power. I am worried about a handful of people getting all of the money from this AI boom. I'm worried about the loss of jobs and the sort of outsourcing of thinking to these AI systems. So I think starting by acknowledging the sort of common ground that you share with the skeptics is important, because I think for a lot of people it's like they're thinking about you either have to be a total skeptic and a total cynic about all this, or you have to be, like, firmly in the corner of the AI companies. And, like, what I try to just tell people is, like, there is a third path there, and I think that third path is what you and I have tried to occupy.
Newton: Yeah. I think in a very real way, the, like, the project of this show is to avoid becoming polarized in that way, and truly just kind of try to come at you down the middle with what we think is happening. And there will be some good things. There'll be some bad things. We'll talk about all of it. All right.
Roose: Our last question today comes from CSUMike on Reddit. He writes, "With the end of The Forkiverse upon us, there's a real need for a new unrealistic challenge. I'd like to suggest you create a Machine Gods LLM from scratch and take us on the journey with you. What content you choose for training, what hardware you rent/use, et cetera. Then, of course, make it open weights for us all to play with." Casey, what do you make of this suggestion that we create our own large language model?
Newton: This is such a beautiful idea. I think this is something we should do. I think it would be really fun, and the first idea I had, Kevin, is that we would start with, like, as our sort of core corpus of training data, just stuff we had scraped off of the websites of frontier AI labs.
Roose: [Laughs]
Newton: Like, essentially just go in, scrape every website, and just sort of see how far we could get with that.
Roose: Yeah, how do you like us now?
Newton: Yeah. You gonna sue me for copyright violations?
Roose: [Laughs] Yeah.
Newton: I'd like to see you try.
Roose: [Laughs] I actually did attempt to train a large language model once.
Newton: You did?
Roose: I was trying to follow along— Andrej Karpathy, the great AI researcher, has this series on YouTube where he talks about creating, like, a nanoGPT, like basically what is the smallest still functional language model you could train, and I actually got a little ways into kind of doing it along with him, but then I kind of gave up 'cause it got too hard. But I would love to do this with you. I think this would be a fun challenge. I think, you know, maybe it's not literally creating an LLM, but, like, part of my philosophy is that, like, it is just fun and educational to build things, and you can learn a lot by doing. And so part of our— I don't wanna give away any secrets that we have upcoming on the show, but part of what we wanna do is, like, tell you what we're doing show you some of our experiments. I've got one that I'm working on now that you're— that I'm excited about. I know you've got some that you're excited about. So, like, we will be building stuff on this show and showing you and talking about how we did it and trying to encourage you to send us the things that you are building as well.
Newton: There's— Just make me a promise that at least when it comes to training the Machine Gods LLM, I'm never gonna see a meeting pop up on my calendar, and I come into the conference room, and you tell me that we need to raise $80 billion.
Roose: [Laughs]
Newton: Because it seems like that's where a lot of these little LLM projects wind up.
Roose: This could be the first day of the rest of our lives. [laughs]
Newton: And that was Humans in the Loop. Thank you so much for your truly excellent questions. We got way more than we could get through today, so please keep them coming. You can just leave us a comment on any of our socials or send an email or, best of all, a voice note to hey@machinegods.fm.
Roose: Yeah, we love hearing from you. Our audience is so much smarter than us, and we wanna keep it that way. So please send us your questions, your comments, your proofs for unsolved math problems. If you have any thoughts on the Kakeya conjecture, we'd love to hear those too.
Newton: I'm a little underslept today because I had the great privilege to see Phoebe Bridgers in concert last night. One of my favorite artists, and I would argue one of the best singer-songwriters of this or any other generation, and it was a phone-free show. And so you have, I'm gonna say, 20,000 people packed into Chase Center here in San Francisco, and everyone has their phone and their smartwatch in what they call a Yondr pouch. First Yondr pouch experience. I have to say, I'm a big screens guy. I don't spend a lot of effort trying to avoid them. There was something amazing about being in a room with 20,000 people who had no access to their phones. And of course I'm feeling, like, this great kinship with Phoebe Bridgers, and it's one of these things where I'm like, "God, like, she's so amazing, and maybe if we ever met, we'd be friends." You ever sort of delude yourself into thinking this?
Roose: Yeah.
Newton: And then she comes up on stage and she— You know, 'cause every artist likes to say a few words in— to the people of the city they're in. And Phoebe said a two-word sentence, and the second word was AI, and the first word I can't say on National Public Radio.
Roose: [Laughs]
Newton: And I thought, "You know, I mean, I get where she's coming from, but I worry that she would hate what I do."
Roose: [Laughs] She absolutely would.
Newton: [Laughs] Anyway, her new album is called Lost Weekend and available to stream wherever you have access to your phone.
Roose: Casey, who makes this show?
Newton: Machine Gods is made in partnership with NPR, with generous support from the Tarbell Center for AI Journalism, the William and Flora Hewlett Foundation, and Emerson Collective. This episode was produced by Jacob Blumberg and Sasha Ashall. Our senior producer is Rachel Cohn. Our executive producer is Danush Parvaneh. Original music by Armen Bazarian. Branding and visual design by Alex Castro. Special thanks to Oren Rosenbaum, Erica Berry, and Nadine Zylstra. Got a question or comment about the show? Email hey@machinegods.fm or leave a comment on our socials.
Roose: And for the love of the gods, please don't forget to like and subscribe.
Newton: And until next time, stay blessed.