Ashley: 0:00

I do really want to know whether LLMs change the way my students think about problems, good or bad. I do want to know if the offloading of cognitive tasks changes my own brain and my own cognition and like welcomes dementia earlier or pushes it off, right? I really wanna know these things. We need to know these things as a society, but to pretend like this paper answers those questions is just completely wrong. So we're gathered here today to talk about a paper that you might have seen. It's titled Your Brain On Chat GPT. I've gotten texts about it, we've seen it on our social media. I'm honestly at this point, surprised my mom didn't text me about it. Like, this is how far reaching this paper has gotten. But we have concerns about it and we wanna talk to you. Yeah. We have thoughts,

Cat: 1:02

We have many, many thoughts. Yes. So let's do a general sketch of this study and what people are saying about it.

Ashley: 1:07

The study is essentially asking this question of what happens when you use an LLM or the internet or just your brain to write an essay.

Cat: 1:18

I'm gonna say, Ashley and I have rerecorded this little bit about the methods of the paper. Four times. And I just want to say that because that is a sign of how confusing this piece was to read. The moment I opened up this preprint, I posted, what am I looking at? And then I said, I'm not gonna read all this on a Sunday night. So we've tried to read it all for you. Okay. But I just wanna shout that out. It's really confusing. Hard to parse paper. And that was my first red flag. Okay. Because we try to write science in a way that is really clear and in a way that brings people in. And this artifact is baffling. So general sketch of the paper, as we can tell from the methods it's done at a couple of different campuses. Primarily led by a researcher with an MIT media lab affiliation. There's 54 participants from whom they've reported data. They assigned these people to three different groups. Participants in one group were told to use open AI's GPT-4 model as their sole source of information for the essay writing task, quote unquote. So all of these people were brought in and told, you're gonna do an essay writing task. So, LLM group uses chat, GPT, search Engine Group., They were told they could use any website to help them with their essay writing tasks, but any LLM was explicitly prohibited. And then there was what they called a brain only group, which is gonna be very important in this paper brain only group for forbidden from using LLM in any online websites for consultation in their paper. I would like to make the point that everybody in this study had a brain

Ashley: 3:02

my brain is thinking so hard about how to talk about this paper in a way that both empowers people and also takes away a lot of the fear that this paper has created.

Cat: 3:14

Mm mm mm Would you say that you have Alpha Band peak activity, or is this theta or what's the deal?

Ashley: 3:23

I would say it's a mix of all four alpha, beta, theta, and gamma simultaneously.

Cat: 3:29

Okay. Does that mean you have massive connectivity going on?

Ashley: 3:34

I have so much connectivity going on in my 86 billion neurons in my brain. And so as they're doing these tasks, they're also recording from all of those brains that the participants have using EEG. Uh, they also record their eye movement, but they don't report it in this study.

Cat: 3:51

Yes, you can see some dazzling little red glasses in the drawn figure. I was confused by that. Um, they did a bunch of things in this paper, so they talk about three big things. They basically talk about. The people like people outcomes. So things like how well did people remember the essays they wrote and their sense of ownership of the essays, the questions that they ask people seem very ad hoc, like they were just written by the researchers. Uh, when I look at this as a social scientist, I think to myself, why are we not using scales that other people have, you know, designed the items for and tested them over large populations? I mean, sometimes you create novel measures, but there's basically nothing reported about the measures themselves here. Um, so that's one thing. There's obviously the EEG results, so that's a huge area that we're gonna talk about the most. And then there's a bunch of analysis about the text itself and we're. Not really gonna dwell on that because there's a lot there actually. But it's just not our focus. And, um, it's not very surprising that LLM generated text looks different from human generated text that's really widely

Ashley: 5:08

already knew that. Yeah.

Cat: 5:09

give people the same model. They will obviously create fairly homogenous text. And this to me gets into the research design question of the study, which we have big, big questions about, um, a large part of the claims in the EEG sections center on ideas about observing certain patterns of connectivity. And I think that's a really major thing to take away here is that that's a big claim that we're pretty skeptical about.

Ashley: 5:38

Yeah, definitely.

Cat: 5:40

This is all sounding very intimidating. Alright, Ashley, we're putting you in the hot seat today and we want to do an overview. We want to talk about our brains on brains, right? How our brains are thinking about brains and this evidence that has been put in front of us. And I, I think that this paper that we're gonna talk about today, this preprint um, this dump of notes,

Ashley: 6:03

I,

Cat: 6:03

it, it's a document in the software file sense.

Ashley: 6:10

Thank you. Because I'm not even sure, like bioRxiv would accept this. I'm just, I'm sorry, is that,

Cat: 6:16

oh, Does.

Ashley: 6:18

went straight for it. I'm

Cat: 6:19

Okay, Dearly beloved, change technically heads. We are gathered here today to talk about a paper and to talk about how to read a paper and to talk about your brain, our brains, and how we don't have to be afraid of brains and maybe we have to be a little bit afraid of the way people talk about brains, but that's okay. Like we are here to help you.

Ashley: 6:46

so I think you and I are sort of weirdly perfectly positioned to talk about this paper because I am a neuroscientist and I think a lot, not only about obviously how the brain works, but also how people talk about brains and use neuroscience information. Sometimes even like as a weapon, I think that's a strong word in this case, but really as a way to like sound like an expert or sound like really technical. And that deeply, deeply, deeply bothers me and sort of my mission as a person is to say, look, the brain is an organ. We can understand it in these ways, but there's also these other things that, uh, people claim to be able to do, to understand it, that are just like too farfetched and the claims are too far. And yeah, so I feel like that's my side of it. And then you're obviously here as the, you know, psychologist and social scientist thinking about a lot of other parts of this paper.

Cat: 7:40

Yeah, so I'm really interested in how we promote and protect human learning in the world, and I've always been interested in that and I have a lot of thoughts and questions about. The ways the folks in this paper tried to study it, and I think it's a really good example of how much research design matters. Um, and also, you know, that we can break this stuff down, you know, the proof is in the pudding. So take a look at the evidence and, and take a look at what someone is showing us and what they've said they've done. And that's how we can unpack these sort of really big claims and headlines that are going around. Right. So maybe let's start by talking about EEG, because this paper has really gotten a lot of traffic and traction because they used EEG. So Ashley, maybe you could give us a mini EEG lesson.

Ashley: 8:32

It's a way of recording from the brain. It's non-invasive, which means we're gonna put electrodes on the outside of the brain. And, um, some of you have maybe even like, been in an EEG study or seeing one of these caps at like a science fair or something. It's very typical thing you'll see for neuroscience recording where someone has like a bunch of little electrodes on their head. Here's the thing about EEG, which is really tough, is that it's outside of the brain. So the signals that you actually record with an EEG electrode are pretty poor in terms of, uh, their ability to be spatially localized. And they're pretty poor in terms of their signal to noise. And when we record EEG, we're not recording single neurons. We're actually recording like thousands of neurons simultaneously, all talking. So the analogy I often use is, and this is not, um, something I came up with, but I think is really, really powerful, is like to think of your brain as kind of like a stadium. And within that like massive stadium of let's say like 30, 40,000 people, every individual person is like a single neuron. And if you were to study this stadium in a few different ways, you could go in and listen to one person talking, but you could also listen to the overall roar of the crowd that happens. And what EEG is listening to is it's listening to the roar of the crowd. And that roar can happen in different frequencies. Like you might have some low frequency, slow thing that happens, like as the ball moves around the field and everybody's like, Ooh, ah. But then you might also have higher frequency, like people clapping and getting excited or even like doing the wave, right? So things are happening at these different speeds within the stadium, and EEG can record all of that, and then we can break it down into these different frequencies of activity.

Cat: 10:22

I think that something that also comes to mind. So, you know, I, I actually did my PhD in an EEG lab. I will fully admit this. I think EEG is kind of boring as a method, so I didn't do it. But, so that's my, my clarity. I like to say I did my PhD in an EEG lab. I did not myself perform EEG. But something that I know from that context is how you analyze this really matters. And I mean, the choices that you make about how you do the analysis bake in certain findings. Um, sometimes, you know, so this is true of many kinds of analysis. You, the model you're using, you know, the kind of assumptions you're making can determine what you find. Um, and this is one reason that research design matters so much. And the answer. To, you know, whether this is good evidence is not always in the statistics, it's in the analytical details and the analysis plan. And is that a good plan? Right? So can you say anything about how we analyze EG data?

Ashley: 11:23

Yeah, totally. EEG is one of these techniques that is super easy to do. Like you actually, if you had like a little bit of background in electronics, you could build an EEG system. It's basically just a little electrode like metal thing connected with a wire to an amplifier. So the,

Cat: 11:40

This is what makes it, this is what makes it a hacker magnet, I will say.

Ashley: 11:44

Totally. Yeah. And, and in recent years, in like the past couple of decades, these systems have come down dramatically in price. I mean, you could go online and order yourself a pretty straightforward, like little EEG cap probably for a hundred bucks or something like that. Um, backyard Brains, which is one of my like favorite, uh, companies for democratizing neuroscience education, sells an EEG system. It's a few hundred bucks. So. This is the kind of technology a lot of people are getting their hands on. A lot of companies are getting their hands on. But as you said, the analysis is really, really the thing that matters here. And like I said, EEG is noisy, it's indirect, and you have to have a really clear idea of what you're looking for and align what you're looking for with the analysis that you're ultimately gonna do. So the most like common thing or sort of like the most bread and butter thing you do with EEG data is you say, okay, we've recorded this roar of the stadium over time. Let's see what kinds of frequencies are in the data. And so you break it down into alpha and beta and theta. And this kind of thing has actually gotten into like the mainstream a little bit because we have this idea that like, oh, gamma means you're like concentrating. And so you

Cat: 12:59

Yeah.

Ashley: 13:00

YouTube channels that are like gamma channels meant to like stimulate your

Cat: 13:05

I've seen this for meditation stuff like there is off the shelf. EEG headbands, people buy and it, it does a huge amount of processing that it doesn't really tell you about. That's kind of opaque. And then it says, oh, you, you access this band or not, right?

Ashley: 13:21

Yeah, totally. And, and some of this stuff is. Is totally legit. And, and the stuff that has been around for a while like this, like I said, kind of bread and butter, just like what frequencies are in the data. Some, some of these things have held up a lot over time. So for example, um, the alpha waves, which are these waves that happen between like eight and 12 hertz, so you know, about 10 times a second, you have a cycle of this wave in the brain. Um, you see many more of these when your eyes are closed, when you're relaxed, when you're meditating. I actually do an experiment in my lab class where students elicit their own alpha waves by just sitting there with their eyes closed and then you can see it. Right in front of you in the data. Like it doesn't even require very sophisticated analysis. Um, also as an aside, this has really made me believe in meditation because the students, just anecdotally, the students that have the strongest alpha waves, I like nine outta 10 times are meditators. Um, and it's super interesting. So, so there, there's real stuff and there's really interesting stuff that we should be trying to understand as a neuroscience community. Like what is this alpha wave? What does it mean, you know, why does it, uh, seem to increase with things like meditation?

Cat: 14:36

You said something really interesting, which is, you know, just by sitting down and closing your eyes, you get this change. Like that to me, shows me, shows me how dynamic the brain states are, you know, how quickly things change, how complicated it can be because you know, your brain is doing everything at once, like motor movements, you know, I mean, if you have one person who's doing a bunch of waving their arms around, right? You know, their brain's gonna look different than someone who isn't. So I think is that part of what you mean by indirect, it being an indirect measure?

Ashley: 15:11

Yeah, that's part of what I mean. And yeah, what I mean by indirect is like if I wanna know, uh, let's say if someone's relaxed, yeah, sure, I can, uh, look at the alpha wave activity, but if I wanna know if someone's thinking of their grandmother. EEG activity is not gonna tell me that it's, it's gonna tell me something about someone's brain state, how engaged they are, how awake they are. That's like generally what I think of EEG as sort of being useful for, it's a, it's a measure of brain state, but it's indirect. It's not, you know, the same thing as like going in, measuring the activity of your dopamine neurons directly to ask how rewarded do you feel right now?

Cat: 15:52

It's not precise. It's not. It's not. Yeah. So it's not like, it's not specific and it's not precise.

Ashley: 16:01

Yeah, totally. And, and something you just raised too is the fact that at any given time you have all of these different frequencies happening at the same time in your brain.'cause your brain's doing a lot of different things at once. Right. So, you know, yeah. You have some low frequency stuff, you have some higher frequency stuff, but given your brain state you'll have like a different sort of mix of these different, um, frequencies happening in the brain.

Cat: 16:25

Okay. Alright.

Ashley: 16:27

yeah, so that's like the, the EEG sort of fundamentals, right? This paper is doing a step away from that, which is, I would say not fully accepted in the world of EEG, which is attempting to measure connectivity using EEG. So there is a community of researchers that use things like FMRI or EEG, which are functional measures of the brain. So they're recording brain activity over time. They're not measuring structure, and they're using those functional measures to try to estimate how connected different areas of the brain are. This is not a direct measure of connectivity. It is not the same thing as me going in and asking, is this brain region actually physically connected to this other region? It's actually asking a question of. Are these brain regions kind of correlated in the fact that like, if one is active, is the other one active? So first and foremost, we need to like address that as what connectivity means in this paper.

Cat: 17:33

okay. Okay. I wanna jump in 'cause I think, you know, there's some interesting stuff to interweave. I've seen a lot of misconceptions as people talk about this paper a lot. Okay. Just baffling levels of social media hype. One thing I wanna say is I've seen a lot of posts that say, oh my gosh, this paper shows that the brain's ability to form new connections is damaged. That is not, that's not at all what connectivity means, right?

Ashley: 18:02

No, no, no, no. Yeah. No, not in this case. So, so this paper, uh, is indirectly measuring connectivity by asking,, does activity in one region, predict activity in another region. They're using this technique that's based on this thing called Grainger Causality, which actually was born out of social science and is this way of trying to estimate whether time series are related, um, which is cool. You know, like in some cases, like you wanna know if particular, you know, if the stock market is correlated to something else happening in the world, you don't have a way of running a causal experiment. So you do this thing called Granger causality, and you ask, is there enough correlation between these two signals in one direction that it seems like one thing is causing the other thing. So that's okay, but it's still not connectivity, right? You haven't directly indicated whether there's a connection between these two things, but this has nothing to do with synapses, right? So when we think about, like you said oh, forming new connections in the brain. What we mean really when we say that is forming new synapses between neurons. Like I said earlier, this paper, EEG, is not measuring single neurons ever, ever. ever. it's measuring thousands of them simultaneously.

Cat: 19:17

yeah, it's not even measuring people's learning outcomes. It's really important to think about what they're saying versus what they're measuring. And they're saying a lot of things about learning and they're saying a lot of things about cognitive debt, which isn't a psychology term, it's just a term they, they made up for this paper. So I don't even know exactly what it means. But they're implying that there will be learning loss. They are not measuring learning loss, they're not measuring a difference in people's ability to solve a new learning task. Um, they're not even measuring self-efficacy. So people's perceptions of their own learning, , or people's belief that in the future they will be able to learn. I think that that's really, really important. Again, just to distinguish between what they're talking about or what they're saying this could have implications for versus what is actually measured, what we actually know.

Ashley: 20:09

Yeah, totally. Do you wanna pivot to that? Because we talked about the EEG measures and, and I mean, you know, the EEG measures. Basically, just to kind of summarize in one big takeaway, like I think, okay, they measured brain activity, they threw this analysis at it, which by the way, there's like a page on this or two pages in the methods is not nearly enough to even assess whether they did this method correctly. Um, and whether they ran the statistics on it correctly. I also think that they're talking about it incorrectly, like for example. This directed transfer function that they're using is actually, was actually developed to look at all of the electrodes. Simultaneously they mentioned looking at pairs of electrodes, which doesn't make sense. I

Cat: 20:59

I had huge concerns with that.

Ashley: 21:01

pairs of electrodes as actually nodes in the network.'cause this is essentially a derivation of network theory anyway. There's a lot of stuff that's like, sort of just weird about the way they write about it and talk about it. No, you

Cat: 21:13

No, you know what, I'm actually, let's not just do this as an aside because this is, this is central. This is central. When I read this paper, I, and I took a lot of notes to myself and we can link a couple of my threads if people wanna read those.'cause it's, a, it's,

Ashley: 21:27

this thing and why?

Cat: 21:29

a 200 page artifact. And I was like, nobody's gonna read it. I'm gonna read it. Challenge accepted. Um, it's about a hundred pages of, of the actual paper,

Ashley: 21:37

I read it on the trolley. I mean, half of it is brain pictures, which are like hard to even interpret, like, because what

Cat: 21:45

is

Ashley: 21:46

It's just there to basically be like, look brains. We did it

Cat: 21:50

okay, we're getting into it. So one of my first thoughts was this analysis is really suspicious. So I want to teach our change technically heads our lovely folks who are here to learn. How can you think about red flags when you see a study? You know, and, and here's some red flags that I thought about. There are tons and tons of pairwise comparisons being made and I don't see a holistic analytic approach that kind of controls the way that we are testing for all these things. That's a very important thing to do. A lot of the time when you are running tons and tons and tons of analyses, you, you want to make sure you are not accidentally inflating your chances of getting a positive result by accident. I don't know EEG as well as you do . Let's talk a little bit more about this analysis.

Ashley: 22:47

Oh, yeah. No, I think their methods section is just really strange. So it like, if you look at the methods section for the EEG, half of it is equations, which by the way,

Cat: 22:56

You don't need to write out

Ashley: 22:58

You don't need to include the equations for stuff that's published in other papers. Immediately to me it signals we're like, we want this to look smart, and we basically want to prevent people from directly engaging with it. Like,

Cat: 23:12

Yep. It, you know what it is to me, it's computer science, paper writing, because this was how people would do it in, when I was in an HCI lab for my postdoc, papers would just have like 20 pages of basic statistical equations, like, you're not reproving what a regression is.

Ashley: 23:31

yeah. Exactly. Exactly. And this stuff is, is is fully written out in the papers that developed this as a technique and the much more common thing to do would be to say we adopted this technique. They're not as far as I can tell, doing anything. Different or novel with it. So you just refer to the paper. So like why did you take the time to put the equations in? Well, at the same time, in like the first paragraph of the EEG analysis section, there is a blatant typo that even my students could catch like this. Actually, I would give them as an exam question, which I would say, what is wrong with this paragraph? And they would tell me that if I ran an analysis where I did a low pass filter at 0.1 hertz and a high pass filter at a hundred hertz, I would've lost every single frequency that I care about in terms of the brain.

Cat: 24:21

Oh, that's alarming. That's alarming.

Ashley: 24:23

it's alarming. So how do you get an error like that, which is just blatant and like, okay, I'll give them a benefit of the doubt. It's a typo. Fine, but you know, then you have these complicated equations like about your transfer function. I don't know, it's, it's major red flag territory.

Cat: 24:40

I felt the same about the learning science stuff because a huge beginning motivator for this whole study is, Hey, maybe people form better memories when they write something and generate it themselves versus when they copy paste something or use assistance. This is a classic, classic cognitive science learning, science psychology finding. There's a huge area on it. They cite Sweller 1970s cognitive load theorist. Um, they cite basically the one guy that HCI people ever cite about cognitive load. And I, I find their discussion of cognitive load incredibly poor in the literature review, but they do not cite. Bjork and Bjork, who are foundational theorists who talk about the concept of desirable difficulty in learning and demonstrate memory formation and the generation effect, which is essentially what they are claiming to rediscover here. It's a completely known effect. It's a classic effect that if you have people generate things, they will remember it better. Like we've replicated this again and again called the Generation Effect, and it's weird to not cite that. You know, it's also means that if you design two tasks where you have one person write a bunch of stuff and you have another person passively read stuff, no matter what the medium is that they're reading it on. Static book a screen, an LLM interface, I don't care. The reading effect will be different. This is completely known and documented. So the fact that they don't cite that or justify, you know, what's different about interacting with an LLM, um, in a really precise way, huge red flag to me. It's also just not good scientific practice. Listen, we all have careers that rise and fall on getting credit for our work. So, you know, you read this and you're just like, excuse me.

Ashley: 26:39

Yeah, no, there's a, there's a ton of stuff that should be cited in other parts of the paper too. Like overall, it's like really under researched. They're making broad claims with one measly study, which like in my mind, especially for this kind of world of EEG, like just saying, oh, well this particular kind of connectivity means they're doing less internal processing. Like that's a really big claim. And you're gonna just pin one study on that. I'm sorry. But extraordinary claims require extraordinary evidence.

Cat: 27:13

Amen. And that is something to also pay attention to, even if you're not a neuroscientist. You know, you can read a paper and you can say, huh. There we're jumping, we're like on, on one hand we're saying, oh, there's like this tiny difference of attention or something. And then the very next sentence we're saying, difference in internal processing. Like what is internal processing or, or difference in neural activity. When I was in grad school one time in front of my PI, um, who, who came from a very famous theory of mind lab, I said that a cognitive process was higher level than another cognitive process. And he interrupted me and said, what do you mean by that? What is high level and low level in cognition? And I didn't know, you know, when I was just a student learning and I was kind of parroting what I'd sort of heard. But it was, it was vague metaphorical reasoning. It wasn't actually a clear mental model and I never did it again because, you know, he said. What do you mean? You can't just call things high level and low level, you can't just hand wave.

Ashley: 28:22

Gosh, that's so good. That's so good. And I think like a big part of what we do in what we try to do in rigorous science is call each other out on stuff like that, you know? And that's the process of becoming a scientist is getting called out for making assumptions. Um, like that, that might be a little bit flawed. I got called out when I was an undergrad. I will never forget this. And so I will never make this mistake I had. A mentor who set me up when I was, oh no, I was a high school student, actually. I was young high school student, knew I wanted to do neuroscience research. My mentor set me up with a meeting with Jonathan Cohen, who is a famous psychologist at Princeton. I went, met with him. I told him something like that. There was a selfishness area of the brain or something, like referring to some headline I had seen. And he was like, um, that's, that's not, that's not how we think about it. You know, like there aren't areas of the brain that are responsible for behaviors in this, like one-to-one sort of thing. And I was like, oh, okay. And I'll never make that same mistake again.

Cat: 29:28

People are trying to scare you by saying something happens in the brain like a lot of the time. Seeing the social media takes, you know, again, which I just go back to because I, I think about how. We can help people navigate this, you know, and I see people who are so scared, and I just want to say, think about, you know, if your brain was gonna melt out your ears because you were bored for 20 minutes, for four times, what, what evolutionary advantage would that be? I mean, I would be more terrified by that than by anything about we have much bigger problems to solve than

Ashley: 30:13

Oh my God.

Cat: 30:15

And, and that's what people are saying like that, that is a ludicrous.

Ashley: 30:21

Oh my God. I think your point too about like, okay, if this actually was a study where they were causing cognitive impairment in like a third of their people, that would be a major ethical thing, right?

Cat: 30:32

Holy shit. So, you know, my trick, my secret trick that's gotten me to my whole career and makes people think I'm a crazy smart research design architect, whatever, whatever, is that I actually read papers and I think to myself, what did people have to do? What did they physically do with their bodies? What did these people feel? What did they think? And so what really happened in this study is a handful of people, about 54, 53, um, of which a bunch of them dropped out before the fourth session. You know, by the way, so the fourth session's only like nine people showed up and. Participated in a series of, I think, 20 minute sessions. Four of them. Very unclear over what time period, by the way. So I couldn't find this in the paper. I don't know if you did, babe, but I couldn't find a place where the intervals between the sessions were actually shown to be the same for everybody. Or whether it was just dependent on when they could schedule people. I think if you're talking about overtime effects and you're sampling people at different intervals, that matters a lot that's the sort of thing a learning scientist would be thinking about if we were actually testing for memory. Obviously massive confound is how long ago did somebody do something? So You wanna

Ashley: 31:52

want that to be as controlled as possible. Like it's always a struggle to, you know, schedule participants, of course. But you'd wanna say every session was five to seven days, let's say apart. Right. But none of those details are in the paper. You're absolutely right.

Cat: 32:08

Another thing that I've made a point about in this study is despite the massive amount of detail, the fact that it's a hundred pages, we have little clarity about the instructions participants were given. So there's a lot of places that this paper flips into sort of narration that's not very detailed, and they say, we gave people these instructions. I wanna see the text by text. I wanna know what the participants really thought they were supposed to do in the LLM condition. Did they think they were supposed to copy paste? You know, because it's, it really matters what people think they're supposed to do in a condition. It also matters that you're getting paid a hundred dollars to show up and sit in this condition. There's a lot of evidence that the people in the LLM condition were bored out of their minds and were doing a task that they thought was stupid. And I think that is different from the task making you stupid.

Ashley: 33:09

Yeah.

Cat: 33:10

You could argue that the measure they have that people were disinvested from the task, they didn't feel ownership of the essay, actually represents a real validity check failure. And I'm not sure that I'm expressing this super clearly, but I'm trying to explain this to people on my social media right now, like when we design tasks that are different, we are forcing them to be different. So it's not a surprise that people experience them differently,

Ashley: 33:37

Yeah, yeah,

Cat: 33:38

And it doesn't prove this is how people are using chat gPT people have really complex lives. Like we can't just leap from this to being to saying this is now a measure of how everyone in the world is using chat, GPT. That's ludicrous. And when you're testing things like human memory and human problem solving, you want to design for tasks that are. Are equivalent in a lot of dimensions that we want to hold equivalent and allow us to test for these differences we think are because of the LRM or because of the search engine.

Ashley: 34:12

Another way to say this point is like you wanna control for something and you're pointing out that the LLM condition maybe didn't control for the fact that participants were disengaged and bored, and that's actually not the use case for LLMs that we think of. Like when I, I just recently learned this term, so I'm gonna use it when I vibe code with my LLMI am deeply engaged, right? I'm like, Hey buddy, let's go. Let's make this simulation, let's do it. Like it's fun, I'm engaged. This is like a good use case.

Cat: 34:51

This is fascinating, and it's also classic learning science. Okay. They say it in the lit review of the paper. If you show up with high self-efficacy, if you show up with a lot of cognitive flexibility, that is probably gonna change the way you use LLMs. And then they don't measure anything about that in the study. They don't measure whether their participants are coming in with different levels of self-efficacy. So they're not checking all kinds of baselines about the participants. They cite all this research, uh, they actually don't cite enough research, but you know, they cite some, at least, I mean, I think it's bafflingly bad lit review. I would give it, you know, a d if I were teaching a research methods class and someone, you know, at this level gave me this, uh, you know, for instance, like they have a section where they're talking about what LLM users do or don't do. They don't define what an LLM user is. I don't think we have a comprehensive observational, you know, understanding of what people do with LLMs, quote unquote in the world. Like there's not, this is not a trait that you're born with. You're not born a LLM user. Like, what does that mean?

Ashley: 35:59

I think you're born a vibe coder, though.

Cat: 36:01

Yeah, you could be, you could be, but these are like, then these are deep, these are psychological things we could talk about. Oh, do you have a high, you know, openness you know, do you have a, do you have a large amount of creativity? Do you come in with a lot of self-efficacy? I mean, you know, I spend a lot of my time in technical circles arguing. We can measure big psychological states people are in, you know, and then learn about how they're interacting with their environment because of that state. But also the environment causes the state as well. You're giving someone a disempowering task. You're telling them not to write an essay, but to like use Chat GPT to write an essay. You're giving them a message about their ownership. So you're baking the finding in, and I just gotta get this out because it really freaking bothered me that there's like this generalization about what LLM users do and they cite one study from 2021 that's about chatbot interactions. It's in an I Triple E outlet. It's like a really short, weird little engineering study. It's by some management IT people, it's 22 people in their study and they're talking about, oh, you know, people offload it's a weird use of cognitive load and

Ashley: 37:14

I mean, that's before large language models as we know them. I

Cat: 37:17

In no way does it tell us. What LLM users are like. So that's the kind of, you know, claim. And if, unless you're a huge nerd like me and you're always going, you're reading the paper that's being cited in the paper that's being cited in the other paper, you miss how we just start to push this garbage claims forward in research.

Ashley: 37:39

Yeah. Yeah, totally. I think, yeah, so, so clearly we're annoyed, right? And frustrated and angry and I kind of wanna articulate why that is, right? And I think for me it's because these are questions worth asking, right? I do really want to know whether LLMs. Uh, change the way my students think about problems, good or bad. I do want to know if the offloading of cognitive tasks changes my own brain and my own cognition and like welcomes dementia earlier or pushes it off, right? I really wanna know these things. We need to know these things as a society, but to pretend like this paper answers those questions is just completely wrong. Like, it, it is, it is irresponsible, in my opinion, to put out a paper that claims to address these really core, important questions. And it, it doesn't, it's sloppy. I was telling Cat if one of my students submitted this as like a final project, I don't know, it would get a pretty poor grade. Like, this is not an a paper. And they're using their prestige as a substitution for rigor, and it's not okay with me. And it, it bothers my deepest core.

Cat: 39:05

And they're, they're using the intimidation of pictures of brains and math to make you feel bad. And we have serious problems to solve in the world, you know, and some of these problems include things like how do we protect the brain health of children? By the way, way, the answer is stuff like social services and sleep, and, you know, solving

Ashley: 39:28

Eating well.

Cat: 39:29

Yeah. The answer is these huge things. It's, and there's this piece of it too for me, which is, it, it actually was very challenging for us to decide to talk about this. It's scary because a lot of people that I respect who otherwise are talking and thinking about human wellbeing jumped on this paper. But my thing is, if we allow neuro bullshit to be the way we answer questions. Tomorrow they're gonna apply this way of doing it to something that you love and care about. And they're gonna tell you, you don't get to do that because your brain sucks, or your brain's gonna melt out of your ears if you relax. Or if you, you know, um, maybe if you belong to a certain racial group or you know, persecuted group, like trans people who could measure differently in some measures, like if we allow pseudoscience to propagate, it can be weaponized. I, I will go there. It can be weaponized to the greatest extent pseudoscience kills. It has

Ashley: 40:35

hm, hmm

Cat: 40:36

You know, the way we talk about human brains has led us to, to consign people. To asylums.

Ashley: 40:45

Yeah.

Cat: 40:46

And so we, yeah, we take it really seriously.

Ashley: 40:50

Yeah. No, absolutely. And I think. Like you said, like we were kind of like trying to decide whether to do this paper or talk about it. And it was like, we have to because this, this is why this is important. It's actually deadly to not address pseudoscience. And I, and I kind of wanna articulate a point too that I think you said to me yesterday, Cat, which was like, you know, like these actually like LLMs are amazing tools for people who could benefit from them. And I think like slapping this idea on it that they're gonna cause your brain to melt out of your ears or whatever, is really damaging to those people who actually could benefit tremendously from using them. Like obviously we have complicated feelings about AI and LLMs in general, so I'm not here to like stand by them in total. But it is a technology with tremendous potential, especially for certain groups of people.

Cat: 41:45

When we say ai, what we are referencing is a huge amount of things. And I am not afraid of models. I've used models my whole life, you know, well, not my whole life, but as soon as I knew what math was, I started, I started thinking about how we use math and probabilities and, and I think we need to distinguish between the large commercial actors that are causing certain sociopolitical forces. You know, there's a political alignment that you and I are both, you know, we're deeply on the side of human wellbeing, you know, we're deeply afraid of, you know, the things that are politically happening. We're here to fight against that. But that's a different thing than saying, you know, what is happening to you, when you sit down and you get texts generated for you, and I've. Uh, take this point very seriously. Using a cognitive aid does not make someone bad or stupid or less worthy. Having, having a difference in having a cognitive impairment doesn't make you less worthy, and I am deeply troubled by the ableism in some of this conversation. I think

Ashley: 42:58

yeah.

Cat: 43:00

it's a very complicated conversation, but we cannot make that the bar, you know?

Ashley: 43:06

Totally.

Cat: 43:07

Something that was interesting to me about the framing of these results, the interpretation of them, and this is something where, gosh, it's such a good skill to learn to read a paper for what it says and then also read it thinking what part of this is the researchers interpretation that they are telling me, and I can make my own interpretation. So they are interpreting the hell out of these EEG results. And I felt very frustrated by that. Um, because I think there's a mix of both the things they observe about the EEG, but then their interpretation, their takeaways, they're like, this could lead to that are all mixed in there in the analysis section. An example that I found particularly confusing is. Trying to come away with a takeaway about this search engine group versus the LLM group. They say this stuff about how in the search engine group, maybe people's visual processing is going more, and maybe actually the LLM group has some positive themes, like the LLM group is doing more planning, something like that to try to find the actual section of the paper. Can I just say that scrolling through this PDF is also a barrier to understanding it? So there's really a lot of, even in this paper, there's a lot of mixed findings. There are moments where they say maybe there's something positive about using LLMs. It's interesting to me that none of that is getting picked up on social media.

Ashley: 44:40

Yeah, and I saw someone make a really good point actually, which is that there's kind of this claim in the paper, you know, the, the, the sentence in the abstract says something like, you know, the amount of activation was like highest for brain, only mid for search engine and lowest for LLM, right? And their interpretation of that is that people are using their brain in the brain only condition. But there's actually a separate argument that someone could make, which is to say that maybe the brain is being more efficient in the LLM condition. Like maybe actually less brain activity is better. And what. Is the reason we think less brain activity is worse. That's a framework that does not exist in this paper. They don't make an argument for it. They don't make a clear argument for why brain activity should be high or low and why that's good or bad. No.

Cat: 45:36

What's freaking annoying is they make both arguments. So a really enlightening section is on page 1 35, this section that's titled Neural Connectivity Patterns, where they try to sort of sum up their thing and they say, they talk out of both sides of their mouth. They say, wow, LLM group has the least extensive connectivity, and this is where they cite these really scary sounding percentages. Which by the way, I think that creating averages and percentage drops like this across highly idiographic effects is something we could probably spend a whole episode on. I mean, it's really, it's really, really easy to mess this kind of comparison up.

Ashley: 46:17

Yeah.

Cat: 46:17

On the one hand saying the LLM group shows the lowest connectivity they have these really scary sounding percentages. 55% reduced total DTF magnitude compared to the brain only group in low frequency semantic and monitoring networks. Do you think any human alive who is not. Ashley right now is reading that or is not a neuroscientist is able to read that and not feel scared. But then on the other hand, a couple paragraphs down, they say, Hmm, well, you know, um, maybe LLMs have a more automated scaffolded cognitive mode. I don't know what a cognitive mode is. Okay. And I have a PhD in psychology with reduced reliance on endogenous semantic construction or visual content evaluation. Essentially, if you're not looking at more pictures, your brain will look different. Okay. So they're talking out of both sides of their mouth constantly. They're using really undefined terms. They're saying things like, this is a cognitive mode, quote unquote, . You know, again, define it for me. I also don't know what cognitive debt means, which is a thing they say a bunch and don't define and sort of just a just. Propose this really bafflingly scary theory that you accrue this debt in your, you know, ability to solve problems. Well, that's a big claim. Okay, so let's talk about how that functions in the brain. I have lots of bones to pick with this. That's my, that's my showing how to read this section a little bit. How about

Ashley: 47:46

Yeah, I think in terms of like thinking about how to read the paper and like things that are red flags, like one thing for me that I think about is the process of doing science is not just collecting data, analyzing it and showing it to people, which is kind of what this artifact is. It's some data. We did some analysis of it, we're showing it to you. The process of science is. Actually developing an argument for something. And that includes, as you said, defining terms like cognitive debt, placing those terms and that argument in the context of other studies of a wealth of literature that's already been done. Creating a framework for what you expect to see or what you might see in the data, testing your data and your analysis against that framework, and then seeing how well it holds up, right? Like that's the process of science. This paper doesn't, like I, I don't, I don't really know what these authors think in terms of their scientific framework because it's not really laid out for me here. There aren't even really clear research questions in this paper. And that to me feels like we threw a bunch of stuff at the wall and we're looking to see what sticks, and we threw this analysis, which by the way is not actually that common in EEG research is a little bit of a niche analysis, but we

Cat: 49:06

Yeah,

Ashley: 49:07

it at the data.

Cat: 49:08

it? Why are

Ashley: 49:09

No, exactly. There's no exactly like there's in this scientific process, we make a argument for using particular analyses because we think that they'll answer particular questions. I don't know why they chose this analysis because, It's it's, they picked something that seemed fancy. They're throwing NLP at it. Why'd you throw NLP at it? Like, why are you doing this named entity recognition thing? Like the why of the paper, which is like the, probably the main comment I ever leave on my students' papers. Why, why, why, why why'd you do this? Why'd you choose this? Like, that is not evident to me here. And that is a major red flag.

Cat: 49:49

It reads like post hoc, oh, it reads like post hoc looking for effects, and the limitations section is bafflingly weak.

Ashley: 49:58

It's so short.

Cat: 49:59

it's so short. It hardly goes into all of the methodological things. You know, there's, there's massive obvious errors. There's typos, there's um, there's the signs and signals that we look for to ask. Has this person really thought through like this methodology, do they know how to use EEG? Do they understand the limitations of it? I don't have that confidence at all. They switch units of analysis constantly. So at, on one hand they're talking about the text on the, then, then suddenly they're analyzing between sessions. They're not clear about the hypotheses, the existing previous literature that justifies looking at it. In this way, we can all analyze data and I could make these little brain charts right now and maybe my research would actually get some shares. My preprints, which I can't get published in software research journals 'cause they don't wanna publish psychology, you know, would actually get some play.

Ashley: 50:53

I think this paper was written to create a big splash to get attention

Cat: 51:00

yeah.

Ashley: 51:01

to scare people about LLMs. I don't know.

Cat: 51:04

To get the shortest path from a weird document someone wrote to a time article, maybe, to prop up the pieces of hardware that are associated with this, this paper maybe, um, I don't know if there's a product in here. I mean, MIT Media Lab

Ashley: 51:21

there are multiple products in here. Um, at least one of them is directly associated with at least one of the authors. Um,

Cat: 51:34

One of the authors is like an angel investor.

Ashley: 51:38

Yeah. I always ask the question, what is this person getting for doing this research or this science communication? Right? Like, I essentially do not trust science communicators who. I don't know. Let's say get money from supplement companies or for selling a particular piece of hardware. It is a huge, huge red flag for me in terms of being able to believe their science and to believe their motivation is in the right place.

Cat: 52:12

I'm not gonna say we can read other people's minds unless we had an EEG cap

Ashley: 52:17

But only if we could do, wait, let's just order one and then we can read

Cat: 52:22

one and then we'll know the internal processing states of all these people and we'll be able to find the grift part of the brain

Ashley: 52:29

oh, there's definitely a grift part of the brain.