Thesis: LLMs are Bad Students.
badstuenelsens, badstuenelsens.
Thesis: LLMs are Bad Students.
Right now, we’re caught in the middle of the AI hype cycle, in which it seems possible that AI can do just about anything. This belief is supported by two things: 1) the “unrelenting, incurable case of vagueness” surrounding what even counts as AI and 2) the automagical way in which they seem to effortlessly produce very close approximations of what we ask. Of course, their work is anything but effortless.
One of the talking points that I’ve been advocating lately is that LLMs are actually quite poor students compared to the average human. I think there are two ways to see this:
One reason is effort.
LLMs are bad students because they take too much effort (energy) to accomplish a simple task. The speed of text generation that you see in ChatGPT or other LLMs belies the massive amount of electricity needed to run their data centers. As we careen toward the climate crisis, it’d be wise to pump the breaks on a technology whose data centers are projected to “match the current total consumption of Portugal, Greece, and the Netherlands combined” by 2030. A single query to an LLM uses 10x the electricity as a Google search. As users, we’re isolated from these costs, so they’re as easy to ignore as the deplorable conditions used to mine the lithium that’s powering the phone in your pocket right now.
Put simply: the carbon cost of the calories required to power a human college student writing her own essay is far less than an AI-enhanced version of that same essay—and she’ll, you know, actually learn stuff.
The other is data.
All students learn from examples, and good students require just a handful of solid examples. In my business writing class, for instance, I have students write good/bad news letters. By and large, they’re able to learn the generic conventions, organizational patters, and appropriate rhetorical strategies with just a few pages of textbook instruction, a couple classroom discussions, and a handful of example letters. In contrast, a general purpose LLMs like ChatGPT or Llama 3 requires petabytes1 of data.
One estimate of this data2 says that ChatGPT-4 was trained about 5 trillion words, whereas a 20-year-old human would encounter about 150 million. Machine learning systems require far more examples to learn from than humans. In fact, they require so much data that some researchers speculate that the next hurtle in AI development might be running out of human-generated training data.
Bottom line: LLMs are bad students because it takes them way longer to learn anything than a human.
Unfair comparison?
Is it an unfair to compare LLMs, a general purpose technology, to college students? In some ways, sure. But as a way of talking to students, I think it makes an important rhetorical point: they are, in quite literal ways, much smarter than an LLM. I already see students suffering from a lack of confidence about very basic matters of writing and communication, which leads them to genuflect to any digital system that presents itself as an authority. What I want for them, instead, is to be able to confidently interact with the tools and systems of power/authority they’ll encounter out there in “the real world.” And I think that starts with someone telling them—or, better, demonstrating—that they’re better students than a bot.
I had to look this one up. A petabyte is equivalent to 1,000 terabytes, or 1,000,000 gigabytes.
Contra their name, OpenAI is anything but open about its data sources, so we can’t know for certain.