Interview: What Does the ‘I’ in AI Really Mean?
Is it possible that AI is — or might one day become — truly intelligent? We put this question to five experts working in a variety of fields, including linguistics, computer science, cognitive science, and psychology. Not surprisingly, they offered very different answers, though if there was one point of agreement, it was that the term “intelligence” is vague and poorly suited for the conversation.
“Intelligence is a heavily contested concept,” Benjamin K. Bergen, a professor of cognitive science at the University of California, San Diego, wrote in an email to Undark. “Psychologists disagree on what it is.” Nevertheless, these five experts helped to parse just what our machines are doing — and what, if anything, separates AI from humans and from the natural world.
Some of these entries are drawn from email exchanges while others came from Zoom interviews. In some cases, they have been edited for length and clarity.
Emily M. Bender
Professor of linguistics, University of Washington
Setting aside the undefined terms “AI” (a marketing term, not a coherent set of technologies) and “intelligent” (a eugenicist way to rank people), can we ever create artificial systems that engage in cognitive behaviors or experiences similar to people? Probably not, and definitely not through language modeling (like LLMs). The only evidence we have that LLMs can “think” comes from our ability to make sense of the synthetic text they extrude.
Linguistics — the study of how languages work and how we work with language — provides two crucial insights here: The first is that languages are systems of signs, which pair form (the spelling or pronunciation of a word) with meaning (what the word means in general and what it is being used to mean). Language models manipulate form, but without reference to meaning.
The second lesson concerns how we interpret language. Far from simply unpacking meaning that an author packed into the words, we instead keep in mind everything we believe about the author’s state of mind, our common ground with them, and their beliefs about their intended audience’s state of mind. Against that background, we ask ourselves: What must the author be trying to convey by using those words and in that way?
In order to interpret text, we have to imagine a mind behind the text. We do this instinctively and reflexively — we can’t help ourselves, making rigorous evaluation of these systems difficult. As we wrote in the Stochastic Parrots paper in 2021: “Our perception of natural language text, regardless of how it was generated, is mediated by our own linguistic competence and our predisposition to interpret communicative acts as conveying coherent meaning and intent, whether or not they do.”
If we don’t account for this, we will not be able to make wise decisions about this technology.
Melanie Mitchell
Computer scientist and professor, the Santa Fe Institute
There’s this notion in AI circles that there’s this easily defined split between what people will call “cognitive intelligence” versus “physical intelligence.” When these big companies, for instance, like DeepMind or OpenAI, define what they’re calling “artificial general intelligence,” they talk only about cognitive tasks, as opposed to physical tasks. So, ChatGPT is not going to come fix your roof.
But it’s not clear to me how you can so easily separate the so-called cognitive kinds of tasks from the more physical kinds of intelligence that our brains were actually evolved to enable us to have. Or the more social kinds of intelligence that we humans rely on so much.
Undark: Can you say more about why it’s hard to draw a distinction between cognitive and physical intelligence? If I ask AI, “how do I fix my dishwasher?” it will give me instructions. Then I go and do the physical act of fixing the dishwasher. That seems like a clean distinction.
But in interpreting those instructions and applying them to your dishwasher, you’re adding quite a bit of the intelligence in that. If it says, “Press the red button on the left side of the door,” you — human — are using a lot of your intelligence to figure out what that means. Maybe there’s two red buttons, but one of them is obviously the one you’re supposed to press.
Undark: My experience with AI is that it’s quite bad at social intelligence.
One notion of social intelligence is that we can coordinate with other people to get things done: So, I don’t know how to fix my car, but I know how to find someone to fix my car. I might ask my friends, “Who do you know who knows how to fix a car?” And then also figuring out, what’s the right way to explain the problem to another person? I think of that as social intelligence.
I don’t know why AI systems are so bad at that. It may be that they don’t have the right training. It may be that they don’t have the right kind of participation in the social world.
Anima Anandkumar
Professor of computing and mathematical Sciences at Caltech
Let me give my definition of intelligence: the ability to learn from data, and the ability to adapt to surroundings, or environment, based on that. You have now many AI systems that can do both.
You have AI that interacts with humans as chatbots and then adapts through that feedback — what we call reinforcement learning. These models can be made to align to certain goals. Could be doing more accurate mathematical proofs. Where it’s been most popular has been language models because we interact with those daily.
Language models mirror a lot of human intelligence, but I want to talk about nature’s intelligence. The ability to learn from microscopic scales — how atoms interact, how reactions happen, that enables us to create better drugs, better materials — to planetary scales and even cosmological scales: understanding how weather systems form. Is there a hurricane forming? How quickly will it make landfall? Can we understand the fine details of it to really get the predictions correct and do that early on? Because that has huge impact on saving human lives, creating evacuation policies early on.
We created the first AI-based weather model about five years ago that’s tens of thousands of times faster than what traditional forecasting could do, and now exceeds even the accuracy of traditional forecasts. So it’s faster, it’s more accurate, and we can create better predictions of extreme weather events and also assess risk. And now with changing climate, you could also make the systems predict longer-term climate, be able to really accurately incorporate the laws of physics, so that they can be accurate, even in newer conditions.
People focus so much on human intelligence. I want to talk about the intelligence that’s all around us — that’s invisible to us — but really, all of life is built upon it.
Alison Gopnik
Professor of psychology at UC Berkeley, author of popular books about babies and cognitive development
This question, and the phrase “artificial general intelligence” assume that there is a single thing called “intelligence” and that creatures, artificial or natural, have more or less of it. This is a common kind of explanation in folk theories and in early scientific ones. But as science progresses, explanations become more complex and cognitive science describes many varied, complex, cognitive capacities that serve different, and often conflicting, functions — many different “intelligences.” The intelligences that allow human children to learn so much are still far beyond current models.
The remarkable success of the current large models depends on the ability to detect statistical patterns in the text and pictures that humans have already produced at scale and produce new text and pictures based on those patterns. This allows human users to take advantage of the information previous humans have produced — these models are a technology for cultural transmission. But this is fundamentally different from the capacities that allow people to produce that information in the first place.
Another kind of “intelligence” is the “exploit” ability to take actions to achieve particular goals. This kind of intelligence is captured by reinforcement learning. AI systems use reinforcement learning to master skills like Atari games or coding, and it plays a crucial role in “post-training” the large models.
Cultural transmission and reinforcement learning play a role in human intelligence. But very young children actively explore a changing world and build new models of it — “explore” capacities that are at odds with “exploit” intelligence. They also rely on a very different kind of intelligence — the intelligence of care. Children can explore because other people take on the cognitively challenging task of caring for them. These capacities go beyond current large models, although different types of artificial intelligences might begin to approach them in the future.
Benjamin K. Bergen
Professor of cognitive science at UC San Diego
For most definitions of intelligence, modern AIs already display behaviors consistent with at least some degree of intelligence. They react to their environment, learn from data, and generalize beyond their training materials. By some measures, they already appear to reason logically and display sensitivity to the context and feelings of the humans they’re interacting with.
But whether these behaviors make them “truly intelligent” … depends, in part, on what you think behavioral evidence can actually tell you. Let’s say you’re trying to decide whether a horse is truly intelligent. When you pose arithmetic problems, the horse stomps its hoof the correct number of times to indicate the answer. Is the horse intelligent? Well, if the horse is able to understand problems articulated in human language, perform mathematical operations, and respond appropriately, you might reasonably conclude that it has some degree of intelligence.
But if you learn instead that the horse is actually just taking subtle cues from its trainer, who knowingly or not makes subtle movements when the horse has reached the correct number of stomps, this horse is displaying a different kind of intelligence. It has learned to stomp in a particular kind of situation, to attend to the trainer for subtle cues, and to stop appropriately. This is an example of learning from experience, generalizing to new situations, and perhaps sensitivity to the emotions of others in a social context.
But the lesson from this case is that intelligent-seeming behavior can be produced by many different kinds of underlying mechanisms, which may be different in the kinds or degree of intelligence that they engage. Similarly, AIs can answer questions that seem to require profound reasoning but which upon further inspection are actually spelled out in their entirety in the voluminous training data the model has consumed.