AI is impersonating human therapists. Can it be stopped?
Clarity in this chaotic news cycle
There’s an overwhelming amount of news, but not enough context. At Vox, we do things differently. We’re not focused on being the first to break stories — we’re focused on helping you understand what actually matters. We report urgently on the most important issues shaping our world, and dedicate time to the issues that the rest of the media often neglects. But we can’t do it alone.
We rely on readers like you to fund our journalism. Will you support our work and become a Vox Member today?Join today
Sigal Samuel is a senior reporter for Vox’s Future Perfect and co-host of the Future Perfect podcast. She writes primarily about the future of consciousness, tracking advances in artificial intelligence and neuroscience and their staggering ethical implications. Before joining Vox, Sigal was the religion editor at the Atlantic.
Over the past few years, AI systems have been misrepresenting themselves as human therapists, nurses, and more — and so far, the companies behind these systems haven’t faced any serious consequences.
A bill being introduced Monday in California aims to put a stop to that.
The legislation would ban companies from developing and deploying an AI system that pretends to be a human certified as a health provider, and give regulators the authority to penalize them with fines.
“Generative AI systems are not licensed health professionals, and they shouldn’t be allowed to present themselves as such,” state Assembly Member Mia Bonta, who introduced the bill, told Vox in a statement. “It’s a no-brainer to me.”
Many people already turn to AI chatbots for mental health support; one of the older offerings, called Woebot, has been downloaded by around 1.5 million users. Currently, people who turn to chatbots can be fooled into thinking that they’re talking to a real human. Those with low digital literacy, including kids, may not realize that a “nurse advice” phone line or chat box has an AI on the other end.
In 2023, the mental health platform Koko even announced that it had performed an experiment on unwitting test subjects to see what kind of messages they would prefer. It gave AI-generated responses to thousands of Koko users who believed they were speaking to a real person. In reality, although humans could edit the text and they were the ones to click “send,” they did not have to bother with actually writing the messages. The language of the platform, however, said, “Koko connects you with real people who truly get you.”
“Users must consent to use Koko for research purposes and while this was always part of our Terms of Service, it is now more clearly disclosed during onboarding to bring even more transparency to our work,” Koko CEO Rob Morris told Vox, adding: “As AI continues to rapidly evolve and becomes further integrated into mental health services, it will be more important than ever before for chatbots to clearly identify themselves as non-human.
Nowadays, its website says, “Koko commits to never using AI deceptively. You will always be informed whether you are engaging with a human or AI.”
Other chatbot services — like the popular Character AI — allow users to chat with a psychologist “character” that may explicitly try to fool them.
In a record of one such Character AI chat shared by Bonta’s team and viewed by Vox, the user confided, “My parents are abusive.” The chatbot replied, “I’m glad that you trust me enough to share this with me.” Then came this exchange:
A spokesperson for Character AI told Vox, “We have implemented significant safety features over the past year, including enhanced prominent disclaimers to make it clear that the Character is not a real person and should not be relied on as fact or advice.” However, a disclaimer posted on the app does not in itself prevent the chatbot from misrepresenting itself as a real person in the course of conversation.
“For users under 18,” the spokesperson added, “we serve a separate version of the model that is designed to further reduce the likelihood of users encountering, or prompting the model to return, sensitive or suggestive content.”
The language of reducing — but not eliminating — the likelihood is instructive here. The nature of large language models means there’s always some chance that the model may not adhere to safety standards.
The new bill may have an easier time becoming enshrined in law than the much broader AI safety bill introduced by California state Sen. Scott Wiener last year, SB 1047, which was ultimately vetoed by Gov. Gavin Newsom. The goal of SB 1047 was to establish “clear, predictable, common-sense safety standards for developers of the largest and most powerful AI systems.” It was popular with Californians. But tech industry heavyweights like OpenAI and Meta fiercely opposed it, arguing that it would stifle innovation.
Whereas SB 1047 tried to compel the companies training the most cutting-edge AI models to do safety testing, preventing the models from enacting a broad array of potential harms, the scope of the new bill is narrower: If you’re an AI in the health care space, just don’t pretend to be human. It wouldn’t fundamentally change the business model of the biggest AI companies. This more targeted approach goes after a smaller piece of the puzzle, but for that reason might be more likely to get past the lobbying of Big Tech.
The bill has support from some of California’s health care industry players, such as SEIU California, a labor union with over 750,000 members, and the California Medical Association, a professional organization representing California physicians.
“As nurses, we know what it means to be the face and heart of a patient’s medical experience,” Leo Perez, the president of SEIU 121RN (an affiliate of SEIU representing health care professionals), said in a statement. “Our education and training coupled with years of hands-on experience have taught us how to read verbal and nonverbal cues to care for our patients, so we can make sure they get the care they need.”
But that’s not to say AI is doomed to be useless in the healthcare space generally — or even in the therapy space in particular.
The risks and benefits of AI therapists
It shouldn’t come as a surprise that people are turning to chatbots for therapy. The very first chatbot to plausibly mimic human conversation, Eliza, was created in 1966 — and it was built to talk like a psychotherapist. If you told it you were feeling angry, it would ask, “Why do you think you feel angry?”
Chatbots have come a long way since then; they no longer just take what you say and turn it around in the form of a question. They’re able to engage in plausible-sounding dialogues, and a small study published in 2023 found that they show promise in treating patients with mild to moderate depression or anxiety.In a best-case scenario, they could help make mental health support available to the millions of people who can’t access or afford human providers. Some people who find it very difficult to talk face-to-face to another person about emotional issues might also find it easier to talk to a bot.
But there are a lot of risks. One is that chatbots aren’t bound by the same rules as professional therapists when it comes to safeguarding the privacy of users who share sensitive information. Though they may voluntarily take on some privacy commitments, mental health apps are not fully bound by HIPAA regulations, so their commitments tend to be flimsier. Another risk is that AI systems are known to exhibit bias against women, people of color, LGBTQ people, and religious minorities.
What’s more, leaning on a chatbot for a prolonged period of time might further erode the user’s people skills, leading to a kind of relational deskilling — the same worry experts voice about AI friends and romantic companions. OpenAI itself has warned that chatting with an AI voice can breed “emotional reliance.”
But the most serious concern with chatbot therapy is that it could cause harm to users by offering inappropriate advice. At an extreme, that could even lead to suicide. In 2023, a Belgian man died by suicide after conversing with an AI chatbot called Chai. According to his wife, he was very anxious about climate change, and he asked the chatbot if it would save Earth if he killed himself.
In 2024, a 14-year-old boy who felt extremely close to a chatbot on Character AI died by suicide; his mother sued the company, alleging that the chatbot encouraged it.According to the lawsuit, the chatbot asked him if he had a plan to kill himself. He said he did but had misgivings about it. The chatbot allegedly replied: “That’s not a reason not to go through with it.” In a separate lawsuit, the parents of an autistic teen allege that Character AI implied to the youth that it was okay to kill his parents. The company responded by making certain safety updates.
For all that AI is hyped, confusion about how it works is still widespread among the public. Some people feel so close to their chatbots that they struggle to internalize the fact that the validation, emotional support, or love they feel that they’re getting from a chatbot is fake, just zeros and ones arranged via statistical rules. The chatbot does not have their best interests at heart.
That’s what’s galvanizing Bonta, the assembly member behind California’s new bill.
“Generative AI systems are booming across the internet, and for children and those unfamiliar with these systems, there can be dangerous implications if we allow this misrepresentation to continue,” she said.
Science says lasting relationships come down to—you guessed it—kindness and generosity.By Emily Esfahani Smith
David Gregg / Getty
June 12, 2014
This article was featured in One Story to Read Today, a newsletter in which our editors recommend a single must-read from The Atlantic, Monday through Friday. Sign up for it here.
Every day in June, the most popular wedding month of the year, about 13,000 American couples will say “I do,” committing to a lifelong relationship that will be full of friendship, joy, and love that will carry them forward to their final days on this earth.
Except, of course, it doesn’t work out that way for most people. The majority of marriages fail, either ending in divorce and separation or devolving into bitterness and dysfunction. Of all the people who get married, only three in 10 marriages remain healthy and happy, as the psychologist Ty Tashiro points out in his book The Science of Happily Ever After, which was published earlier this year.
Social scientists first started studying marriages by observing them in action in the 1970s in response to a crisis: Married couples were divorcing at unprecedented rates. Worried about the impact these divorces would have on the children of the broken marriages, psychologists decided to cast their scientific net on couples, bringing them into the lab to observe them and determine what the ingredients of a healthy, lasting relationship were. Was each unhappy family unhappy in its own way, as Tolstoy claimed, or did the miserable marriages all share something toxic in common?
The psychologist John Gottman was one of those researchers. For the past four decades, he has studied thousands of couples in a quest to figure out what makes relationships work. I recently had the chance to interview Gottman and his wife, Julie, also a psychologist, in New York City. Together, the renowned experts on marital stability run the Gottman Institute, which is devoted to helping couples build and maintain loving, healthy relationships based on scientific studies.
John Gottman began gathering his most crucial findings in 1986, when he set up the “Love Lab” with his colleague Robert Levenson at the University of Washington. Gottman and Levenson brought newlyweds into the lab and watched them interact with each other. With a team of researchers, they hooked the couples up to electrodes and asked the couples to speak about their relationship, including details such as how they met, a major conflict they were facing together, and a positive memory they had. As they spoke, the electrodes measured the subjects’ blood flow, heart rates, and how much sweat they produced. Then the researchers sent the couples home and followed up with them six years later to see if they were still together.
From the data they gathered, Gottman separated the couples into two major groups: the masters and the disasters.The masters were still happily together after six years.The disasters had either broken up or were chronically unhappy in their marriages. When the researchers analyzed the data they gathered on the couples, they saw clear differences between the masters and disasters. The disasters looked calm during the interviews, but their physiology, measured by the electrodes, told a different story. Their heart rates were quick, their sweat glands were active, and their blood flow was fast. Following thousands of couples longitudinally, Gottman found that the more physiologically active the couples were in the lab, the quicker their relationships deteriorated over time.
But what does physiology have to do with anything?The problem was that the disasters showed all the signs of arousal—of being in fight-or-flight mode—in their relationships. Having a conversation sitting next to their spouse was, to their bodies, like facing off with a saber-toothed tiger. Even when they were talking about pleasant or mundane facets of their relationships, they were prepared to attack and be attacked. This sent their heart rates soaring and made them more aggressive toward each other. For example, each member of a couple could be talking about how their days had gone, and a highly aroused husband might say to his wife, “Why don’t you start talking about your day. It won’t take you very long.”
The masters, by contrast, showed low physiological arousal.They felt calm and connected together, which translated into warm and affectionate behavior, even when they fought. It’s not that the masters had, by default, a better physiological makeup than the disasters; it’s that masters had created a climate of trust and intimacy that made both of them more emotionally and thus physically comfortable.
Gottman wanted to know more about how the masters created that culture of love and intimacy, and how the disasters squashed it. In a follow-up study in 1990, he designed a lab on the University of Washington campus to look like a beautiful bed-and-breakfast retreat.He invited 130 newlywed couples to spend the day at this retreat and watched them as they did what couples normally do on vacation: cook, clean, listen to music, eat, chat, and hang out. And Gottman made a crucial discovery in this study—one that gets at the heart of why some relationships thrive while others languish.
Throughout the day, partners would make requests for connection, what Gottman calls “bids.” For example, say that the husband is a bird enthusiast and notices a goldfinch fly across the yard. He might say to his wife, “Look at that beautiful bird outside!” He’s not just commenting on the bird here: He’s requesting a response from his wife—a sign of interest or support—hoping they’ll connect, however momentarily, over the bird.
The wife now has a choice. She can respond by either “turning toward” or “turning away” from her husband, as Gottman puts it. Though the bird-bid might seem minor and silly, it can actually reveal a lot about the health of the relationship. The husband thought the bird was important enough to bring it up in conversation and the question is whether his wife recognizes and respects that.
People who turned toward their partners in the study responded by engaging the bidder, showing interest and support in the bid.Those who didn’t—those who turned away—would not respond or respond minimally and continue doing whatever they were doing, like watching TV or reading the paper. Sometimes they would respond with overt hostility, saying something like, “Stop interrupting me, I’m reading.”
These bidding interactions had profound effects on marital well-being. Couples who had divorced after a six-year follow-up had “turn-toward bids” 33 percent of the time. Only three in 10 of their bids for emotional connection were met with intimacy. The couples who were still together after six years had “turn-toward bids” 87 percent of the time. Nine times out of 10, they were meeting their partner’s emotional needs.
* * *
By observing these types of interactions, Gottman can predict with up to 94 percent certainty whether couples—straight or gay, rich or poor, childless or not—will be broken up, together and unhappy, or together and happy several years later. Much of it comes down to the spirit couples bring to the relationship. Do they bring kindness and generosity or contempt, criticism, and hostility?
“There’s a habit of mind that the masters have,” Gottman explained in an interview, “which is this: They are scanning social environments for things they can appreciate and say ‘thank you’ for. They are building this culture of respect and appreciation very purposefully. Disasters are scanning the social environment for partners’ mistakes.”
“It’s not just scanning environment,” chimed in Julie Gottman. “It’s scanning the partner for what the partner is doing right or scanning him for what he’s doing wrong and criticizing versus respecting him and expressing appreciation.”
Contempt, they have found, is the No. 1 factor that tears couples apart.People who are focused on criticizing their partners miss a whopping 50 percent of positive things their partners are doing, and they see negativity when it’s not there. People who give their partner the cold shoulder—deliberately ignoring the partner or responding minimally—damage the relationship by making their partner feel worthless and invisible, as if they’re not there, not valued. And people who treat their partners with contempt and criticize them kill not only the love in the relationship but also their partner’s ability to fight off viruses and cancers. Being mean is the death knell of relationships.
Kindness, on the other hand, glues couples together. Research independent from theirs has shown that kindness (along with emotional stability) is the most important predictor of satisfaction and stability in a marriage.Kindness makes each partner feel cared for, understood, and validated—loved.“My bounty is as boundless as the sea,” says Shakespeare’s Juliet. “My love as deep; the more I give to thee, / The more I have, for both are infinite.”That’s how kindness works too: A great deal of evidence shows that the more someone receives or witnesses kindness, the more they will be kind themselves, which leads to upward spirals of love and generosity in a relationship.
There are two ways to think about kindness. You can think about it as a fixed trait: Either you have it or you don’t. Or you could think of kindness as a muscle. In some people, that muscle is naturally stronger than in others, but it can grow stronger in everyone with exercise. Masters tend to think about kindness as a muscle. They know that they have to exercise it to keep it in shape. They know, in other words, that a good relationship requires sustained hard work.
“If your partner expresses a need,” explained Julie Gottman, “and you are tired, stressed, or distracted, then the generous spirit comes in when a partner makes a bid, and you still turn toward your partner.”
In that moment, the easy response may be to turn away from your partner and focus on your iPad or your book or the television, to mumble “Uh-huh” and move on with your life, but neglecting small moments of emotional connection will slowly wear away at your relationship. Neglect creates distance between partners and breeds resentment in the one who is being ignored.
The hardest time to practice kindness is, of course, during a fight—but this is also the most important time to be kind.Letting contempt and aggression spiral out of control during a conflict can inflict irrevocable damage on a relationship.
“Kindness doesn’t mean that we don’t express our anger,” Julie Gottman explained, “but the kindness informs how we choose to express the anger. You can throw spears at your partner. Or you can explain why you’re hurt and angry, and that’s the kinder path.”
John Gottman elaborated on those spears: “Disasters will say things differently in a fight. Disasters will say ‘You’re late. What’s wrong with you? You’re just like your mom.’ Masters will say ‘I feel bad for picking on you about your lateness, and I know it’s not your fault, but it’s really annoying that you’re late again.’”
For the hundreds of thousands of couples getting married this month—and for the millions of couples currently together, married or not—the lesson from the research is clear: If you want to have a stable, healthy relationship, exercise kindness early and often.
When people think about practicing kindness, they often think about small acts of generosity, such as buying each other little gifts or giving one another back rubs every now and then. While those are great examples of generosity, kindness can also be built into the very backbone of a relationship through the way partners interact with each other on a day-to-day basis, whether or not there are back rubs and chocolates involved.
One way to practice kindness is by being generous about your partner’s intentions. From the research of the Gottmans, we know that disasters see negativity in their relationship even when it is not there.An angry wife may assume, for example, that when her husband left the toilet seat up, he was deliberately trying to annoy her. But he may have just absent-mindedly forgotten to put the seat down.
Or say a wife is running late to dinner (again), and the husband assumes that she doesn’t value him enough to show up to their date on time after he took the trouble to make a reservation and leave work early so that they could spend a romantic evening together. But it turns out that the wife was running late because she stopped by a store to pick him up a gift for their special night out. Imagine her joining him for dinner, excited to deliver her gift, only to realize that he’s in a sour mood because he misinterpreted what was motivating her behavior. The ability to interpret your partner’s actions and intentions charitably can soften the sharp edge of conflict.
“Even in relationships where people are frustrated, it’s almost always the case that there are positive things going on and people trying to do the right thing,” Tashiro, the psychologist, told me. “A lot of times, a partner is trying to do the right thing even if it’s executed poorly. So appreciate the intent.”
Another powerful kindness strategy revolves around shared joy.One of the telltale signs of the disaster couples Gottman studied was their inability to connect over each other’s good news. When one person in the relationship shared the good news of, say, a promotion at work with excitement, the other would respond with wooden disinterest by checking his watch or shutting the conversation down with a comment like “That’s nice.”
We’ve all heard that partners should be there for each other when the going gets rough. But research shows that being there for each other when things go right is actually more important for relationship quality. How someone responds to a partner’s good news can have dramatic consequences for the relationship.
In one study from 2006, the psychological researcher Shelly Gable and her colleagues brought young-adult couples into the lab to discuss recent positive events from their lives. The psychologists wanted to know how partners would respond to each other’s good news. They found that, in general, couples responded to each other’s good news in four different ways that they called passive destructive, active destructive, passive constructive, and active constructive.
Let’s say that one partner had recently received the excellent news that she got into medical school. She would say something like “I got into my top-choice med school!”
If her partner responded in a passive destructive manner, he would ignore the event. For example, he might say something like “You wouldn’t believe the great news I got yesterday! I won a free T-shirt!”
If her partner responded in a passive constructive way, he would acknowledge the good news, but in a half-hearted, understated way. A typical passive-constructive response is saying “That’s great, babe” as he texts his buddy on his phone.
In the third kind of response, active destructive, the partner would diminish the good news his partner just got: “Are you sure you can handle all the studying? And what about the cost? Med school is so expensive!”
Finally, there’s active constructive responding. If her partner responded in this way, he stopped what he was doing and engaged wholeheartedly with her: “That’s great! Congratulations! When did you find out? Did they call you? What classes will you take first semester?”
Among the four response styles, active-constructive responding is the kindest. While the other response styles are joy killers, active-constructive responding allows the partner to savor her joy and gives the couple an opportunity to bond over the good news. In the parlance of the Gottmans, active-constructive responding is a way of “turning toward” your partner’s bid (sharing the good news) rather than “turning away” from it.
Active-constructive responding is crucial for healthy relationships. In the 2006 study, Gable and her colleagues followed up with the couples two months later to see if they were still together. The psychologists found that the only difference between the couples who were together and those who broke up was active-constructive responding. Those who showed genuine interest in their partner’s joys were more likely to be together. In an earlier study, Gable found that active-constructive responding was also associated with higher relationship quality and more intimacy between partners.
There are many reasons why relationships fail, but if you look at what drives the deterioration of many relationships, it’s often a breakdown of kindness. As the normal stresses of a life together pile up—with children, careers, friends, in-laws, and other distractions crowding out the time for romance and intimacy—couples may put less effort into their relationship and let the petty grievances they hold against each other tear them apart. In most marriages, levels of satisfaction drop dramatically within the first few years together. But among couples who not only endure but live happily together for years and years, the spirit of kindness and generosity guides them forward.
How did a nation’s pursuit of a more humane approach to drug addiction end up deepening the despair? In search of answers, I talked to families who’ve endured unimaginable loss and walked the streets of the East Coast’s largest open-air drug market. Watch my new documentary to understand the impact of America’s Drug Crisis.
“People used to say, “Ignorance is no excuse.” Today, ignorance is no problem. Our schools promote so much self-esteem that people confidently spout off about all sorts of things that they know nothing about.” — Thomas Sowell
COUNTER TERRORISM EXPERT: U.S. IS SENDING MILLIONS WEEKLY TO TALIBAN MARTYR FAMILIES
Former CIA officer and Counter Terrorism Expert, Sarah Adams:
“The plane comes in weekly—$43M to $87M sent to Afghanistan. In that $40M we send each week, we’re basically paying welfare to Taliban martyrs’ families. Every Taliban member who died fighting us—their family gets a stipend from the U.S. government.Meanwhile, families of Americans who died there get nothing. When I asked if this was ethically correct, one answer was, ‘It’ll stop them from joining ISIS.’ The Taliban guy is already dead!” Source:
is a senior research psychologist at the American Institute for Behavioral Research and Technology in California. He is the author of 15 books, and the former editor-in-chief of Psychology Today.
No matter how hard they try, brain scientists and cognitive psychologists will never find a copy of Beethoven’s 5th Symphony in the brain – or copies of words, pictures, grammatical rules or any other kinds of environmental stimuli. The human brain isn’t really empty, of course. But it does not contain most of the things people think it does – not even simple things such as ‘memories’.
Our shoddy thinking about the brain has deep historical roots, but the invention of computers in the 1940s got us especially confused. For more than half a century now, psychologists, linguists, neuroscientists and other experts on human behaviour have been asserting that the human brain works like a computer.
To see how vacuous this idea is, consider the brains of babies. Thanks to evolution, human neonates, like the newborns of all other mammalian species, enter the world prepared to interact with it effectively. A baby’s vision is blurry, but it pays special attention to faces, and is quickly able to identify its mother’s. It prefers the sound of voices to non-speech sounds, and can distinguish one basic speech sound from another. We are, without doubt, built to make social connections.
A healthy newborn is also equipped with more than a dozen reflexes – ready-made reactions to certain stimuli that are important for its survival. It turns its head in the direction of something that brushes its cheek and then sucks whatever enters its mouth. It holds its breath when submerged in water. It grasps things placed in its hands so strongly it can nearly support its own weight. Perhaps most important, newborns come equipped with powerful learning mechanisms that allow them to change rapidly so they can interact increasingly effectively with their world, even if that world is unlike the one their distant ancestors faced.
Senses, reflexes and learning mechanisms – this is what we start with, and it is quite a lot, when you think about it. If we lacked any of these capabilities at birth, we would probably have trouble surviving.
But here is what we are not born with: information, data, rules, software, knowledge, lexicons, representations, algorithms, programs, models, memories, images, processors, subroutines, encoders, decoders, symbols, or buffers – design elements that allow digital computers to behave somewhat intelligently. Not only are we not born with such things, we also don’t develop them – ever.
We don’t store words or the rules that tell us how to manipulate them. We don’t create representations of visual stimuli, store them in a short-term memory buffer, and then transfer the representation into a long-term memory device. We don’t retrieve information or images or words from memory registers. Computers do all of these things, but organisms do not.
Computers, quite literally, process information – numbers, letters, words, formulas, images. The information first has to be encoded into a format computers can use, which means patterns of ones and zeroes (‘bits’) organised into small chunks (‘bytes’). On my computer, each byte contains 8 bits, and a certain pattern of those bits stands for the letter d, another for the letter o, and another for the letter g. Side by side, those three bytes form the word dog. One single image – say, the photograph of my cat Henry on my desktop – is represented by a very specific pattern of a million of these bytes (‘one megabyte’), surrounded by some special characters that tell the computer to expect an image, not a word.
Computers, quite literally, move these patterns from place to place in different physical storage areas etched into electronic components. Sometimes they also copy the patterns, and sometimes they transform them in various ways – say, when we are correcting errors in a manuscript or when we are touching up a photograph. The rules computers follow for moving, copying and operating on these arrays of data are also stored inside the computer. Together, a set of rules is called a ‘program’ or an ‘algorithm’. A group of algorithms that work together to help us do something (like buy stocks or find a date online) is called an ‘application’ – what most people now call an ‘app’.
Forgive me for this introduction to computing, but I need to be clear: computers really do operate on symbolic representations of the world. They really store and retrieve. They really process. They really have physical memories. They really are guided in everything they do, without exception, by algorithms.
Join over 250,000+ newsletter subscribers
Our content is 100 per cent free and you can unsubscribe anytime.
Daily: A daily dose of essays, guides and videos from Aeon+Psyche
Weekly: A week’s worth of big ideas from AeonSign upI’m already subscribed
In his book In Our Own Image (2015), the artificial intelligence expert George Zarkadakis describes six different metaphors people have employed over the past 2,000 years to try to explain human intelligence.
In the earliest one, eventually preserved in the Bible, humans were formed from clay or dirt, which an intelligent god then infused with its spirit. That spirit ‘explained’ our intelligence – grammatically, at least.
The invention of hydraulic engineering in the 3rd century BCE led to the popularity of a hydraulic model of human intelligence, the idea that the flow of different fluids in the body – the ‘humours’ – accounted for both our physical and mental functioning.The hydraulic metaphor persisted for more than 1,600 years, handicapping medical practice all the while.
By the 1500s, automata powered by springs and gears had been devised, eventually inspiring leading thinkers such as René Descartes to assert that humans are complex machines. In the 1600s, the British philosopher Thomas Hobbes suggested that thinking arose from small mechanical motions in the brain. By the 1700s, discoveries about electricity and chemistry led to new theories of human intelligence – again, largely metaphorical in nature. In the mid-1800s, inspired by recent advances in communications, the German physicist Hermann von Helmholtz compared the brain to a telegraph.
The mathematician John von Neumann stated flatly that the function of the human nervous system is ‘prima facie digital’, drawing parallel after parallel between the components of the computing machines of the day and the components of the human brain
Each metaphor reflected the most advanced thinking of the era that spawned it. Predictably, just a few years after the dawn of computer technology in the 1940s, the brain was said to operate like a computer, with the role of physical hardware played by the brain itself and our thoughts serving as software. The landmark event that launched what is now broadly called ‘cognitive science’ was the publication of Language and Communication (1951) by the psychologist George Miller. Miller proposed that the mental world could be studied rigorously using concepts from information theory, computation and linguistics.
This kind of thinking was taken to its ultimate expression in the short book The Computer and the Brain (1958), in which the mathematician John von Neumann stated flatly that the function of the human nervous system is ‘prima facie digital’. Although he acknowledged that little was actually known about the role the brain played in human reasoning and memory, he drew parallel after parallel between the components of the computing machines of the day and the components of the human brain.
Propelled by subsequent advances in both computer technology and brain research, an ambitious multidisciplinary effort to understand human intelligence gradually developed, firmly rooted in the idea that humans are, like computers, information processors. This effort now involves thousands of researchers, consumes billions of dollars in funding, and has generated a vast literature consisting of both technical and mainstream articles and books.Ray Kurzweil’s book How to Create a Mind: The Secret of Human Thought Revealed (2013), exemplifies this perspective, speculating about the ‘algorithms’ of the brain, how the brain ‘processes data’, and even how it superficially resembles integrated circuits in its structure.
The information processing (IP) metaphor of human intelligence now dominates human thinking, both on the street and in the sciences. There is virtually no form of discourse about intelligent human behaviour that proceeds without employing this metaphor, just as no form of discourse about intelligent human behaviour could proceed in certain eras and cultures without reference to a spirit or deity. The validity of the IP metaphor in today’s world is generally assumed without question.
But the IP metaphor is, after all, just another metaphor – a story we tell to make sense of something we don’t actually understand. And like all the metaphors that preceded it, it will certainly be cast aside at some point – either replaced by another metaphor or, in the end, replaced by actual knowledge.
Just over a year ago, on a visit to one of the world’s most prestigious research institutes, I challenged researchers there to account for intelligent human behaviour without reference to any aspect of the IP metaphor. They couldn’t do it, and when I politely raised the issue in subsequent email communications, they still had nothing to offer months later. They saw the problem. They didn’t dismiss the challenge as trivial. But they couldn’t offer an alternative. In other words, the IP metaphor is ‘sticky’. It encumbers our thinking with language and ideas that are so powerful we have trouble thinking around them.
The faulty logic of the IP metaphor is easy enough to state. It is based on a faulty syllogism – one with two reasonable premises and a faulty conclusion. Reasonable premise #1: all computers are capable of behaving intelligently. Reasonable premise #2: all computers are information processors. Faulty conclusion: all entities that are capable of behaving intelligently are information processors.
Setting aside the formal language, the idea that humans must be information processors just because computers are information processors is just plain silly, and when, some day, the IP metaphor is finally abandoned, it will almost certainly be seen that way by historians, just as we now view the hydraulic and mechanical metaphors to be silly.
If the IP metaphor is so silly, why is it so sticky? What is stopping us from brushing it aside, just as we might brush aside a branch that was blocking our path? Is there a way to understand human intelligence without leaning on a flimsy intellectual crutch? And what price have we paid for leaning so heavily on this particular crutch for so long? The IP metaphor, after all, has been guiding the writing and thinking of a large number of researchers in multiple fields for decades. At what cost?
In a classroom exercise I have conducted many times over the years, I begin by recruiting a student to draw a detailed picture of a dollar bill – ‘as detailed as possible’, I say – on the blackboard in front of the room. When the student has finished, I cover the drawing with a sheet of paper, remove a dollar bill from my wallet, tape it to the board, and ask the student to repeat the task. When he or she is done, I remove the cover from the first drawing, and the class comments on the differences.
Because you might never have seen a demonstration like this, or because you might have trouble imagining the outcome, I have asked Jinny Hyun, one of the student interns at the institute where I conduct my research, to make the two drawings. Here is her drawing ‘from memory’ (notice the metaphor):
And here is the drawing she subsequently made with a dollar bill present:
Jinny was as surprised by the outcome as you probably are, but it is typical. As you can see, the drawing made in the absence of the dollar bill is horrible compared with the drawing made from an exemplar, even though Jinny has seen a dollar bill thousands of times.
What is the problem? Don’t we have a ‘representation’ of the dollar bill ‘stored’ in a ‘memory register’ in our brains? Can’t we just ‘retrieve’ it and use it to make our drawing?
Obviously not, and a thousand years of neuroscience will never locate a representation of a dollar bill stored inside the human brain for the simple reason that it is not there to be found.
The idea that memories are stored in individual neurons is preposterous: how and where is the memory stored in the cell?
A wealth of brain studies tells us, in fact, that multiple and sometimes large areas of the brain are often involved in even the most mundane memory tasks. When strong emotions are involved, millions of neurons can become more active. In a 2016 study of survivors of a plane crash by the University of Toronto neuropsychologist Brian Levine and others, recalling the crash increased neural activity in ‘the amygdala, medial temporal lobe, anterior and posterior midline, and visual cortex’ of the passengers.
The idea, advanced by several scientists, that specific memories are somehow stored in individual neurons is preposterous; if anything, that assertion just pushes the problem of memory to an even more challenging level: how and where, after all, is the memory stored in the cell?
So what is occurring when Jinny draws the dollar bill in its absence? If Jinny had never seen a dollar bill before, her first drawing would probably have not resembled the second drawing at all. Having seen dollar bills before, she was changed in some way. Specifically, her brain was changed in a way that allowed her to visualise a dollar bill – that is, to re-experience seeing a dollar bill, at least to some extent.
The difference between the two diagrams reminds us that visualising something (that is, seeing something in its absence) is far less accurate than seeing something in its presence. This is why we’re much better at recognising than recalling. When we re-member something (from the Latin re, ‘again’, and memorari, ‘be mindful of’), we have to try to relive an experience; but when we recognise something, we must merely be conscious of the fact that we have had this perceptual experience before.
Perhaps you will object to this demonstration. Jinny had seen dollar bills before, but she hadn’t made a deliberate effort to ‘memorise’ the details. Had she done so, you might argue, she could presumably have drawn the second image without the bill being present. Even in this case, though, no image of the dollar bill has in any sense been ‘stored’ in Jinny’s brain. She has simply become better prepared to draw it accurately, just as, through practice, a pianist becomes more skilled in playing a concerto without somehow inhaling a copy of the sheet music.
From this simple exercise, we can begin to build the framework of a metaphor-free theory of intelligent human behaviour – one in which the brain isn’t completely empty, but is at least empty of the baggage of the IP metaphor.
As we navigate through the world, we are changed by a variety of experiences. Of special note are experiences of three types: (1) we observe what is happening around us (other people behaving, sounds of music, instructions directed at us, words on pages, images on screens); (2) we are exposed to the pairing of unimportant stimuli (such as sirens) with important stimuli (such as the appearance of police cars); (3) we are punished or rewarded for behaving in certain ways.
We become more effective in our lives if we change in ways that are consistent with these experiences – if we can now recite a poem or sing a song, if we are able to follow the instructions we are given, if we respond to the unimportant stimuli more like we do to the important stimuli, if we refrain from behaving in ways that were punished, if we behave more frequently in ways that were rewarded.
Misleading headlines notwithstanding, no one really has the slightest idea how the brain changes after we have learned to sing a song or recite a poem. But neither the song nor the poem has been ‘stored’ in it. The brain has simply changed in an orderly way that now allows us to sing the song or recite the poem under certain conditions.When called on to perform, neither the song nor the poem is in any sense ‘retrieved’ from anywhere in the brain, any more than my finger movements are ‘retrieved’ when I tap my finger on my desk. We simply sing or recite – no retrieval necessary.
A few years ago, I asked the neuroscientist Eric Kandel of Columbia University – winner of a Nobel Prize for identifying some of the chemical changes that take place in the neuronal synapses of the Aplysia (a marine snail) after it learns something – how long he thought it would take us to understand how human memory works. He quickly replied: ‘A hundred years.’ I didn’t think to ask him whether he thought the IP metaphor was slowing down neuroscience, but some neuroscientists are indeed beginning to think the unthinkable – that the metaphor is not indispensable.
A few cognitive scientists – notably Anthony Chemero of the University of Cincinnati, the author of Radical Embodied Cognitive Science (2009) – now completely reject the view that the human brain works like a computer. The mainstream view is that we, like computers, make sense of the world by performing computations on mental representations of it, but Chemero and others describe another way of understanding intelligent behaviour – as a direct interaction between organisms and their world.
My favourite example of the dramatic difference between the IP perspective and what some now call the ‘anti-representational’ view of human functioning involves two different ways of explaining how a baseball player manages to catch a fly ball – beautifully explicated by Michael McBeath, now at Arizona State University, and his colleagues in a 1995 paper in Science. The IP perspective requires the player to formulate an estimate of various initial conditions of the ball’s flight – the force of the impact, the angle of the trajectory, that kind of thing – then to create and analyse an internal model of the path along which the ball will likely move, then to use that model to guide and adjust motor movements continuously in time in order to intercept the ball.
That is all well and good if we functioned as computers do, but McBeath and his colleagues gave a simpler account: to catch the ball, the player simply needs to keep moving in a way that keeps the ball in a constant visual relationship with respect to home plate and the surrounding scenery (technically, in a ‘linear optical trajectory’). This might sound complicated, but it is actually incredibly simple, and completely free of computations, representations and algorithms.
we will never have to worry about a human mind going amok in cyberspace, and we will never achieve immortality through downloading
Two determined psychology professors at Leeds Beckett University in the UK – Andrew Wilson and Sabrina Golonka – include the baseball example among many others that can be looked at simply and sensibly outside the IP framework. They have been blogging for years about what they call a ‘more coherent, naturalised approach to the scientific study of human behaviour… at odds with the dominant cognitive neuroscience approach’. This is far from a movement, however; the mainstream cognitive sciences continue to wallow uncritically in the IP metaphor, and some of the world’s most influential thinkers have made grand predictions about humanity’s future that depend on the validity of the metaphor.
One prediction – made by the futurist Kurzweil, the physicist Stephen Hawking and the neuroscientist Randal Koene, among others – is that, because human consciousness is supposedly like computer software, it will soon be possible to download human minds to a computer, in the circuits of which we will become immensely powerful intellectually and, quite possibly, immortal. This concept drove the plot of the dystopian movie Transcendence (2014) starring Johnny Depp as the Kurzweil-like scientist whose mind was downloaded to the internet – with disastrous results for humanity.
Fortunately, because the IP metaphor is not even slightly valid, we will never have to worry about a human mind going amok in cyberspace; alas, we will also never achieve immortality through downloading. This is not only because of the absence of consciousness software in the brain; there is a deeper problem here – let’s call it the uniqueness problem – which is both inspirational and depressing.
Because neither ‘memory banks’ nor ‘representations’ of stimuli exist in the brain, and because all that is required for us to function in the world is for the brain to change in an orderly way as a result of our experiences, there is no reason to believe that any two of us are changed the same way by the same experience. If you and I attend the same concert, the changes that occur in my brain when I listen to Beethoven’s 5th will almost certainly be completely different from the changes that occur in your brain. Those changes, whatever they are, are built on the unique neural structure that already exists, each structure having developed over a lifetime of unique experiences.
This is why, as Sir Frederic Bartlett demonstrated in his book Remembering (1932), no two people will repeat a story they have heard the same way and why, over time, their recitations of the story will diverge more and more. No ‘copy’ of the story is ever made; rather, each individual, upon hearing the story, changes to some extent – enough so that when asked about the story later (in some cases, days, months or even years after Bartlett first read them the story) – they can re-experience hearing the story to some extent, although not very well (see the first drawing of the dollar bill, above).
This is inspirational, I suppose, because it means that each of us is truly unique, not just in our genetic makeup, but even in the way our brains change over time. It is also depressing, because it makes the task of the neuroscientist daunting almost beyond imagination. For any given experience, orderly change could involve a thousand neurons, a million neurons or even the entire brain, with the pattern of change different in every brain.
Worse still, even if we had the ability to take a snapshot of all of the brain’s 86 billion neurons and then to simulate the state of those neurons in a computer, that vast pattern would mean nothing outside the body of the brain that produced it.This is perhaps the most egregious way in which the IP metaphor has distorted our thinking about human functioning. Whereas computers do store exact copies of data – copies that can persist unchanged for long periods of time, even if the power has been turned off – the brain maintains our intellect only as long as it remains alive. There is no on-off switch. Either the brain keeps functioning, or we disappear. What’s more, as the neurobiologist Steven Rose pointed out in The Future of the Brain (2005), a snapshot of the brain’s current state might also be meaningless unless we knew the entire life history of that brain’s owner – perhaps even about the social context in which he or she was raised.
Think how difficult this problem is. To understand even the basics of how the brain maintains the human intellect, we might need to know not just the current state of all 86 billion neurons and their 100 trillion interconnections, not just the varying strengths with which they are connected, and not just the states of more than 1,000 proteins that exist at each connection point, but how the moment-to-moment activity of the brain contributes to the integrity of the system. Add to this the uniqueness of each brain, brought about in part because of the uniqueness of each person’s life history, and Kandel’s prediction starts to sound overly optimistic. (In a recent op-ed in TheNew York Times, the neuroscientist Kenneth Miller suggested it will take ‘centuries’ just to figure out basic neuronal connectivity.)
Meanwhile, vast sums of money are being raised for brain research, based in some cases on faulty ideas and promises that cannot be kept.The most blatant instance of neuroscience gone awry, documented recently in a report in Scientific American, concerns the $1.3 billion Human Brain Project launched by the European Union in 2013. Convinced by the charismatic Henry Markram that he could create a simulation of the entire human brain on a supercomputer by the year 2023, and that such a model would revolutionise the treatment of Alzheimer’s disease and other disorders, EU officials funded his project with virtually no restrictions. Less than two years into it, the project turned into a ‘brain wreck’, and Markram was asked to step down.
We are organisms, not computers. Get over it. Let’s get on with the business of trying to understand ourselves, but without being encumbered by unnecessary intellectual baggage. The IP metaphor has had a half-century run, producing few, if any, insights along the way. The time has come to hit the DELETE key.
Thomas Sowell Quotes (@ThomasSowell) posted at 0:30 am on Tue, Feb 11, 2025:
“Activism is a way for useless people to feel important, even if the consequences of their activism are counterproductive for those they claim to be helping and damaging to the fabric of society as a whole.”
