What crisis counseling at 988 taught me about the limits of AI
By George Cassidy Payne
“Are you a bot?”
It is often one of the first questions I get as a 988 text and chat crisis counselor. Before anything else, before the story unfolds, before trust has a chance to take shape, there is that test. The person on the other end is trying to figure out who or what they are talking to.
I always answer the same way. I tell them I am a real person. But what I have come to understand is that the question is not only about identity. It is about safety. It is about whether there is someone present who can actually understand and be accountable for what happens next.
As artificial intelligence becomes more fluent in the language of empathy, that question is becoming harder to answer in ways that feel obvious. If a machine can sound caring, reflective, even warm, what exactly distinguishes that from human care? And in moments of crisis, does the difference matter?
From where I sit, it does.
In crisis work, empathy is not just about saying the right thing. It is about being able to feel the weight of what is being said, even through a screen. The people I talk to are often scared, overwhelmed or unsure if they even want to keep talking. They may contradict themselves, minimize what they are going through or suddenly go silent. Sometimes the most important part of the interaction is not what is said, but what is not said.
You begin to notice patterns, but not in a mechanical way. A pause that lingers. A shift in tone. A response that feels just slightly out of sync. Over time, you learn to trust that sense that something is off. It is not a formula. It is a kind of attunement that develops through experience and through being human yourself.
AI can recognize patterns in language. It can be trained to respond in ways that sound empathic. But it does not feel urgency. It does not carry stakes. It is not changed by the interaction. When I am on a chat, I know that what I say and how I respond might matter in ways that extend far beyond the conversation. That awareness shapes every choice I make.
Responsibility is a constant presence in this work. When someone expresses thoughts of suicide or hints at immediate danger, decisions have to be made. Do you move toward emergency intervention? Do you stay with the person and continue to build trust? Do you nudge a little harder or give them more space?
There is no perfect script for these moments. Move too quickly and the person may shut down or disconnect. Move too slowly and you may miss a critical window. Every choice carries weight. And when the chat ends, that weight does not disappear. You carry it with you.
That is something a machine does not do.
Even mistakes feel different when they are human. There are moments when I have said something that did not land the way I intended. Maybe I moved too fast or misread the tone. But sometimes those moments open a different kind of connection. When you acknowledge the misstep and stay present, trust can deepen. The person on the other end recognizes that you are real, that you are trying, that you are willing to own your part in the conversation.
That kind of repair is part of the work. It is also part of what makes the work human.
Another thing I have learned is that “no” is rarely the end of the conversation. When someone refuses a suggestion or resists an idea, it is usually telling you something important. Maybe they are protecting themselves. Maybe they have had experiences that make trust difficult. Instead of pushing for compliance, you get curious. You explore what is underneath the resistance.
This is where collaboration happens. Not in getting someone to agree with you, but in creating a space where they feel heard enough to keep going. A machine might be able to guide someone toward a desired outcome. But collaboration depends on something more than direction. It depends on relationship.
There is also a physical side to this work that is hard to explain unless you have experienced it. Sometimes after a conversation, there is a feeling that stays with you. A heaviness or a sense that something unresolved is lingering. That feeling pushes you to reflect, to talk with a supervisor, to think more carefully about what happened. It is part of how you grow as a counselor.
AI does not have that kind of feedback. It can measure outcomes, but it cannot feel responsibility or the need to repair.
Timing is another place where the difference shows up.
In text and chat, you are constantly making decisions about when to respond, when to wait, when to let silence do some of the work. Too fast can feel intrusive. Too slow can feel like abandonment. Finding the right rhythm is not about following rules. It is about sensing where the other person is and meeting them there.
This becomes even more important in conversations shaped by trauma. People are often deciding in real time whether it is safe to share something that feels overwhelming or even unspeakable. That decision is not based only on the words they see on the screen. It is based on whether they feel there is a real person with them, someone who can understand, even imperfectly, what it means to be vulnerable.
AI can be trained on descriptions of trauma. It cannot inhabit that vulnerability.
So, when someone asks me, “Are you a bot?” I hear more than curiosity. I hear a question about presence, about trust, about whether anyone is really there.
The deeper issue is not whether AI can sound empathetic. It is whether we are willing to confuse sounding human with being human. In everyday situations, that distinction might not matter much. In moments of crisis, it matters a great deal.
Technology will continue to shape how care is delivered. It may expand access, reduce wait times and offer support in new ways. But in the moments where safety, dignity and human life are at stake, something more is required.
Not just a response, but a person who can stand behind it.