How to Add Emotion Detection to an AI Voice Agent
Customer Intelligence
Science

Georgia Allen

Most voice agents are deaf to the most informative part of the signal they receive. In the standard cascaded pipeline, speech-to-text converts the caller's audio into words, a large language model (LLM) decides what to say, and text-to-speech (TTS) says it. The first step keeps the lexical content and discards the prosody: pitch, loudness, tempo, rhythm and voice quality. Prosody is where much of the emotional information in speech lives.
The result is an agent that responds to what a caller said and has no record of how they said it. A caller who answers "yes" with rising, energetic pitch and one who answers "yes" in a flat, compressed voice produce identical transcripts, and the agent treats them identically.
This guide sets out a four-step method for restoring that information: read emotion from the raw audio, pass it to the LLM as context, use it to set the TTS tone, and measure it across the full call.
Why Does Emotion Get Lost in a Voice Agent?
Emotion gets lost because transcription keeps the words and drops the acoustic features that carry feeling. Vocal emotion is encoded in measurable properties of the signal, and those properties have no place in a transcript.
The research on this is extensive. A review by Juslin and Laukka in Psychological Bulletin covered 104 studies of vocal expression and found that discrete emotions are communicated through consistent patterns of acoustic cues, including pitch level, pitch variability, intensity and speech rate. Listeners decode those patterns reliably, which is why a voice can sound tense, warm or defeated regardless of the words it carries.
The voice may also be the most reliable channel for emotion in conversation. In five experiments with more than 1,700 participants, Kraus at Yale found that people judged others' emotions more accurately when they only listened than when they could both see and hear them. The one condition that produced the worst accuracy was a synthetic voice reading the transcript of the interaction, the closest human analogue to a pipeline that has stripped the prosody out.
Step 1: Read Emotion From the Audio, in Parallel With Transcription
The first step is to fork the audio stream so one branch goes to speech-to-text and the other goes to a vocal emotion model. Running the two in parallel, rather than in sequence, keeps the emotion signal from adding a serial delay to the turn.
Timing matters because human conversation runs on a tight clock. In a study of 10 languages across five continents, Stivers and colleagues found that speakers consistently avoid overlap and minimize silence between turns, with the most common gap between a question and its answer falling between 0 and 200 milliseconds. Any component that sits in series between the caller and the response eats into that budget.
Valence's Pulse API is built for this position in the pipeline. It classifies roughly five-second segments of speech and returns a result in about 50 milliseconds of processing time once it has the audio, running alongside transcription. On a held-out test set of diverse voices crowdsourced worldwide, none of which appeared in training, the model classifies emotion with 92% accuracy. It is trained on real, self-labeled conversational speech rather than acted emotion, across ages, accents, genders and neurotypes.
Step 2: Give the LLM the Emotional Context
The second step is to merge the emotion label with the transcript before the LLM sees the turn. The LLM then reasons over two inputs, what the caller said and how they said it, instead of one.
In practice, this means annotating each user turn with the detected emotion and adding instructions to the system prompt that define what the agent should do with it. A turn labeled frustrated can prompt the agent to acknowledge the problem before offering a fix. A turn labeled confused can prompt a shorter answer and a clarifying question. A turn where positive words arrive in a negative tone can prompt the agent to check in rather than proceed.
That last case is the one transcripts miss entirely. When the words and the tone disagree, the agent needs both signals to decide how to respond, and an LLM that receives only text has no way to detect the disagreement.
Step 3: Tell the TTS How to Sound
The third step is to use the caller's emotion to choose the tone of the agent's reply. Modern TTS engines accept tone controls, such as audio tags in ElevenLabs and emotion tags in Cartesia Sonic, and Pulse output maps to those tags directly. The LLM decides what to say; the emotion signal decides how it should sound.
The design principle here is regulation, not mirroring. Hatfield, Cacioppo and Rapson defined emotional contagion as the tendency to automatically mimic another person's expressions and vocalizations and, as a consequence, converge with their emotional state. Contagion runs in both directions. An agent that answers an angry caller in an agitated voice gives the caller more agitation to converge with, while a calm, steady reply offers a different state to converge toward.
A practical mapping follows from this. Frustration calls for a calm, even tone. Sadness calls for a warm, slower delivery. Excitement can be met with matching energy. Confusion calls for a measured, patient pace. The mapping belongs in configuration where your team can test and tune it, not hard-coded into the agent.
Step 4: Measure Emotion Across the Whole Call
The fourth step is to treat emotion as a time series, not a single reading. One label describes a moment; the trajectory across a call shows whether the agent is helping.
Pulse includes the Happiness Quotient, a score from 0 to 100 produced by a recurrent neural network for every turn. Scores from 0 to 50 indicate negative valence and scores from 50 to 100 indicate positive valence. Plotting the score turn by turn shows where a conversation turned, which agent responses preceded a recovery, and which ones preceded a decline.
This gives your team a live measure on every call, rather than a post-call survey answered by the few callers who choose to respond. It also gives you an evaluation set: when you change a prompt or a tone mapping, you can compare the emotional trajectory of calls before and after the change.
Two Inputs, Two Outputs
An emotionally aware voice agent comes down to a simple architecture. Two signals come in from the caller, the words and the voice, and two decisions go out, what to say and how to say it. Most pipelines today keep one input and control one output. Restoring the second input lets you control the second output. New to emotion AI? Start with our guide to what emotion AI can do for voice agents.
Try Pulse free for 14 days and see what your agent has been missing.




