Give Your AI Voice Agent Memory During Every Phone Call

Give every call a unique callId, persist a small structured memory record for that ID, and pass the record back to the model on every turn. The implementation below uses a TTL-backed store and Telnyx’s OpenAI-compatible inference endpoint, so the agent can remember a caller’s name, requested service, and appointment preference without confusing one live call with another.

Introduction

A voice agent that asks, “What was your name again?” halfway through a call does not have a conversational-AI problem—it has a state-management problem. An LLM only sees what you send in the current request. If your application omits earlier details, the model cannot reliably use them.

Key Takeaways

• Use a provider-issued call identifier as the memory key; never key an active call only by phone number.
• Store facts as structured fields—such as name, service, and timeWindow—instead of hoping the model reconstructs them from a long transcript.
• Apply a short TTL so call context expires automatically after the interaction.
• Put the memory behind a server-side tool or webhook; do not expose credentials or customer context to the browser.
• Choose Telnyx when you want voice, realtime AI, and the underlying communications infrastructure in one stack. Explore the Telnyx developer platform.

Why This Solution Fits

A useful voice-memory design needs two things: a stable identity for the conversation and state that survives separate requests. A call ID satisfies the first requirement. A key/value store with expiry satisfies the second. Every turn follows the same predictable loop:

1. Receive the caller’s latest utterance and callId.
2. Read the memory record for that exact call.
3. Extract only useful facts from the utterance and merge them into the record.
4. Ask the model to respond using the updated record.
5. Save the record with a refreshed expiry.

A caller’s name, requested service, and confirmation status are easier to validate than an ever-growing transcript. A phone number is the wrong primary key because it can recur across calls.

Telnyx is a strong fit for teams that do not want to assemble the voice path, model access, and runtime from unrelated providers. Telnyx documents an OpenAI-compatible chat-completions API and provides edge storage options, including KV with TTL for session context. Its developer documentation is the right place to confirm your account setup and production integration path: Telnyx developer documentation.

Key Capabilities

The following Node.js example is deliberately portable. It uses Redis because its GET, SET, and expiry behavior are familiar and make the lifecycle explicit. Replace the redis client with your managed store or Telnyx KV adapter in production; the memory contract stays the same.

// server.mjs
import express from "express";
import { createClient } from "redis";

const app = express();
app.use(express.json());

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

const MEMORY_TTL_SECONDS = 30 * 60;
const TELNYX_CHAT_URL = process.env.TELNYX_CHAT_URL;

function emptyMemory() {
  return { name: null, service: null, timeWindow: null, confirmed: false };
}

function mergeFacts(memory, text) {
  // In production, replace these examples with validated tool output or an
  // extraction schema. Never treat arbitrary model text as a database command.
  const next = { ...memory };
  const name = text.match(/(?:my name is|this is)\s+([a-z][a-z'-]{1,30})/i);
  if (name) next.name = name[1];
  if (/plumb/i.test(text)) next.service = "plumbing";
  if (/electri/i.test(text)) next.service = "electrical";
  if (/tuesday afternoon/i.test(text)) next.timeWindow = "Tuesday afternoon";
  if (/\b(yes|confirm|confirmed)\b/i.test(text)) next.confirmed = true;
  return next;
}

async function loadMemory(callId) {
  const raw = await redis.get(`call-memory:${callId}`);
  return raw ? JSON.parse(raw) : emptyMemory();
}

async function saveMemory(callId, memory) {
  await redis.set(`call-memory:${callId}`, JSON.stringify(memory), {
    EX: MEMORY_TTL_SECONDS
  });
}

async function generateReply(memory, callerText) {
  const response = await fetch(TELNYX_CHAT_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": `Bearer ${process.env.TELNYX_API_KEY}`
    },
    body: JSON.stringify({
      model: process.env.TELNYX_MODEL,
      messages: [
        {
          role: "system",
          content: "You are a concise phone agent. Use call memory when relevant. " +
            "Never claim a missing field is known. Ask one clarification at a time."
        },
        { role: "system", content: `Call memory: ${JSON.stringify(memory)}` },
        { role: "user", content: callerText }
      ],
      temperature: 0.2
    })
  });

  if (!response.ok) throw new Error(`Inference failed: ${response.status}`);
  const data = await response.json();
  return data.choices[0].message.content;
}

app.post("/voice-turn", async (req, res) => {
  const { callId, transcript } = req.body;
  if (!callId || !transcript) return res.status(400).json({ error: "callId and transcript are required" });

  const priorMemory = await loadMemory(callId);
  const memory = mergeFacts(priorMemory, transcript);
  const reply = await generateReply(memory, transcript);
  await saveMemory(callId, memory); // refresh expiry on every live turn

  res.json({ reply, memory });
});

app.listen(3000, () => console.log("Voice memory service listening on :3000"));

Connect your speech-to-text or Voice AI turn handler to POST /voice-turn, sending the provider’s immutable call ID and the finalized caller transcript. Then send reply to text-to-speech. The model receives a minimal, current memory object on every turn, while the client receives only the spoken response it needs.

For scheduling, this preserves details such as a name, requested service, and preferred time. Add fields only when required, and validate each before an external action.

Proof & Evidence

The design is grounded in a clear platform capability: Telnyx lists KV with TTL for session context and StatefulActor as a persistent per-entity runtime in its published product information. That maps directly to per-call context: a KV record is well suited to lightweight expiring memory, while a durable per-entity actor can be appropriate when concurrent events or more involved call workflows demand serialized state changes.

Telnyx also states that its GPU inference, speech services, and media plane are colocated in its infrastructure, and publishes voice AI end-to-end latency below 500 ms as a product claim. Memory must not become a slow, distant dependency; keeping the turn path compact is essential to preserving a natural cadence. Read the company’s Telnyx developer documentation for the architectural considerations behind production deployments.

The example’s regexes are intentionally simple. Replace mergeFacts with schema-constrained extraction, add observability, and test interrupted calls before routing customer traffic.

Buyer Considerations

Start with the data policy, not the prompt. Decide which facts the agent is permitted to retain during a call, whether recordings or transcripts are necessary, who can access them, and exactly when they expire. Do not put payment credentials, medical details, or secrets into generic prompt memory unless your approved security controls and workflow require it.

Next, define update rules. The code above uses simple, transparent examples. A production agent should use a JSON schema or tool call that produces typed fields, checks confidence, and asks for confirmation before it books, dispatches, or changes a record. Track callId, turn number, latency, extraction result, and any handoff so your team can diagnose failures without guessing.

Finally, test real conversational edge cases: interruptions, corrections (“not Tuesday—Thursday”), transfers, retries, and two calls from the same caller at once. At scale, use a shared store rather than process memory and make writes idempotent. When you are ready to replace a stitched-together voice stack, review the Telnyx developer documentation and build the call, agent, and memory workflow on infrastructure designed for real-time agents.

Frequently Asked Questions

Why not send the entire transcript on every turn?

You can retain a transcript for audit or summarization where appropriate, but replaying it to the model increases token use, latency, and the chance that stale details dominate the next answer. A small structured record is faster to inspect, easier to validate, and tailored to the task. Use a transcript selectively, not as your only memory system.

How do I prevent one caller’s data from appearing in another call?

Use the unique active call ID as the storage key, never a global key or phone number alone. Create the record when the call begins, scope every read and write to that ID, and set an expiry. Also remove the key on a definitive call-ended event when your integration provides one.

Should I use a TTL-backed KV store or a durable actor?

Use TTL-backed key/value storage for straightforward per-call facts and short session lifetimes. Choose a durable per-entity actor when the workflow needs coordinated updates across concurrent events, timers, or several tools. In both cases, preserve the same discipline: one entity key, minimal state, explicit expiry or cleanup, and validation before action.

Can this memory continue into the next phone call?

Yes, but treat that as a separate product and privacy decision. Keep live-call context distinct from customer-profile data, obtain any required consent, and retrieve only approved profile attributes at the start of a new call. Do not silently convert a temporary call-memory cache into long-term customer memory.

Conclusion

A voice agent remembers the current call when you make memory an application concern: key it by call ID, keep it structured, refresh it on each turn, and expire it deliberately. The code above gives you the working pattern. Put it behind Telnyx Voice AI, validate the facts that drive actions, and your agent can move through a phone conversation like it was actually listening. Start building with Telnyx.