Notes from the build

A demo voice agent can take two seconds to answer and still look fine in a screen recording. Put a real person on the line, a 74-year-old on a care call or a billing rep at a payer, and two seconds breaks the call. They talk over the agent. They hang up. They decide it is broken.

People expect fast replies because they get them from each other. A 2009 study in PNAS recorded conversations in 10 languages, from small indigenous communities to major world languages. Every one of them avoided overlapping talk and kept the silence between turns short, and each language's average gap sat within 250 ms of the cross-language mean (Stivers et al.). So the number we chase is end-to-end latency under about 250 milliseconds, from the moment the caller stops talking to the first syllable of the reply.

Where the time goes

A naive pipeline runs each stage in order, and every stage waits for the one before it to finish.

• Endpointing. The system has to decide the caller is done. Waiting for a long silence is safe and slow. Cutting in early interrupts people mid-thought.
• Recognition. Batch speech-to-text starts only after the audio ends.
• Reasoning. The model reads the whole prompt before it produces the first token.
• Synthesis. Text-to-speech waits for a full sentence before it plays anything.

Add those up and you get the two-second pause that makes people talk over the agent.

How we cut it

We stream audio into speech-to-text and read interim results as they arrive. Google's Speech-to-Text documentation describes the split we rely on. An interim result is the current guess for a stretch of audio and may change, and a final result is the last, best guess for it (Google Cloud docs). We start reasoning on the partial transcript, so most of the thinking is done by the time the final result lands.

We pick a different model for each stage. The fast one sits on the critical path. The larger one runs where we can hide its latency. We stream the reply into text-to-speech token by token, so the first syllable plays while the rest of the sentence is still being written.

We also handle barge-in. When the caller starts to speak, the agent stops its own audio and listens, the way a person would. Without this, the agent keeps talking over a patient who is trying to correct it.

What it gets us

Our agents run roughly 4 to 6x faster than typical healthcare voice AI. A patient with hearing aids and a lot on their mind does not notice a delay. None of this shows up in a demo. It shows up on call number ten thousand, when a patient decides whether to keep picking up.

One of the AI healthcare startups we built for needed to get paid faster. Visiting-nurse reimbursement means calling insurers, sitting through phone trees, waiting on hold and answering the same questions many times a day. An agent can do this work if it survives contact with a real interactive voice response system, or IVR.

IVRs are hard on agents. Every payer's tree is different and changes without notice. One call mixes keypad menus ("press 3"), speech prompts ("say your member ID"), dead air, and hold music that can sound like a person picking up. A naive agent loses its place within seconds.

A state machine, not a chat

We treat the call as a control problem. The agent runs a state machine over the call. At each moment it classifies the audio as a menu, a prompt, hold, or a live human, and acts for that state. It keeps a model of where it sits in the tree, so when the payer moves an option it can back up and recover instead of guessing.

Keypresses are tones

Pressing 3 on a phone sends two tones at once. DTMF signaling, standardized in ITU-T Recommendation Q.23, uses a low group of 697, 770, 852 and 941 Hz and a high group of 1209, 1336, 1477 and 1633 Hz. The digit 3 is 697 Hz plus 1477 Hz (DTMF). Many IVR menus only listen for tones, so the agent plays the tone pair on the audio channel instead of saying "three".

Hold music is not a person

Telling hold music from a human pickup is its own classifier. Twilio's answering machine detection shows the shape of the problem. It measures stretches of speech and silence, and by default it treats a speech segment longer than 2400 ms as a likely machine and 1200 ms of silence as the end of a segment (Twilio docs). A recorded loop talks in long, regular runs. A person says a short greeting and stops to wait for an answer. Timing features of this kind keep the agent from pitching its questions to elevator music.

The agent also has a hard escalation line. When its confidence drops below the threshold, it hands the call to a person. Whether a nurse gets paid is not something to guess on.

A version that walks one happy path takes a weekend. The version that holds up across hundreds of payers is the product.

Here is a failure a demo will not show. On a care call, a patient says they have been tired and a little dizzy. The AI reviews their medications and asks about the ones unrelated to those symptoms. It never checks the blood pressure medication. The patient had been taking twice their prescribed dose of lisinopril. The AI missed the one thing that mattered.

Why dizziness points at the blood pressure pill

Lisinopril lowers blood pressure. MedlinePlus lists dizziness as a common side effect and lists lightheadedness and fainting among the signs of an overdose (MedlinePlus). In an older patient, a double dose can drop blood pressure far enough to cause a fall. The CDC reports that over 14 million adults aged 65 and older, 1 in 4, report falling every year (CDC).

A map from symptoms to drug classes

A chatbot answers questions. A clinical agent has to ask them. The layer most teams skip is a map from reported symptoms to the drug classes most likely to cause them. With that map, dizziness and fatigue raise the priority of asking about antihypertensives, anticoagulants and insulin, so the agent probes the dangerous drug first. That question is the one that keeps someone out of the ER.

Keep the map as reviewable data, outside the prompt. A clinician can then read it line by line and change it without anyone touching the model.

Evals that score the catch

A better prompt will not get you this. Evals will. We run the agent against real call transcripts and score one thing, whether it surfaced the actual risk. Fluency earns no points. A test case for this problem holds the following.

• The transcript up to the moment the patient mentions symptoms.
• The patient's medication list.
• The expected behavior, here a question about the lisinopril dose before the call moves on.

Run the same case many times. A model that asks the right question four times out of five still misses the fifth patient, so track the rate across repeated runs and treat any miss on a safety case as a release blocker. Turn every real miss found in production into a new test case. The set then grows with the failures that matter.

A fluent agent is easy to build. An agent that is correct on the calls that matter is the job, and it is also where the product becomes defensible.

Anyone can build a RAG demo. Embed some documents, put the top chunks into a prompt, ship a chatbot. It works until people with real stakes depend on it, like a private equity team deciding whether to buy a company, where one confident wrong answer is expensive.

The diligence tool we built for a private equity firm read large amounts of target-company material and surfaced what could move a deal. "Pretty good" was not acceptable. Most of the work went into retrieval quality, grounding every claim to a source a person can check, and an eval harness that catches regressions before they reach the client.

Hybrid retrieval

Embeddings find passages with similar meaning. They are weak on exact strings like a company name, a parcel number or a clause reference. BM25, the ranking function from the Okapi system, scores documents on how often the query terms appear, with a length normalization parameter b that is usually set to 0.75 (BM25). We run both searches, merge the two ranked lists with rank fusion, and rerank the top results.

Anthropic published numbers for this pattern. Adding context to chunks before indexing, searching with both embeddings and BM25, and reranking cut their top-20 retrieval failure rate from 5.7% to 1.9% (Contextual Retrieval).

Grounding and evals

Every claim in an answer links to the passage it came from, so an analyst can verify it before it goes into a memo. The eval set holds questions with known answers and known source passages. A change to chunking, prompts or models ships only if retrieval and answer scores hold.

Cost and speed

A naive setup pays top-model prices on every query and waits for them. We push the hot path onto smaller fine-tuned models, cache aggressively, and save the expensive model for the hard calls. Research on model cascades shows how much room there is. FrugalGPT learned which model to call for each query and matched GPT-4's performance with up to 98% lower cost (FrugalGPT).

The tool ended up saving the firm about $32M a year. That came from the pipeline around the model, which made the output something the team could trust.

A v1 is easy now. Making it reliable, defensible and affordable at scale is the hard part, and it is the work we do.

Most healthcare voice AI is a single model on a phone line. It does fine in the middle of a conversation and fails at the edges, when a patient mentions chest pain, trails off mid-sentence, or quietly admits they stopped taking a medication.

We run a constellation instead. One main model talks to the patient. Behind it, a dozen specialized listeners read the same transcript in parallel, and each one watches for a single thing.

• The medication listener tracks drug names, doses, and lines like "I forgot what this one's for".
• The emergency listener watches for chest pain, stroke signs and falls.
• Cognitive, emotion, symptom, social, environment and service listeners each watch their own signal.

Each listener is its own model, tuned to catch what a generalist misses.

Four priority levels

Listeners do more than observe. Each one can push context into the main model through a priority queue with four levels, background, queued, critical and immediate interrupt. Higher levels reach the main model sooner. An immediate interrupt stops the main model mid-word and is reserved for emergencies.

What an interrupt has to do

Preemption is the hard part of this design. When an interrupt arrives, the reply in progress is already partly spoken. The system has to stop text-to-speech playback, throw away the rest of the generated reply, and prompt the main model again with the new context at the top. Anything less and the agent finishes a cheerful sentence about the weather after the patient said their chest hurts.

Listeners also fire more than once on the same signal, because every new transcript chunk gets re-read. Give each event a key, such as the listener, the signal and the call, and drop repeats. Otherwise one mention of a missed pill fills the queue.

Run the listeners as separate workers that subscribe to the transcript stream, and keep them stateless between chunks except for what they write to the call record. One slow listener then cannot delay the others or the main model, and a crashed listener restarts without dropping the call.

When the emergency listener fires, it cuts the conversation, starts the 911 protocol, notifies family and pre-compiles the patient's history for EMS.

Why this shape

The patient hears one voice and never sees the models behind it. Because the listeners are modular, we add a new one without retraining the model that talks. A single prompt cannot do that. It is also why one nurse can stand behind ten times the panel. The listeners do the catching, and the nurse makes the calls that need a human.

We did not add clinician supervision at the end to pass a compliance review. It is the architecture.

What Medicare requires

Medicare chronic care management is built around a supervising practitioner. The CMS booklet on these services says eligible patients have two or more chronic conditions expected to last at least 12 months, and practitioners bill for at least 20 minutes of care coordination a month. Clinical staff work under "general supervision", which means the billing practitioner directs and controls the service without being physically present (CMS MLN909188).

The same booklet assigns the clinical staff codes, CPT 99490, 99439, 99487 and 99489, to general supervision under the physician fee schedule. Physicians, nurse practitioners, physician assistants, clinical nurse specialists and certified nurse-midwives can bill these services, and clinical staff deliver much of the work under their direction. An AI agent fits into that structure as a tool the clinical staff use. It does not replace the person the rules are written around.

Trust needs the same structure. Patients and clinicians will not hand their health to a black box. So the system follows one rule, the nurse decides. The AI listens, extracts, drafts and flags. It never makes the clinical decision.

What that rule means in the software

• Reviewable. The nurse sees every action the AI proposes before it changes the care plan.
• Attributable. Each flag carries the transcript lines that triggered it, so the nurse can check the evidence before acting.
• Escalatable. Any item can go to the person whose license is on the line.

This sounds like a limit. It is what makes the rest possible. Because a human makes the decision, we can let the AI be aggressive where it is strong, catching the double dose, surfacing risk, writing the note, without betting a patient's safety on the model being right.

Tuning what reaches the nurse

A supervised system fails in two directions. It can miss a flag, or it can bury the nurse in flags until they stop reading them. Tune both on the same eval set. Give each test call a list of what a nurse should see from it, then measure how many of those items the system raised and how many extra items came along. Ship a threshold change only if the misses stay at zero on the safety cases.

Ordering matters as much as filtering. Sort the nurse's queue by urgency first and by patient second, so a possible double dose for one patient sits above a refill reminder for another.

The AI's job is to make sure nothing important reaches the nurse late and nothing unimportant reaches them at all. In healthcare, that supervision layer is what makes the system defensible.

Most AI healthcare companies sell software and hope someone uses it. We took the other path. Own the operator, and you own every interaction.

When the listeners sit on every call, you deliver care and you also build a longitudinal record no one else has. It covers medication adherence over time, symptom progression, mobility and fall risk, and behavioral signals. A clinic sees episodic snapshots. This is a continuous picture of a patient between visits, structured and queryable.

With enough of it, you stop reacting and start predicting, and you can flag the patient drifting toward an ER visit before it happens. A service becomes an asset. You only get there by owning delivery instead of licensing a tool into someone else's workflow.

Per-patient learning

Every call writes back what the patient committed to, who they mentioned, and when they actually pick up. A model fine-tunes per patient on what works. The system learned that bringing up someone's grandson before asking about medications lifts honest adherence reporting by about 45%, and it now does that by default.

Memory as a graph

Plain vector retrieval finds passages that look like the question. Microsoft's GraphRAG work documents where that breaks, when the answer needs facts connected across many documents (GraphRAG). A patient's history is that kind of problem. The grandson, the pharmacy and the missed Tuesday visit come up in different calls weeks apart.

So memory is a hybrid of a knowledge graph and a vector store. The graph holds people, medications, providers and events as entities with relationships, and one query can walk from the patient to everything connected. Vectors still handle fuzzy recall of what was said. Rebuilding memory this way cut retrieval latency in half. Most voice AI forgets the patient between calls. This system remembers.

Keeping it usable

A record like this only helps if the fields mean the same thing on every call. Have the listeners write structured events with a type, a value, a time and the transcript lines they came from, instead of free-text summaries. A dose change, a missed visit or a new fall then becomes a row a clinician or a model can query across months. Keep the transcript linked from every event, so nobody has to trust a summary they cannot check.

The company that owns the calls owns the data, and the data gets better with every call.

By the time an elderly patient hangs up, the refill is ordered, the follow-up visit is booked, the family is notified, and the visit note is in the chart. Nobody does paperwork after the call, because there is no after.

While the conversation flows, the listeners trigger background agents that work silently.

• An appointment agent checks availability, books a slot, and lets the main model offer it naturally ("does Thursday at 10 work?").
• A refill agent places the order and coordinates pharmacy delivery.
• A notification agent texts the documented caregiver and logs which channel they actually answered on.
• A recovery agent calls the vendor on a parallel line when something upstream failed, such as a missed nurse visit or a late delivery, and rebooks it mid-conversation.
• A documentation agent writes the visit note into the chart, so it is done before the patient says goodbye.

Doing things once

Agents that act during a live call will retry. A pharmacy API times out, a network drops, a listener fires twice on the same sentence. A careless retry books two appointments or orders two refills, which is the double-dose problem in software.

The fix is the pattern Stripe documented for payments. The client generates a unique key for each operation and sends it with the request. If the request fails and the client retries with the same key, the server returns the cached result of the first success instead of acting again (Stripe).

For a call agent, derive the key from the call and the intent, for example the call ID, the action and the drug. A second trigger in the same call produces the same key and does nothing new.

Ordering

Some agents depend on others. The notification agent should text the caregiver the appointment time only after the appointment agent has a confirmed slot. Model those dependencies explicitly, and let independent work run in parallel.

Agents should also report back into the conversation. When the appointment agent confirms a slot, push the confirmed time to the main model as background context, so it can mention the time before the call ends. When an agent fails, push that too. The main model then tells the patient someone will follow up, instead of promising something that did not happen.

A demo agent can book an appointment. The real version does six things at once while holding a warm conversation with a 78-year-old, and none of them collide. That orchestration is the job.

AI in healthcare is often killed by the six to eighteen month EHR integration project that has to happen before anything ships. Operators run Athena, Epic, Cerner, eClinicalWorks and a long tail of smaller systems. Each one is its own slog of APIs and vendor sign-off.

Why the standard API does not solve it

US certified health IT has to offer a standardized API under ONC criterion 170.315(g)(10), built on FHIR Release 4.0.1, the US Core implementation guide and SMART App Launch. The criterion covers read services only (ONC). An app can read a chart through it. Writing a visit note, coding an encounter or updating a care plan still needs vendor-specific write access, and that comes with the sign-off process.

Use the EHR like a nurse does

We skipped the whole category. The agent uses the EHR through a browser extension that clicks, types and navigates the same screens a nurse would. No API, no SDK, no vendor approval. It writes the visit note, codes the encounter and updates the care plan inside whatever system is already in place.

Making screen automation hold up

Screen automation has a reputation for breaking when a page changes. The cause is usually selectors tied to page structure. Playwright's documentation recommends locating elements by role and label, the way users and assistive technology see the page, because CSS and XPath selectors break when the DOM changes. It also finds the element fresh before every action (Playwright). An extension can follow the same rules.

• Find the save button by its role and label, not its position.
• Wait for the page to be ready before typing.
• Read back what was written and compare it with what was meant.
• Stop and ask a person when the screen does not match what the agent expects.

Log every write with a screenshot of the form before saving, the values entered and the record identifier, so an auditor can see what the agent did and when. The same checks catch drift. When a vendor ships a new screen layout, the read-back check fails, the agent stops, and someone updates the locator before it runs again.

This collapses the timeline from quarters to days. Buy the agency Friday, install the extension Monday, document into their existing EHR by Tuesday. RPA is unglamorous. It is also the difference between a pilot that ships and a roadmap that does not.