Take: A Real-Time Talking Avatar, Not a Rendered Video: Katherine Solano

01 · The pain

What was actually broken.

Most 'AI avatar' tools render a finished video after the fact. A live conversation needs frames rendered while the audio is still playing, a different problem, not a slower version of the same one.
The three funded leaders in this space (D-ID Agents, Tavus, HeyGen LiveAvatar) don't publish how self-hostable their approach is, or what it costs on rented infrastructure instead of their metered API.
Existing open-source pieces (speech-to-text, an LLM, text-to-speech, a lip-sync model) are all real, but nobody had chained them into one documented real-time loop with honest latency and cost numbers.

02 · Who this is for

Persona.

A team that wants a talking avatar without an API-per-seat bill Products like D-ID Agents, Tavus, and HeyGen LiveAvatar all sell the same real-time avatar pattern behind a metered API. A team that wants to own the stack, or just wants it cheaper at volume, has to know whether that pattern can actually be self-hosted, and what it costs to run.

03 · The problem

What the design had to solve.

Work out whether a live, talking avatar, listens, thinks, and answers back through a photo in real time, can be built self-hosted on rented GPU hardware, using the same architectural pattern the funded players use, and be honest about where it wins and loses against them.

07 · Process

From skeleton to final screens.

Started with the wrong problem, caught it before building

Scoped this first as batch generation, photo and audio in, video file out, competing with D-ID's Creative Studio product. Realized that's a different system entirely from 'a photo that talks back in a live conversation.' A rendered clip and a real-time reply don't share an architecture: one is request-then-wait, the other needs a GPU held for the call's duration and a model fast enough to render as the audio plays.

Found the real pattern, not a guess

Tavus, HeyGen, and D-ID Agents all follow the same documented shape: speech-to-text into an LLM into text-to-speech, then hand that audio to a separate avatar-rendering layer that publishes video into a WebRTC room. LiveKit ships this as a plugin architecture with 14+ avatar providers behind one interface, confirmed this is a solved, replicable pattern, not something to reverse-engineer from scratch.

Picked the model on evidence, not familiarity

Rejected SadTalker (built for batch rendering, seconds-to-minutes per clip, wrong tool for a live loop) and LatentSync (needs an existing video as input, not a single photo). Landed on MuseTalk, Tencent's open-source, MIT-licensed model built for real-time lip-sync at 30+ fps, verified against a working open-source reference stack already running the same Whisper-plus-MuseTalk combination.

Did the honest cost and latency math before writing code

Funded leaders hit sub-600ms (Tavus Phoenix-4) to roughly 1-2s (HeyGen LiveAvatar) end to end. A realistic fully self-hosted stack on this architecture lands 1.5-3s, a real gap, not a rounding error. Priced the GPU floor too: a live session needs a persistent RunPod A100 Pod, not Serverless, at $1.39-1.59/hr per concurrent conversation, before any STT/LLM/TTS API cost. That number is what decides whether this is viable at any given price point, not a guess.

09 · Goals & intended outcomes

What this was designed to do.

Goals

Prove the real-time avatar pattern (STT into LLM into TTS into a rendering layer into WebRTC) is genuinely replicable outside the funded platforms, with real open-source pieces, not just plausible on paper.
Get honest, sourced numbers on the latency and per-conversation GPU cost of running it self-hosted, so the tradeoff against Tavus/HeyGen/D-ID Agents is a real comparison, not a hunch.
Ship one working demo end to end before making any claim about the product being live.

Intended outcomes

Full real-time architecture designed and sourced: LiveKit for the WebRTC/avatar-plugin layer, MuseTalk for lip-sync, a documented open-source reference implementation confirming the approach works end to end.
Concrete, sourced cost and latency numbers instead of vendor marketing: ~1.5-3s self-hosted latency versus 600ms-2s funded, ~$1.39-1.59/hr GPU floor per live conversation on RunPod Pods.
Caught and abandoned the wrong architecture (batch video generation) before writing throwaway code against it, by pressure-testing what 'a photo that talks back' actually requires.
Currently mid-build: the worker repo and real-time scaffold exist, hosting isn't stood up end to end yet. This entry will update when a live demo is running.