The open-source infrastructure to build customisable speech-to-speech systems.

A framework for wiring speech, language and action models into one pipeline. Configure it in config.cuelo, run it from the CLI or SDK, and trace audio, transcripts and latency on every turn.

Get started Star on GitHub

01Providers

Bring any model.

Pick a provider for each stage, or one speech-to-speech model end to end. Swapping is a one-line change.

Speech in

Smallest.ai

Smallest.ai · Deepgram · AssemblyAI · Whisper

Reasoning

Anthropic

Anthropic · OpenAI · Gemini · Groq

Speech out

ElevenLabs

ElevenLabs · Smallest.ai · Cartesia · Rime

orOpenAI Realtimeend to end

Anything else with an HTTP or WebSocket API plugs in as a custom provider.

02Config

Describe the system.

Talker, reasoning, actions, turn-taking and shared memory in one readable file. Change a line and the pipeline reloads.

config.cuelo
# config.cueloname: support-line talker:  asr: smallest  tts: elevenlabs  voice: "aria" llm:  model: anthropic  prompt: ./prompts/support.md actions:  - tools/lookup_order  - tools/update_address turns:  endpointing: semantic  barge_in: true memory:  shared: redis serve:  api: openai-realtime

running pipeline

  1. speech insmallesttalker.asr
  2. turn-takingsemanticturns
  3. reasoninganthropicllm.model
  4. actions2 toolsactions
  5. speech outelevenlabstalker.tts
  6. memoryredismemory

Hot reload keeps the call up. In-flight turns finish on the old config.

03Serve

One command. Your endpoint.

cuelo serve stands the pipeline up behind an API. Speak the OpenAI Realtime protocol so existing clients just work, or declare your own routes.

~/support-line

$ cuelo serve config.cuelo

✓talkersmallest → elevenlabs

✓llmanthropic · 2 tools

✓turnssemantic endpointing

✓memoryredis

✓storesqlite://cuelo.db

support-line · openai-realtime

WSws://localhost:4141/v1/realtime

POSThttp://localhost:4141/v1/realtime/sessions

GEThttp://localhost:4141/v1/models

ready in 412ms · watching config.cuelo

Pythonstock OpenAI SDK
CUELO = "http://localhost:4141"
client = AsyncOpenAI(
  base_url=f"{CUELO}/v1")

rt = client.beta.realtime
async with rt.connect(
  model="support-line") as call:
  ...
TypeScriptbrowser or Node
const url = "ws://localhost:4141"
const call = await connect(url)

mic.pipe(call)
call.on("audio", play)
Goservers and edge
call, err := cuelo.Connect(url)
if err != nil {
	return err
}
defer call.Close()

call.Stream(mic, speaker)

Any OpenAI Realtime client works unchanged. Point its base URL at Cuelo.

04Observe

See every turn.

Every turn is traced: audio, VAD, transcripts, turn decisions, tool calls and each model's timing. Cuelo points at the slow stage and tells you which line to change.

support-linelivesess_7f3a · turn 4 of 9

ttfb p50

488ms

ttfb p95

612ms

vad tail

144ms

llm ttft

248ms

tts ttfa

96ms

barge-ins

1

caller
vad
silence 144ms
asr
final +80ms
turn
end of turn · 0.94
llm
first token 248ms
tool
lookup_order 41ms
tts
first audio 96ms
talker

Where to improve

  1. Reasoning is half of response time on short replies.

    llm.fast: groqroute one-line answers to a faster model

  2. VAD waits 144ms after the caller stops.

    turns.min_silence: 400 → 300semantic endpointing already covers false stops

  3. Turn 6: caller barged in 1.2s into a reply.

    turns.barge_in.min_words: 2ignore backchannels like "mm-hm"

Call summary written by the agent

Caller asked to change the delivery address on order #48213. The talker looked up the order, confirmed the new address back to them and updated it. Resolved in 9 turns, 1m 54s.

  • intent · address_change
  • resolved
  • sentiment · neutral → positive
  • handoff · none

00:41.2callerHi, I need to change my delivery address.

00:43.1talkerSure. What's the new address?

saved to sqlite://cuelo.db9 turns · 2 tool calls · 1m 54s of audio

05Data

Your calls, your database.

Sessions, turns, transcripts, audio and summaries land in SQLite out of the box. Point Cuelo at Postgres or MySQL, or write an adapter for anything else.

config.cuelo

store:  url: sqlite://cuelo.db  audio: ./recordings   # or s3://  retain: 30d
  • sqlitedefault, zero setup
  • postgresfor production and teams
  • mysqlif that's what you run
  • customa small adapter interface
  • sessionspipeline, caller, started_at, duration, outcome
  • turnssession_id, index, ttfb_ms, vad_ms, barge_in
  • eventsturn_id, stage, name, at_ms, payload
  • transcriptsturn_id, speaker, text, confidence
  • audioturn_id, speaker, uri, duration_ms
  • summariessession_id, intent, resolved, text

06Agents

Hand it to your agent.

Cuelo ships an MCP server, so your coding agent can build, trace, test and tune pipelines with the same tools you use. Connect it once, then just ask.

1 · connect
$claude mcp add cuelo -- npx -y cuelo mcp

2 · paste a prompt

agent · ~/support-linecuelo mcp connected

>Build a support line that speaks Hindi and English, uses Smallest.ai for speech, and can look up orders.

●cuelo.inittemplate: support

●cuelo.config.edittalker.asr: smallest · talker.tts: smallest

●cuelo.config.editlanguages: [hi, en]

●cuelo.tool.addlookup_order

●cuelo.calltest call · 6 turns · no errors

✓support-line is live on ws://localhost:4141/v1/realtime

tools your agent gets

  • cuelo.init
  • cuelo.config.edit
  • cuelo.tool.add
  • cuelo.call
  • cuelo.trace
  • cuelo.eval
  • cuelo.test

Prefer the terminal? It's all a plain CLI too: cuelo init cuelo dev cuelo call cuelo test cuelo eval cuelo trace.