Add support for OpenAI `realtime` voice models.

[homanp]

Feature Description

Add support for the new OpenAI realtime modes.

Use Case

Building Voice to Voice realtime apps.

Additional context

https://openai.com/index/introducing-the-realtime-api/

[kousaku-maron]

As realtime models use websocket, maybe we have to create abstracted functions first, such as streamText and streamObject.

[homanp]

As realtime models use websocket, maybe we have to create abstracted functions first, such as streamText and streamObject.

I think that is reasonable. Would love to get the @lgrammel take on how this would best implemented.

[lgrammel]

This is currently limited to openai. In the near term, I recommend using their SDK while we explore how to best implement this feature - there are no gains in compatibility between different providers yet. It'll be a major change and it's only for a single provider at this point, so I want to take some time to think about it.

[mrasoahaingo]

Okay but it stil in BETA, so...

[jermahs]

I have been asked to implement this in an upcoming app, this would be great to have exposed through useChat somehow or another supported hook so we don't have to change things.

[spullara]

now that openai has released this on their own web app it would be great to be able to offer similar support in our web apps that are currently using this sdk\

[mrasoahaingo]

Did someone made a benchmark between the realtime and completion token usage? Does the realtime send all the history to the model but it's done under the hood?

[Kae7in]

So long as it moves from beta to GA, is this something Vercel plans to support?

[ai-fy]

This is currently limited to openai. In the near term, I recommend using their SDK while we explore how to best implement this feature - there are no gains in compatibility between different providers yet. It'll be a major change and it's only for a single provider at this point, so I want to take some time to think about it.

Meanwhile Google Multimodel Live API and Elevenlabs Realtime voice Agents have joined OpenAI and all three are providing realtime voice APIs, did you reconsider the implementation? Would be amazing to have an abstraction layer from vercel.

[da1z]

eleven labs uses multistep approach for theirs realtime voice agent: speech to text -> generateText -> text to speech . maybe it is something you can explore and abstract in vercel ai sdk? so we can easily change models for each step

[brandonwheat]

any update on this now that their are multiple providers with this feature?

[leoreisdias]

Right now this issue is related to #4082 .

Both OpenAI and Google's APIs use WebSocket for real-time streaming. OpenAI's API streams text, while Google's supports multiple data types like images and text. They have similar connection approaches but differ in payload formats and authentication. A unified abstraction layer with provider-specific adapters in the Vercel AI SDK could support both APIs seamlessly.