Add support for Gemini Vertex

[the-gigi]

See full discussion here:
#221

Here is the POC of Gemini in a simple-openai fork:
https://github.com/the-gigi/simple-openai/commit/40fe114eefdf83b589abe27f74a4b20dc1a81138

[sashirestela]

Gemini API has two implementations:

• Vertex AI Gemini API
• Google AI Gemini API

See differences here

The PoC is related to Vertex.

[the-gigi]

@sashirestela I discovered in my experiments with Gemini that sometimes it returns an invalid chatCompletion response where the content field is an array of strings instead of being just a string or null as specified here: https://platform.openai.com/docs/api-reference/chat/create.

This is a bug of Gemini and I'll see if I can reach out to them and get it fixed, but on the simple-openai/cleverclient side I think we can handle this by adding a response interceptor (similar to the request interceptor). Then, if a response needs to be massaged a little bit to comply with the OpenAI API then it can be done there and the core of simple-openai doesn't need to change to handle incompatible providers.

I noticed it only for tool calls so far (other chat competions return a single string correctly). In this case the array can be merged into a single string.

[the-gigi]

I reported the bug here:
https://ai.google.dev/gemini-api/docs/openai

[sashirestela]

@the-gigi I can't see your bug report, I just can see the documentation and a warning saying that the support to OpenAI is still in beta.

[the-gigi]

@sashirestela I used the "send feedback" form on the page. It is not visible there. I may add it to the developer forum too.

Yes, the API is in beta, but I think it's useful to support it in simple-openai even if some aspects are not implemented yet or broken as in this case. My suggestion for adding a response interceptor will allow fixing these category of problems for any provider.

[the-gigi]

@sashirestela I reported it in the developer forum too with a simple curl reproducer
https://discuss.ai.google.dev/t/openai-compatibility-bug-in-response-object/58484

[the-gigi]

@sashirestela what are your thoughts on adding response interceptor? also, should it be a separate issue? the need emerges from the Gemini Vertex OpenAI compatibility later, but it's a general-purpose feature that can be useful for other providers (or even OpenAI itself if you want to have a hook perform some global check or modification to all responses)

[sashirestela]

I think you could add a feature request in cleverclient for that.
However, you should push the fix on Gemini Vertex before to include it as part of simple-openai.

NEW ADDITION:
I'm afraid that response interceptor will be harder to implement than request interceptor, so it won't be available in the short term.

[sashirestela]

Have you checked if some request parameters can avoid that non-standard content?

If it happens with tools, it means that support for functions is invalid and we cannot deliver it in that state.

[sashirestela]

Just in case, have you checked if the other implementation, Gemini Google AI, has the same issue?

[the-gigi]

@sashirestela Function calling works for some requests. It doesn't work for the particular request that I reported on (which happens to be part of our integration tests). The open AI compatibility layer of Gemini API seems to be very buggy at the moment. I see multiple new bugs reported constantly in the developer forum. I didn't play with request parameters as this is a straight forward bug that violates the documented format of the response's message content.

If you don't want to add it to simple-openai in such state then let's wait. We're not going to use it anyway until it's more stable. Anyway, with the new approach for providers we can add it ourselves from the outside (this is how I'm running out integration tests against Gemini).

re: Gemini Vertex AI vs Gemini Google AI. I didn't try it.

[sashirestela]

@the-gigi I think it's not a question of wanting to, it would be a bit irresponsible to deliver it in its current state with errors. Let's put it on hold while it matures

[the-gigi]

@sashirestela sure. that makes sense. I'll keep my fork around and monitor the progress on Gemini API compatibility layer. I assume there is no point in opening a PR at this point.