Feature Proposal: Configurable LLM Client Timeouts

[hellovai]

Description:

Currently, our LLM clients (e.g., Gemini Flash) sometimes experience excessive delays or errors (e.g., "stream closed because of a broken pipe") due to unbounded waiting times. This has led to multiple user reports and requests to configure timeouts for the client request lifecycle.

We propose adding configurable timeout settings directly in BAML so that developers can define:

• Connection Timeout: Maximum time to establish a connection.
• Response Timeout: Maximum time to wait for the first response or header.
• Total Timeout: Overall limit for the request lifecycle before the request is aborted.

Proposed Design Ideas:

1. Extend Client Options Syntax:

   • Allow users to specify timeout values in the options block for an LLM client. For example:

     client<llm> MyClient {
       provider "openai"
       options {
         model "gpt-4o"
         connection_timeout 4   // seconds to establish the connection
         response_timeout 4     // seconds to wait for the first byte/response
         total_timeout 4        // overall request limit
       }
     }
     

2. Integration with Fallbacks/Retry Policy:

• would only have a total_timeout option for the entire fallback
• if a client within the fallback failed due to timeout, then it would retry

3. Expose a new BamlTimeoutError which derives off of BamlClientError to expose the client that failed.

Notes, this may cause a stream to be interrupted in the middle.

Other thoughts, do we expose the concept of a time unit into baml? ie.

Does 1s translate to an implicit Duration type? What is the interface for this in other languages?

[linear]

BAML-267 Feature Proposal: Configurable LLM Client Timeouts

[hellovai]

Inspired from this thread: https://discord.com/channels/1119368998161752075/1351607012194254960

[vgatcg]

Hitting this too, I notice easy repro with long context windows where it can take a long time to upload (no compression on the API) or Gemini takes a long time to respond.

[hellovai]

@codearena-bot do this issue. change the design to:

client<llm> MyClient {
  provider "openai"
  options {
    model "gpt-4o"
    timeout {
       connection 4s   // seconds to establish the connection
       response 4s     // seconds to wait for the first byte/response
       total 4s        // overall request limit
    }
  }
}

and also the fallback and retries should have: timeout.total and timeout.each inside of timeouts map.

[codearena-bot]

I queued a fresh run at https://codearena.com/41be8355-b38a-4d0a-927e-750fc9886958! I will comment again once the run is complete.

[codearena-bot]

The agents have finished running & changes are ready to be viewed!