Skip to content

Conversation

@mudler
Copy link
Owner

@mudler mudler commented Apr 18, 2025

Description

This pull request introduces changes to ensure consistent gRPC message size limits across various backends and client implementations. The updates set maximum message sizes to 50MB for sending, receiving, and overall message length. These changes improve the handling of large payloads and prevent errors caused by exceeding default gRPC limits.

Backend Updates

  • C++ Backend (grpc-server.cpp): Added SetMaxMessageSize, SetMaxSendMessageSize, and SetMaxReceiveMessageSize to set the maximum gRPC message size to 50MB.
  • Python Backends:
    • Updated serve functions in multiple files (e.g., autogptq, bark, coqui, diffusers, exllama2, faster-whisper, kokoro, rerankers, transformers, vllm) to include gRPC options for max_message_length, max_send_message_length, and max_receive_message_length, all set to 50MB. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]

Client Updates

  • Go Client (client.go):
    • Updated multiple methods (e.g., HealthCheck, Embeddings, Predict, LoadModel, GenerateImage, TTS, AudioTranscription, etc.) to include grpc.WithDefaultCallOptions for setting MaxCallRecvMsgSize and MaxCallSendMsgSize to 50MB. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

Signed-off-by: Ettore Di Giacinto <[email protected]>
@netlify
Copy link

netlify bot commented Apr 18, 2025

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 217c241
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/6802bcf662016700087bc417
😎 Deploy Preview https://deploy-preview-5212--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mudler mudler merged commit 8abecb4 into master Apr 19, 2025
25 checks passed
@mudler mudler deleted the chore/bump_grpc_limits branch April 19, 2025 06:53
@mudler mudler added the enhancement New feature or request label May 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants