Skip to content

Conversation

@mudler
Copy link
Owner

@mudler mudler commented Feb 13, 2025

Description

This pull request includes changes to the backend/cpp/llama/grpc-server.cpp file to handle context capacity and prevent redundant checks. The most important changes include adding a check for context capacity and improving logging when context is exhausted.

Context capacity handling:

  • Added a check to set truncated and stopped_limit flags and log a message when the context capacity is reached. (backend/cpp/llama/grpc-server.cpp)

Redundant check removal:

  • Removed redundant checks and improved logging when context is exhausted by releasing the slot and logging an error message. (backend/cpp/llama/grpc-server.cpp)

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

@netlify
Copy link

netlify bot commented Feb 13, 2025

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 652d4b6
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/67af310860aaa80008ea81f9
😎 Deploy Preview https://deploy-preview-4820--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mudler mudler force-pushed the feat/improve_context_shift branch from bd0c800 to 652d4b6 Compare February 14, 2025 12:03
@mudler mudler added the bug Something isn't working label Feb 14, 2025
@mudler mudler merged commit 9e32fda into master Feb 14, 2025
25 checks passed
@mudler mudler deleted the feat/improve_context_shift branch February 14, 2025 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants