-
Notifications
You must be signed in to change notification settings - Fork 19.5k
Open
Labels
feature requestrequest for an enhancement / additional functionalityrequest for an enhancement / additional functionality
Description
Checked other resources
- This is a feature request, not a bug report or usage question.
- I added a clear and descriptive title that summarizes the feature request.
- I used the GitHub search to find a similar feature request and didn't find it.
- I checked the LangChain documentation and API reference to see if this feature already exists.
- This is not related to the langchain-community package.
Feature Description
Claude support Prompt-Caching feature, which can optimizes api usage and reduces processing time.
AnthropicPromptCachingMiddleware can perfectly support this feature。
However,there are some other claude model provider,such as openrouter. Which can only use ChatOpenAI client.
Use Case
I want to add an ChatOpenAIPromptCachingMiddleware,just like AnthropicPromptCachingMiddleware.
Proposed Solution
First, add an ChatOpenAIPromptCachingMiddleware, all code is same.
Second, modify ChatOpenAI to support "cache_control" in kwargs.
Code may be like this. I have tested in my AI agent.
def _get_request_payload(
self,
input_: LanguageModelInput,
*,
stop: list[str] | None = None,
**kwargs: Any,
) -> dict:
wrap_text = lambda text :{"type":"text","text":text} if isinstance(text,str) else {**text}
if "cache_control" in payload:
dup_message = {** payload["messages"][-1]}
last_message_content = dup_message["content"]
if isinstance(last_message_content,str):
dup_message["content"]=[wrap_text(last_message_content)]
elif isinstance(last_message_content,list):
dup_message["content"] = [wrap_text(content) for content in last_message_content]
dup_message["content"][-1]["cache_control"] = payload.pop("cache_control")
payload["messages"][-1] = dup_message
return payload
Alternatives Considered
No response
Additional Context
No response
Metadata
Metadata
Assignees
Labels
feature requestrequest for an enhancement / additional functionalityrequest for an enhancement / additional functionality