Skip to content

Support cache_control for ChatOpenAI #33757

@jingyugao

Description

@jingyugao

Checked other resources

  • This is a feature request, not a bug report or usage question.
  • I added a clear and descriptive title that summarizes the feature request.
  • I used the GitHub search to find a similar feature request and didn't find it.
  • I checked the LangChain documentation and API reference to see if this feature already exists.
  • This is not related to the langchain-community package.

Feature Description

Claude support Prompt-Caching feature, which can optimizes api usage and reduces processing time.
AnthropicPromptCachingMiddleware can perfectly support this feature。
However,there are some other claude model provider,such as openrouter. Which can only use ChatOpenAI client.

Use Case

I want to add an ChatOpenAIPromptCachingMiddleware,just like AnthropicPromptCachingMiddleware.

Proposed Solution

First, add an ChatOpenAIPromptCachingMiddleware, all code is same.
Second, modify ChatOpenAI to support "cache_control" in kwargs.
Code may be like this. I have tested in my AI agent.

    def _get_request_payload(
        self,
        input_: LanguageModelInput,
        *,
        stop: list[str] | None = None,
        **kwargs: Any,
    ) -> dict:
        wrap_text = lambda text :{"type":"text","text":text} if isinstance(text,str) else {**text}
        if "cache_control" in payload:
            dup_message = {** payload["messages"][-1]}
            last_message_content = dup_message["content"]
            if isinstance(last_message_content,str):
                dup_message["content"]=[wrap_text(last_message_content)]
            elif isinstance(last_message_content,list):
                dup_message["content"] = [wrap_text(content) for content in last_message_content]
            dup_message["content"][-1]["cache_control"] = payload.pop("cache_control")
            payload["messages"][-1] = dup_message
        return payload

Alternatives Considered

No response

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature requestrequest for an enhancement / additional functionality

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions