From aea0d8b51baabf5992c19e344237b286e9187ff8 Mon Sep 17 00:00:00 2001
From: "codeflash-ai[bot]"
 <148906541+codeflash-ai[bot]@users.noreply.github.com>
Date: Thu, 4 Dec 2025 00:22:40 +0000
Subject: [PATCH] Optimize LRUCache.get
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The optimization achieves a **144% speedup** by replacing an expensive `try/except` pattern with a more efficient conditional check for cache lookups.

**Key optimizations applied:**

1. **Eliminated expensive `.pop()` operation**: The original code used `self.cache.pop(key)` followed by `self.cache[key] = value` to move the key to the end of the OrderedDict. This approach requires removing and re-inserting the key-value pair, which is costly for OrderedDict operations.

2. **Used `.move_to_end()` method**: The optimized version leverages OrderedDict's built-in `.move_to_end(key)` method, which efficiently updates the ordering without removing and re-inserting the entry.

3. **Replaced try/except with conditional check**: Changed from exception handling (`try/except KeyError`) to a simple membership test (`if key in self.cache`), avoiding the overhead of exception creation and handling when keys are missing.

**Performance impact analysis:**
- The line profiler shows the original code spent 58.4% of time in the `.pop()` operation alone
- The optimized version reduces total execution time from 14.462μs to 6.319μs
- Cache miss scenarios (testing non-existent keys) show particularly strong improvements: 148-204% faster according to the annotated tests

**Why this optimization works:**
In Python, exception handling has significant overhead, especially when exceptions are frequently raised. For cache implementations where cache misses are common, avoiding KeyError exceptions provides substantial performance benefits. Additionally, OrderedDict's `.move_to_end()` is specifically optimized for LRU cache patterns, making it much more efficient than manual pop/reassign operations.

This optimization is especially valuable for cache-heavy workloads where frequent lookups of missing keys occur, as evidenced by the strong performance gains in the "empty cache" and "missing key" test scenarios.
---
 inference/core/cache/lru_cache.py | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/inference/core/cache/lru_cache.py b/inference/core/cache/lru_cache.py
index 5f801f3451..67ffc3bf04 100644
--- a/inference/core/cache/lru_cache.py
+++ b/inference/core/cache/lru_cache.py
@@ -17,11 +17,10 @@ def enforce_size(self):
             self.cache.popitem(last=False)
 
     def get(self, key):
-        try:
-            value = self.cache.pop(key)
-            self.cache[key] = value
-            return value
-        except KeyError:
+        if key in self.cache:
+            self.cache.move_to_end(key)
+            return self.cache[key]
+        else:
             return None
 
     def set(self, key, value):