Issue: Memory Leak #2097
Description
The Issue:
- I'm having a memory leak going on for around 2 months (somewhere from August) in my system.
- Tried to run a profiler and found out that somehow langchain's tracing module is adding up in the memory, when i run around 200 langgraph agent executions or spans. I've attached the profiler's memory diff before I ran my agent, and after i ran 200 executions of it.
- For context, there hasn't been much changes in the time being, system was stable earlier, the leak started recently. The minor changes include:
- added some pydantic models to the langgraph's State dict. (which are serialized (
.model_dump(mode="json"), before sending to the LLM using model.ainvoke) - Can't remember any other major changes, if you require more detail, let me know, i'll dig a bit.
- added some pydantic models to the langgraph's State dict. (which are serialized (
- Solution -> And when I turned off Langchain tracing by setting
LANGCHAIN_TRACING_V2: false, the leak got fixed, now there's no leak, and the system has been working fine with constant600 MBof mem usage for more than 1 week. But still couldn't get the exact root cause of this?
Memory Increase plot (3 hr window):
Tracemalloc Profiler output (shows the top 20 memory allocations, that couldn't be freed, even after gc.collect()):
{
"message": "Compared snapshot 1 to 0",
"top_differences": [
{
"trace": "/usr/local/lib/python3.10/copy.py:76",
"size_diff": ""61.11033630371094 MB",
"count_diff": "116256 Mem Blocks",
"size": "61.11033630371094 MB",
"count": "261091 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/langsmith/client.py:248",
"size_diff": "17.27130889892578 MB",
"count_diff": "294 Mem Blocks",
"size": "38.66460990905762 MB",
"count": "693 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/json/decoder.py:353",
"size_diff": "13.877202987670898 MB",
"count_diff": "74366 Mem Blocks",
"size": "40.55583953857422 MB",
"count": "276523 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/copy.py:231",
"size_diff": "13.17572021484375 MB",
"count_diff": "60373 Mem Blocks",
"size": "29.326629638671875 MB",
"count": "134041 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/copy.py:228",
"size_diff": "8.26947021484375 MB",
"count_diff": "135487 Mem Blocks",
"size": "18.5465087890625 MB",
"count": "303866 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/langsmith/client.py:1445",
"size_diff": "8.016353607177734 MB",
"count_diff": "8 Mem Blocks",
"size": "18.036795616149902 MB",
"count": "18 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/pydantic/main.py:776",
"size_diff": "6.1854400634765625 MB",
"count_diff": "90082 Mem Blocks",
"size": "13.823204040527344 MB",
"count": "201315 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/copy.py:202",
"size_diff": "4.402397155761719 MB",
"count_diff": "82433 Mem Blocks",
"size": "9.854965209960938 MB",
"count": "184530 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/pydantic/main.py:193",
"size_diff": "3.5172882080078125 MB",
"count_diff": "27653 Mem Blocks",
"size": "9.106864929199219 MB",
"count": "71558 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/pydantic/v1/main.py:760",
"size_diff": "2.9032974243164062 MB",
"count_diff": "32021 Mem Blocks",
"size": "6.66303825378418 MB",
"count": "72789 Mem Blocks"
},
{
"trace": "/app/components/community_management/recommendation/new_comment_ai_moderation.py:203",
"size_diff": "2.843647003173828 MB",
"count_diff": "145 Mem Blocks",
"size": "6.412914276123047 MB",
"count": "327 Mem Blocks"
},
{
"trace": "/app/components/community_management/recommendation/new_comment_ai_moderation.py:515",
"size_diff": "2.4863243103027344 MB",
"count_diff": "145 Mem Blocks",
"size": "5.607089996337891 MB",
"count": "327 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/copy.py:206",
"size_diff": "2.3960800170898438 MB",
"count_diff": "55477 Mem Blocks",
"size": "5.366813659667969 MB",
"count": "124221 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/langchain_core/tracers/langchain.py:74",
"size_diff": "2.1421356201171875 MB",
"count_diff": "7826 Mem Blocks",
"size": "4.799964904785156 MB",
"count": "17581 Mem Blocks"
},
{
"trace": "/app/components/community_management/recommendation/rubric_evaluator.py:47",
"size_diff": "2.046140670776367 MB",
"count_diff": "145 Mem Blocks",
"size": "4.514639854431152 MB",
"count": "327 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/langsmith/client.py:1178",
"size_diff": "1.8699188232421875 MB",
"count_diff": "6396 Mem Blocks",
"size": "4.174720764160156 MB",
"count": "14256 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/langsmith/client.py:1535",
"size_diff": "1.8380126953125 MB",
"count_diff": "3346 Mem Blocks",
"size": "4.1275634765625 MB",
"count": "7514 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/langchain_core/callbacks/base.py:922",
"size_diff": "1.4249725341796875 MB",
"count_diff": "8894 Mem Blocks",
"size": "3.2484054565429688 MB",
"count": "20275 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/config.py:186",
"size_diff": "1.3290023803710938 MB",
"count_diff": "8295 Mem Blocks",
"size": "3.0285873413085938 MB",
"count": "18903 Mem Blocks"
},
{
"trace": "/usr/local/lib/python3.10/site-packages/langchain_core/callbacks/base.py:923",
"size_diff": "1.271484375 MB",
"count_diff": "7936 Mem Blocks",
"size": "2.8941650390625 MB",
"count": "18064 Mem Blocks"
}
]
}