add token estimation to token counting for truncation#3286

timothycarambat wants to merge 2 commits intomasterMintplex-Labs/anything-llm:masterfrom

3280-token-counting-server-side-truncation-improvementsMintplex-Labs/anything-llm:3280-token-counting-server-side-truncation-improvementsCopy head branch name to clipboard