Skip to content

add token estimation to token counting for truncation#3286

Draft
timothycarambat wants to merge 2 commits intomasterfrom
3280-token-counting-server-side-truncation-improvements
Draft

add token estimation to token counting for truncation#3286
timothycarambat wants to merge 2 commits intomasterfrom
3280-token-counting-server-side-truncation-improvements

Conversation

@timothycarambat
Copy link
Member

Pull Request Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • 📝 docs

Relevant Issues

resolves #3280

What is in this change?

  • Removes cannonball truncation in favor of end-truncation methods
  • Simplifies counting and tokenization of very large inputs by token estimate to bypass performance issues on low-end devices.

Additional Information

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: pinned documents that have ~200k tokens (~0.5MB .txt file) breaking AnythingLLM Docker version

1 participant