Skip to content

DRY sampler improvements#6053

Merged
oobabooga merged 7 commits intooobabooga:devfrom
belladoreai:dev-dry-optimization2
Jun 13, 2024
Merged

DRY sampler improvements#6053
oobabooga merged 7 commits intooobabooga:devfrom
belladoreai:dev-dry-optimization2

Conversation

@belladoreai
Copy link
Copy Markdown
Contributor

@belladoreai belladoreai commented May 25, 2024 •
edited
Loading

I was asked by @p-e-w to split some of the changes from #6047 into a separate PR here.

This PR contains the data type performance improvement for DRY, and a minor fix to prevent crash on large repetitive inputs.

Edit: now also contains change to cap the max match length to 50

See the main PR for more info.

Checklist:

@belladoreai belladoreai mentioned this pull request May 25, 2024
Closed
1 task
p-e-w
p-e-w reviewed May 26, 2024
View reviewed changes
@belladoreai belladoreai requested a review from p-e-w May 27, 2024 13:38
p-e-w
p-e-w approved these changes May 28, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@p-e-w p-e-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM modulo style nits.

@belladoreai belladoreai requested a review from p-e-w May 28, 2024 09:39
@belladoreai belladoreai mentioned this pull request May 30, 2024
Merged
3 tasks
@belladoreai
Copy link
Copy Markdown
Contributor Author

belladoreai commented May 30, 2024

@oobabooga Ready for merge

p-e-w
p-e-w approved these changes May 31, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@p-e-w p-e-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now! Probably makes sense to merge this before DRY is merged into master.

@Hunterius8
Copy link
Copy Markdown

Hunterius8 commented May 31, 2024

Quickly compared this version to the previous two, looks like it has still gotten a little faster.

drycomparison2

The tokens per second only decrease by ~5% at a range of about 21500 tokens now, compared to the previous version's ~11%.

@belladoreai belladoreai mentioned this pull request Jun 3, 2024
Closed
@jojje
Copy link
Copy Markdown

jojje commented Jun 3, 2024

LGTM.

Seems @p-e-w is also good with it. What's your view @oobabooga ?

It is an improvement over the original. If further optimization turns out to be necessary, such can be done in separate PRs.

@Vhallo
Copy link
Copy Markdown
Contributor

Vhallo commented Jun 10, 2024

Good to see the performance issues being solved. Seems like it might be worthwhile to integrate this into Exllamav2 / TabbyAPI now as well?

@Vhallo Vhallo mentioned this pull request Jun 10, 2024
Closed
@oobabooga
Copy link
Copy Markdown
Owner

oobabooga commented Jun 13, 2024

Thanks for the reviews, merging now before merging DRY to the main branch.

@oobabooga oobabooga merged commit 3abafee into oobabooga:dev Jun 13, 2024
PoetOnTheRun pushed a commit to PoetOnTheRun/text-generation-webui that referenced this pull request Oct 22, 2024
@belladoreai @PoetOnTheRun
DRY sampler improvements (oobabooga#6053)
834e2e9
gitzaidi referenced this pull request in aphrodite-engine/aphrodite-engine Dec 1, 2024
@selalipop @AlpinDale
sampler: add output_tokens to DRY sampler (#849)
14ac216 
* Add Debug Statements

* Test Token Fix

* Remove Debug Statements

* perform concat after checking for 0 multipliers

---------

Co-authored-by: AlpinDale <alpindale@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@p-e-w p-e-w p-e-w approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@belladoreai @Hunterius8 @jojje @Vhallo @oobabooga @p-e-w