Skip to content

Increase the max threads from 32 to 256#5889

Merged
oobabooga merged 1 commit intooobabooga:devfrom
chraac:dev-max-threads
May 19, 2024
Merged

Increase the max threads from 32 to 256#5889
oobabooga merged 1 commit intooobabooga:devfrom
chraac:dev-max-threads

Conversation

@chraac
Copy link
Copy Markdown
Contributor

@chraac chraac commented Apr 20, 2024 •
edited
Loading

In #5292, we've requested to increase the threads_batch in ui to allow > 32 threads, and here's the PR, to enlarge it.

Minor changes, tested on my rig with llama.cpp loader, work as expected.
947559cc1db69b66a2950aefcb6f8a9

Checklist:

chraac
chraac commented Apr 20, 2024
View reviewed changes
@chraac chraac changed the base branch from main to dev April 20, 2024 03:22
@Ph0rk0z
Copy link
Copy Markdown
Contributor

Ph0rk0z commented Apr 20, 2024

I think sometimes using more threads than physical cores is desirable. For some intel CPUs. Depends on your setup.

@chraac
Copy link
Copy Markdown
Contributor Author

chraac commented Apr 21, 2024

I think sometimes using more threads than physical cores is desirable. For some intel CPUs. Depends on your setup.

hmm, then I'll leave the maximum to 256, let the user decide how many threads they want to use

@chraac chraac mentioned this pull request Apr 22, 2024
Closed
@chraac
Copy link
Copy Markdown
Contributor Author

chraac commented Apr 25, 2024 •
edited
Loading

@Ph0rk0z do you mind to merge this PR regarding its a trival change, verified on my machine, and also the linter show no errors.

@chraac chraac force-pushed the dev-max-threads branch 2 times, most recently from 0d30efe to 8d6bfe6 Compare April 25, 2024 02:39
@Ph0rk0z
Copy link
Copy Markdown
Contributor

Ph0rk0z commented Apr 26, 2024

Heh, I can't merge. Its on oobabooga.

@chraac
Copy link
Copy Markdown
Contributor Author

chraac commented Apr 27, 2024

Heh, I can't merge. Its on oobabooga.

Ahh, got it, anyway, thanks for the comments

@chraac chraac force-pushed the dev-max-threads branch 2 times, most recently from 5c71395 to fa2dfbc Compare May 4, 2024 06:45
@chraac chraac force-pushed the dev-max-threads branch from fa2dfbc to 1fc668f Compare May 12, 2024 03:13
@chraac chraac force-pushed the dev-max-threads branch from 1fc668f to fee7122 Compare May 15, 2024 10:37
@suspicious-pineapple
Copy link
Copy Markdown

suspicious-pineapple commented May 15, 2024

I'd also like to see this merged

@chraac
Copy link
Copy Markdown
Contributor Author

chraac commented May 18, 2024

@oobabooga , sorry for bothering you, could you please have a look on this PR when you have a monent? I'm open to any feedback you might have and am ready to make any necessary adjustments.. Thanks!

@oobabooga
Copy link
Copy Markdown
Owner

oobabooga commented May 19, 2024

Thanks for the PR -- it's a fair change, I just didn't have the time to review it before. Merging now.

@oobabooga oobabooga merged commit 6b546a2 into oobabooga:dev May 19, 2024
@chraac chraac deleted the dev-max-threads branch May 20, 2024 02:42
anon-contributor-0 pushed a commit to anon-contributor-0/text-generation-webui that referenced this pull request May 30, 2024
@chraac @anon-contributor-0
llama.cpp: increase the max threads from 32 to 256 (oobabooga#5889)
a477657
PoetOnTheRun pushed a commit to PoetOnTheRun/text-generation-webui that referenced this pull request Oct 22, 2024
@chraac @PoetOnTheRun
llama.cpp: increase the max threads from 32 to 256 (oobabooga#5889)
70897df
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@chraac @Ph0rk0z @suspicious-pineapple @oobabooga