[SPARK] [Scheduler] Configurable default scheduling mode #19456
Conversation
|
Could you elaborate on the scenario that you should need to make these settings configurable? |
|
I have a multiuser application where I use the userId as the name of the scheduling pool so that users are balanced equally by spark and within a user's workload I can set the scheduling mode to FAIR (or whatever I want). It is unreasonable to specify xml for each user and restart spark, so this patch allows me to specify the default configuration used by undefined pools. |
varunvish910
left a comment
varunvish910
left a comment
There was a problem hiding this comment.
Can you guarantee that your users share an even workload? With this setup you could potentially have a user with a high workload using 1/#users resources. Wouldn't using the queue feature be better?
|
Can one of the admins verify this patch? |
|
ping @blyncsy-david-lewis for @varuvish 's comment. Otherwise let me propose to close it for now. |
Closes apache#17422 Closes apache#17619 Closes apache#18034 Closes apache#18229 Closes apache#18268 Closes apache#17973 Closes apache#18125 Closes apache#18918 Closes apache#19274 Closes apache#19456 Closes apache#19510 Closes apache#19420 Closes apache#20090 Closes apache#20177 Closes apache#20304 Closes apache#20319 Closes apache#20543 Closes apache#20437 Closes apache#21261 Closes apache#21726 Closes apache#14653 Closes apache#13143 Closes apache#17894 Closes apache#19758 Closes apache#12951 Closes apache#17092 Closes apache#21240 Closes apache#16910 Closes apache#12904 Closes apache#21731 Closes apache#21095 Added: Closes apache#19233 Closes apache#20100 Closes apache#21453 Closes apache#21455 Closes apache#18477 Added: Closes apache#21812 Closes apache#21787 Author: hyukjinkwon <[email protected]> Closes apache#21781 from HyukjinKwon/closing-prs.
5 participants
Pulling default values for scheduling mode from spark conf.