[Seeking Feedback] Flux Resource Model Strawman

[dongahn]

I implemented a strawman to help me further design the upcoming flux resource comms. module -- a new service that selects the best-matching resources for each job. At this point, it will be extremely helpful if I can get some feedback from advanced scheduling researchers to make sure I will have a future-scheduling-proof design. The strawman code you can play with is at https://github.com/dongahn/resource_module_strawman

Using your feedback, I would like to determine:

• What are the resource representations suited well for various advanced scheduling you are working on? Answer, if we can use our primitives to model, traverse, and match on such representations.

• Programmability -- can scheduler writers easily express their algorithms using our graph match visitor patterns.

This is a strawman and so doesn't have a whole lot in it. So, do assume some of other core technologies as you evaluate this. In particular, please assume you can use planner on any level in each subsystem
and keep aggregate information about the vertex’s subtree in a scalable fashion. This is descried in here and here

@SteVwonder: Please look at this from our IO-Aware scheduling algorithms angle.

• Use PFS1 IO bandwidth subsystem as the dominant subsystem; and see if and how you would replicate your algorithm in our HPDC 16 paper. I configured a simple model as a starting point under
  --matcher=PFS1BA option (See below for the graph rendered for a mini scale system)

• Use the containment subsystem as your dominant subsystem and PFS1 IO bandwidth subsystem as an auxiliary, and see if you can implement a similar IO Aware algorithm. I configured a simple model as a starting point under --matcher=C+PFS1BA option (See below for the graph rendered for mini scale)

• In both cases, I suspect that you might need a bit different graph representations. Please let me know if you want to / need to have the graph model refined for your algorithms

@tpatki: Please look at this from a Power-Aware scheduling algorithm angle

• Use the power distribution subsystem as the dominant subsystem; and see if and how you would implement power-aware scheduling algorithms. I configured a simple model as a starting point under
  --matcher=PA option (See below for the graph rendered for mini scale)

• Use the containment subsystem as the dominant subsystem and power distribution subsystem as an auxiliary subsystem; and see if and how you can implement effective algorithms. I configured a simple model as a starting point under --matcher=C+PA option (See below for the graph rendered for mini scale)

• In both cases, I suspect you might need a bit different graph models. Please let me know what are the models that are most suitable for your algorithms. I happened to talk with an IBM power-scheduling researcher at my CORAL face to face meeting, and he believes that future compute nodes may need to be grouped together based on their power efficiency levels. I do believe this can be easily done by refining power hierarchy model. If you can suggest some models based on your expertise, this would be very helpful at this point.

Nikil (I don't have his github id): Please look at this from the standpoint of your IB-network-connection-aware scheduling algorithm:

• Use the IB connection subsystem as the dominant subsystem; and see if an dhow you would implement pod-based scheduling algorithms. I configured a simple model as a starting point under --matcher=IBA option (See below for the graph rendered for mini scale)

• Use the containment subsystem as the dominant subsystem and the IB connection subsystem as an auxiliary; see if you can implement these algorithms as well. I configured a simple model as a starting point under --matcher=C+IBA option (See below for the graph rendered for mini scale)

• In your case, I highly suspect we would want to refine the network subsystem graph model. This opens up a good test for me because I want to see how flexible our new flux model strawman implementation allows a scheduler plugin writer like you can do this.

@lipari and @morrone: any early feedback from you two should also be high useful, of course!

[dongahn]

For @SteVwonder:

--matcher=PFS1BA option

--matcher=C+PFS1BA

[dongahn]

For @tpatki:

--matcher=PA

--matcher=C+PA

[dongahn]

For Nikil:

--matcher=IBA

--matcher=C+IBA

[dongahn]

Oh, forgot to mension. The strawman currently doesn't build on TOSS2 because their compilers are so ancient. Please use a TOSS3 system or your laptop.

[springme]

Thanks for working on flux on the week end, Dong. I trust that you will all spend tomorrow with your kids if you have them. Happy Father's Day to all of the dads! Becky

…

Sent from my iPhone On Jun 17, 2017, at 6:22 PM, Dong H. Ahn <notifications@github.com<mailto:notifications@github.com>> wrote: For @SteVwonder<https://github.com/stevwonder>: [pfs1ba mini]<https://user-images.githubusercontent.com/862123/27257387-77d821dc-5389-11e7-931a-ccd844353fd7.png>

--matcher=PFS1BA option [c pfs1ba mini]<https://user-images.githubusercontent.com/862123/27257392-9390e95e-5389-11e7-9758-e05dc19b2f5f.png>

--matcher=C+PFS1BA — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#247 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEdU0EYB0nmLn9ZFc39fWth6LPBIKHD9ks5sFHuxgaJpZM4N9X8v>.

[tpatki]

Thanks, Dong, I’ll take a look at this shortly. Thanks for putting this together over the weekend! Sincerely, T From: Becky Springmeyer <notifications@github.com> Reply-To: flux-framework/flux-sched <reply@reply.github.com> Date: Saturday, June 17, 2017 at 8:22 PM To: flux-framework/flux-sched <flux-sched@noreply.github.com> Cc: "Patki, Tapasya" <patki1@llnl.gov>, Mention <mention@noreply.github.com> Subject: Re: [flux-framework/flux-sched] [Seeking Feedback] Flux Resource Model Strawman (#247) Thanks for working on flux on the week end, Dong. I trust that you will all spend tomorrow with your kids if you have them. Happy Father's Day to all of the dads! Becky

…

Sent from my iPhone On Jun 17, 2017, at 6:22 PM, Dong H. Ahn <notifications@github.com<mailto:notifications@github.com>> wrote: For @SteVwonder<https://github.com/stevwonder>: [pfs1ba mini]<https://user-images.githubusercontent.com/862123/27257387-77d821dc-5389-11e7-931a-ccd844353fd7.png>

--matcher=PFS1BA option [c pfs1ba mini]<https://user-images.githubusercontent.com/862123/27257392-9390e95e-5389-11e7-9758-e05dc19b2f5f.png>

--matcher=C+PFS1BA — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#247 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEdU0EYB0nmLn9ZFc39fWth6LPBIKHD9ks5sFHuxgaJpZM4N9X8v>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#247 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABhgTlir3jydTZL7478ayMulqxVyA9a6ks5sFJgEgaJpZM4N9X8v>.

[SteVwonder]

@dongahn nice work!

Poking around a bit, it seems that the C+PFS1BA works exactly as I expected: it performs a depth-first search through the dominant hierarchy (containment) and then when it hits the auxiliary hierarchy (IO) it performs an upwalk. The PFS1BA matcher isn't working as I expected. I expected it to perform a depth first search through the dominant hierarchy (IO) (e.g. pfsbw->lnetbw->coreswitchbw->leafswitchbw->nicbw->node), but it seems to only visit the root vertex:

→ ./resource --matcher=PFS1BA
[INFO] Load the matcher ...
----dom_discover_vtx: pfs1bw
----dom_finish_vtx: pfs1bw
*********************************************************
* Elapse time 0.000062
*   Start Time: 1497908308.232591
*   End Time: 1497908308.232653
*********************************************************

I am not sure what is going on here. Any thoughts?

In the meantime, I am working on a matcher that works for the C+PFS1BA case.

EDIT: does it maybe have to do with the directionality of the links between the resources? Maybe when IO is dominant, it should no longer be a "flowsup" relationship but a relationship with the reverse directionality.
EDIT2: nevermind, I think I figured it out. The call to add_subsystem should use flows_down not flows_up.

[dongahn]

Ah. Thank you for looking at this so quickly. I will take a look shortly.

[dongahn]

EDIT2: nevermind, I think I figured it out. The call to add_subsystem should use flows_down not flows_up.

Great.

Your schedulers can also choose all relationships for both cases with:

        if ( !rc && (rc = subsystem_exist (ctx, "PFS1BA")) == 0)
            matcher.add_subsystem ("pfs1bw", "*");

For the depth first search on the dominant subsystem, I color the vertex so that I won't traverse further whenever detecting a back edge.

[dongahn]

Also FYI, future work will include providing richer idioms for scheduler to use to select edge types within each target subsystem. Probably, string pattern matching fucntions for inclusion and exclusion.

[dongahn]

@SteVwonder: I also created coloring_for_upwalk dev-branch in the strawman and pushed my changes. As I was adding support for graphml, I found that I was actually adding parallel edges! So I fixed that and also added coloring support for up-walks -- with this the DFU traverser can detect loops during the up-walk as well. This led to slight changes to matcher callback name.

[dongahn]

@nikhil-jain: So that this this tracking ticket will make into your email. @SteVwonder has been looking at this strawman as well as been involved in our recent scheduling discussions. @SteVwonder should be a good resource for you.

[dongahn]

As @SteVwonder, @morrone and I are discussing and debating the capabilities and limitations of the resource strawman, it seems the classes of matching problems that the current strawman+planner can and can’t solve start to emerge. So, I thought I should document them here.

First, there are at least two main classes of advanced matching problems our current scheme can solve on a single dominant hierarchy beyond simple flat matching.

• Hierarchical Exact matching, for example,

  1. If the scheduler choses the containment hierarchy as dominant: e.g., cluster[1]->rack[2]->node[4]
  2. If the scheduler choses the IB network connection hierarchy as dominant: e.g., edge_switch[2]->node[4]

• Hierarchical Aware matching: for example, choose node resources in a way to maximize or minimize certain hierarchical quantities

  1. Choose node[4] such a way that we minimize the number of edge switches to use given the current resource state (using the IB network connection hierarchy as dominant along with planners embedded at various levels)
  2. Choose node[4] such a way that we spread them across as many PDUs as possible given the current state (using the power hierarchy as the dominant hierarchy and its planners)

There are also some classes of multi-hierarchy matching problems that the DFU+planner can solve exactly. That is, when we arrange certain hierarchies such that one becomes dominant and the rest become an ordered set of auxiliary.

• Hierarchical Exact on dom hierarchy and Flow Exact matching on aux hierarchies: for example,

  1. When the per-node flow resource quantifies are given along with exact hierarchical request on containment hierarchy, e.g., cluster[1]->rack[2]->node[4], each node of 8 nodes requires at least 20MB/s or 100 Watts. The Flow Exact constraint can be checked during each up-walk.

• Hierarchical Exact on dom hierarchy and Hierarchical Aware matching on aux: for example,

  1. Within the hard hierarchical constraint on the dominant hierarchy, you want to minimize or maximize certain quantities on one or more auxiliary hierarchies: "within rack[3]->nodes[18] hierarchical constraint - we can select those nodes such a way to use the edges switches that have highest or lowest number of available nodes underneath them first.”

Now, we also realized that there are certain classes of matching problems that DFU+planner alone cannot solve exactly. Most notably: Hierarchical Exact matching over multiple hierarchies: e.g., cluster[1]->rack[2]->node[18]<-edge_switch[2].

So, If I summarize this into a capability table of DFU+planner:

Dom\Aux	None	Hierarchical Exact	Hierarchical Aware	Flow Exact
Hierarchical Exact	Yes	No	Yes	Yes
Hierarchical Aware	Yes	NA	Yes	Yes
Flow Exact	Yes	NA	Yes	Yes

For the Multi Hierarchical Exact column, I have to believe that this can be solved by introducing a bit more advanced traversal type (which I call Loop-enabled DFU) and a variant of planner (which I call up-aggregator). The idea is to perform DFU on some of the subtrees "twice" to find the exact matches, the first iteration to generate some high-level information on aux hierarchies and the second to find the exact match. E.g., at a rack level, the first DFU deposits the swich info of the current loop iteration to the root of the IB connection aux hierarchy and the second iteration, we use that as high level switch locality info for the actual selection.

The loop traversal concept would be similar to: https://github.com/tinkerpop/gremlin/wiki/Loop-Pattern.

Of course, it will be less likely we will do anything for more than simple DFU as our short-term directions but I thought it would be good to flesh out details on what can and can't.

[dongahn]

In case Depth First on dominant hierarchy and Up on auxiliary hierarchies (DFU) and envisioned match scoring schemes are not clear enough:

Also, for Planner

[lipari]

Thank you @dongahn for documenting your discussions. It provides a concise synopsis of the frontiers of the resource selection thinking. At this point, I don't have much to add. I know you are a proponent for a one-pass traversal solution to maximize performance. So I will review the loop traversal doc you provided to understand what it has to offer.