coll: CSEL redesign #7547
Draft
coll: CSEL redesign #7547
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hzhou
force-pushed
the
2508_csel
branch
3 times, most recently
from
23d90b4 to
98abcd7
Compare
hzhou
force-pushed
the
2508_csel
branch
15 times, most recently
from
7018ca6 to
502260c
Compare
Warning under -Wall.
We will use a single-level JSON for algorithm selection including device-specific algorithms. Remove the collective ADI for now. We'll add the mechanism of selecting device-level algorithms later. gen_coll.py is updated to skip calling MPID_ collectives. Device collective CVARs are removed.
Do not hide the script. Move it to maint/ as the reset of the autogen scripts.
We will add the mechanism of selecting device-layer algorithms later.
Temporarily comment out the composition code that calls netmod/shm collectives since we will remove these apis next. Some NULL composition functions are removed.
We will replace the device-algorithm selelction later at MPIR-layer.
ipc_p2p.h references MPIDI_POSIX_am_eager_limit, which is defined in shm_am.h. Not sure how did it work before.
The fallback collectives (e.g. MPIR_Bcast_fallback) are manual "auto" functions that may not be the best algorithms for the system, but are sufficient for internal usages during init and object constricutions.
Use MPIR_Barrier_fallback instead of MPIR_Bcast_allcomm_auto (doesn't exist now).
The auto selection should take care of restrictions. Error rather than fallback. If user use CVAR to select specific algorithm, we should check restrictions before jumping the the algorithm. We will design a common fallback handling there.
Eventually we will make nonblocking compositional algorithms work by having the JSON tree check the sched framework types. For now, remove the json search and just use fallbacks.
For consistency, call MPIR collectives (e.g. MPIR_Bcast) in compositional algorithms. TODO: rewrite compositional algorithms using coll_sig and container.
Rename coll_info to coll_sig for type MPIR_Csel_coll_sig_s. This is the convention other than in csel.c. Let's make it consistent.
Pass coll_sig by pointer rather than by copy. The structure can be big and pass by pointer avoids the extra copy. Also allows coll_sig to serve as a persistent state throughout the collective selelction chain.
MPIR_Csel_coll_sig_s contains input arguments to an MPI collective. MPII_Csel_container_s contains extra parameters to an MPI algorithm. We define a unified collective algo function interface using both MPIR_Csel_coll_sig_s and MPII_Csel_container_s. Both structures will need MPID extensions to support MPID-specific algorithms. Defining both in the same header for easier management.
Two main JSON tuning files. "coll_composition.json" selects compositional algorithms, and "coll_selection.json" selects basic algorithms. Basic algorithm does not call another collectives.
Add the two auto functions that executes CSEL search.
A universal nb alglorithm for blocking collectives.
They are replaced by MPIR_Coll_nb.
Define an abstract collective algorithm function interface that uses MPII_Csel_container_s and MPIR_Csel_coll_sig_s. Both structure will have mechanism for device layer to extend with its own fields. All collective algorithms will be populated in a global MPIR_Coll_algo_table. Device layer can fillin its device-specific entries in MPID_Init. MPIR_Coll_auto and MPIR_Coll_composition_auto serve as auto collective functions that runs Csel search then call the selected algorithm by looking up the entries from MPIR_Coll_algo_table.
Dump a wrapper function for each algorithm that takes (cont, coll_sig). Separately Declare algorithm prototypes. Separately Decleare sched_auto prototypes.
Generate collective implement functions that assemble coll_sig and call MPIR_Coll_composition_auto.
Current compositional algorithms call MPIR collectives. We will refactor them later. But for now, generate a wrapper MPIR functions that calls _impl functions.
Move MPIR_Csel_coll_sig_s and MPII_Csel_container_s definitions to mpir_coll.h since they are now common interface to all collective algorithms. Move the rest of the csel header to coll_csel.h and only include it where needed.
We can add separate caching mechanism to expedite search later. For now, simplify by directly use csel_node_s.
Generate those IDs, table entries, and json parsing from coll_algorithms.txt.
We can easily create alias algorithms by defining a separate algorithm function that calls the generic routines. Thus, simplify the design by removing the alias feature in coll_algorithms.txt. This ensures a one-to-one entry for each collective algorithms with a matching algorithm function. Add iallreduce tsp_recexch algorithm since the function is used in multiple places. Similarly, add ibcast tsp_scatterv_allgatherv algorithm since it is used elsewhere internally. Remove enums such as IREDUCE_RECEXCH_TYPE_DISTANCE_DOUBLING/HALVING. The actual parameter is more like a boolean.
Replaced by MPIR_Csel_composition and MPIR_Csel_selection.
Add MPIR_init_coll_sig and MPID_init_coll_sig so we can add arbitrary attr bits or additional fields without hacking maint/gen_coll.py.
Generate tables based on coll_algorithms.txt and use the tables to facilitate csel parsing and error reporting. If user sets an algorithm CVAR, directly construct a container for the cvar-specified algorithm and call it if all restrictions are met. All restrictive checkers are represented by either a bit in coll_sig->attr or a boolean checker function. All restrictions and their checkers are configured in coll_algorithms.txt.
Contributor
Author
|
test:mpich/custom |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Description
WIP
[skip warnings]
Discussion
Reference: #7544
Author Checklist
Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
Commits are self-contained and do not do two things at once.
Commit message is of the form:
module: short descriptionCommit message explains what's in the commit.
Whitespace checker. Warnings test. Additional tests via comments.
For non-Argonne authors, check contribution agreement.
If necessary, request an explicit comment from your companies PR approval manager.