Make evicted_rows a UVA buffer#3079
Closed
sryap wants to merge 2 commits into
Closed
Conversation
✅ Deploy Preview for pytorch-fbgemm-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D62114877 |
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D62114877 |
sryap
force-pushed
the
export-D62114877
branch
from
a32cd8a to
4e1aefd
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D62114877 |
sryap
added a commit
to sryap/FBGEMM
that referenced
this pull request
Sep 4, 2024
Summary: Pull Request resolved: pytorch#3079 X-link: facebookresearch/FBGEMM#173 Prior to this diff, SSD-TBE used a combination of a pinned CPU buffer and the GPU buffer for `evicted_rows` (the buffer for staging rows that are evicted from L1 cache). It explicitly performed asynchronous memory copy (via `cudaMemcpyAsync`) to transfer `evicted_rows` from device to host. Since the number of evicted rows is known only on the device, SSD-TBE overallocated the `evicted_rows` CPU and GPU buffers. Therefore, it transferred extra data during the device-host memory copy. Such the extra data could be large and could make the memory copy a bottleneck of an execution. This diff mitigates the problem mentioned above by using a unified address buffer for `evicted_rows` and using a kernel (namely `masked_index_select` to load/store data instead of using a CUDA memory copy operation. This mechanism can avoid the extra memory copy. However, the memory copy can be less efficient (might not be able to fully saturate the available memory bandwidth) since it does not use the copy engine. Moreover, since it uses SMs for memory copy, when overlapping the operator with other computes, it can potentially compete for the SM resources with others. Differential Revision: D62114877
sryap
force-pushed
the
export-D62114877
branch
from
4e1aefd to
67c04d4
Compare
Differential Revision: D62190777
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D62114877 |
sryap
added a commit
to sryap/FBGEMM
that referenced
this pull request
Sep 5, 2024
Summary: Pull Request resolved: pytorch#3079 X-link: facebookresearch/FBGEMM#173 Prior to this diff, SSD-TBE used a combination of a pinned CPU buffer and the GPU buffer for `evicted_rows` (the buffer for staging rows that are evicted from L1 cache). It explicitly performed asynchronous memory copy (via `cudaMemcpyAsync`) to transfer `evicted_rows` from device to host. Since the number of evicted rows is known only on the device, SSD-TBE overallocated the `evicted_rows` CPU and GPU buffers. Therefore, it transferred extra data during the device-host memory copy. Such the extra data could be large and could make the memory copy a bottleneck of an execution. This diff mitigates the problem mentioned above by using a unified address buffer for `evicted_rows` and using a kernel (namely `masked_index_select` to load/store data instead of using a CUDA memory copy operation. This mechanism can avoid the extra memory copy. However, the memory copy can be less efficient (might not be able to fully saturate the available memory bandwidth) since it does not use the copy engine. Moreover, since it uses SMs for memory copy, when overlapping the operator with other computes, it can potentially compete for the SM resources with others. Reviewed By: q10 Differential Revision: D62114877
sryap
force-pushed
the
export-D62114877
branch
from
67c04d4 to
ad1ec29
Compare
Summary: Pull Request resolved: pytorch#3079 X-link: facebookresearch/FBGEMM#173 Prior to this diff, SSD-TBE used a combination of a pinned CPU buffer and the GPU buffer for `evicted_rows` (the buffer for staging rows that are evicted from L1 cache). It explicitly performed asynchronous memory copy (via `cudaMemcpyAsync`) to transfer `evicted_rows` from device to host. Since the number of evicted rows is known only on the device, SSD-TBE overallocated the `evicted_rows` CPU and GPU buffers. Therefore, it transferred extra data during the device-host memory copy. Such the extra data could be large and could make the memory copy a bottleneck of an execution. This diff mitigates the problem mentioned above by using a unified address buffer for `evicted_rows` and using a kernel (namely `masked_index_select` to load/store data instead of using a CUDA memory copy operation. This mechanism can avoid the extra memory copy. However, the memory copy can be less efficient (might not be able to fully saturate the available memory bandwidth) since it does not use the copy engine. Moreover, since it uses SMs for memory copy, when overlapping the operator with other computes, it can potentially compete for the SM resources with others. Reviewed By: q10 Differential Revision: D62114877
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D62114877 |
sryap
force-pushed
the
export-D62114877
branch
from
ad1ec29 to
b059883
Compare
Contributor
|
This pull request has been merged in 53d84ad. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Prior to this diff, SSD-TBE used a combination of a pinned CPU buffer
and the GPU buffer for
evicted_rows(the buffer for staging rowsthat are evicted from L1 cache). It explicitly performed asynchronous
memory copy (via
cudaMemcpyAsync) to transferevicted_rowsfromdevice to host. Since the number of evicted rows is known only on the
device, SSD-TBE overallocated the
evicted_rowsCPU and GPU buffers.Therefore, it transferred extra data during the device-host memory
copy. Such the extra data could be large and could make the memory
copy a bottleneck of an execution.
This diff mitigates the problem mentioned above by using a unified
address buffer for
evicted_rowsand using a kernel (namelymasked_index_selectto load/store data instead of using a CUDAmemory copy operation. This mechanism can avoid the extra memory
copy. However, the memory copy can be less efficient (might not be
able to fully saturate the available memory bandwidth) since it does
not use the copy engine. Moreover, since it uses SMs for memory copy,
when overlapping the operator with other computes, it can potentially
compete for the SM resources with others.
Differential Revision: D62114877