[rocprofiler-systems] [ROCpd] Add OMPT callbacks to ROCpd by kcossett-amd · Pull Request #1016 · ROCm/rocm-systems

kcossett-amd · 2025-09-16T15:03:24Z

Motivation

Purpose: Enter OMPT callbacks to the database.

Technical Details

Main Changes

Callbacks to be put inside ROCpd are inserted in three maps:
- ompt_parallel_cb_storage: Exclusively used to handle parallel_begin and parallel_end callbacks. Their parallel_data value is used as their unique key.
- Theoretically, there should also be one for thread_begin and thread_end callbacks, but we do not support these as of now due to finalization issue.
- Instant events are handled using ompt_cache_instant_event.
Once a "start" part of a callback is received, we use the respective ompt_push... function to insert it into the map. Once the end is received, the ompt_pop... function is used to exrtact the data then cache it using cache_ompt_region.
Any entries left in the map (as their "end" is not received) will be treated as orphan events and is handled by ompt_finalize_orphan_events. This function is called in tool_fini.
An "orphan event" is treated the same as an "instant" event.

Other changes

Moved the "is_first_implicit_task" check into tool_tracing_callback and updated its comment to be more precise.
Added ompt_get_unified_name as a function that will return the correct track name.
Added name checking in post processing.

Test Plan

Tested using my own cpu.f90, gpu.f90 and mutex.f90.

Also tested by spawning nested parallel regions for N = [10000, 100000]

Test Result

Using the rocpd2perfetto tool, the callbacks appear as expected. The same holds for the visualizer.

For the callback renaming, logs show that operations have been renamed and all categories (and their operations) that follow overwritten cat are present.

Category 15:
[0] omp_thread
[1] omp_thread
[2] omp_parallel
[3] omp_parallel
...
[29] omp_error
[30] omp_callback_functions

Category 16:
[0] MEMORY_ALLOCATION_NONE
[1] MEMORY_ALLOCATION_ALLOCATE
[2] MEMORY_ALLOCATION_VMEM_ALLOCATE
...

Result of nested parallel regions showed that the maps had to be made thread_local.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

projects/rocprofiler-systems/source/lib/core/trace_cache/sample_type.hpp

projects/rocprofiler-systems/source/lib/rocprof-sys/library/rocprofiler-sdk.cpp

projects/rocprofiler-systems/source/lib/core/trace_cache/rocpd_post_processing.cpp

projects/rocprofiler-systems/source/lib/rocprof-sys/library/rocprofiler-sdk.cpp

projects/rocprofiler-systems/source/lib/core/trace_cache/metadata_registry.cpp

projects/rocprofiler-systems/source/lib/core/trace_cache/metadata_registry.hpp

projects/rocprofiler-systems/source/lib/core/trace_cache/metadata_registry.cpp

projects/rocprofiler-systems/source/lib/rocprof-sys/library/rocprofiler-sdk.cpp

dgaliffiAMD

Thanks. Please see comments and suggestions.

projects/rocprofiler-systems/source/lib/core/trace_cache/metadata_registry.hpp

dgaliffiAMD · 2025-10-07T23:00:26Z

Reviewed "ubuntu 22.04 * gfx950" failure. It is not related and is being investigated in parallel.

(#1016) * Add OMPT to ROCpd * Use correct category * Added wrapper functions for future control * Formatting * Fix naming * Comment change * Remove ompt_get_cb_args * Switched to using region_sample for OMPT * Remove relic function * Remove get_use_rocpd that was used in this pr (one still remains) * Rename ompt_get_args_string and reuse in tool_tracing_callback_stop * Make lock init and destroy cb instant * [Prototype] ROCPD Name fix * [Prototype] ROCPD Name fix P1 * [Prototype] ROCPD Name fix P2 * ROCPD Name fix * Var name changes * Rewrite cb overwrite to single function * [Important] Use parallel_data as key for parallel callback map * Fix workflow failure * Make cpp USE_ROCM consistent with hpp and use default constructor if USE_ROCM = 0 * Add missing ROCPROFILER_VERSION check * Improve readability * Make ompt storage maps thread local * Part 1: Variable name fix, memory cleanup, and fixed asserts * Part 2: Add comments * Part 3: Add CI_THROW * Part 4: Formatting * Part 5: Move #include to cpp [rocm-systems] ROCm/rocm-systems#1016 (commit 0c53a12)

* Add OMPT to ROCpd * Use correct category * Added wrapper functions for future control * Formatting * Fix naming * Comment change * Remove ompt_get_cb_args * Switched to using region_sample for OMPT * Remove relic function * Remove get_use_rocpd that was used in this pr (one still remains) * Rename ompt_get_args_string and reuse in tool_tracing_callback_stop * Make lock init and destroy cb instant * [Prototype] ROCPD Name fix * [Prototype] ROCPD Name fix P1 * [Prototype] ROCPD Name fix P2 * ROCPD Name fix * Var name changes * Rewrite cb overwrite to single function * [Important] Use parallel_data as key for parallel callback map * Fix workflow failure * Make cpp USE_ROCM consistent with hpp and use default constructor if USE_ROCM = 0 * Add missing ROCPROFILER_VERSION check * Improve readability * Make ompt storage maps thread local * Part 1: Variable name fix, memory cleanup, and fixed asserts * Part 2: Add comments * Part 3: Add CI_THROW * Part 4: Formatting * Part 5: Move #include to cpp

* [rocprofiler-systems] [ROCpd] Add OMPT callbacks to ROCpd (#1016) * Add OMPT to ROCpd * Use correct category * Added wrapper functions for future control * Formatting * Fix naming * Comment change * Remove ompt_get_cb_args * Switched to using region_sample for OMPT * Remove relic function * Remove get_use_rocpd that was used in this pr (one still remains) * Rename ompt_get_args_string and reuse in tool_tracing_callback_stop * Make lock init and destroy cb instant * [Prototype] ROCPD Name fix * [Prototype] ROCPD Name fix P1 * [Prototype] ROCPD Name fix P2 * ROCPD Name fix * Var name changes * Rewrite cb overwrite to single function * [Important] Use parallel_data as key for parallel callback map * Fix workflow failure * Make cpp USE_ROCM consistent with hpp and use default constructor if USE_ROCM = 0 * Add missing ROCPROFILER_VERSION check * Improve readability * Make ompt storage maps thread local * Part 1: Variable name fix, memory cleanup, and fixed asserts * Part 2: Add comments * Part 3: Add CI_THROW * Part 4: Formatting * Part 5: Move #include to cpp * Add missing counter events handling for ROCPD (#1305) * Add missing counter events handling for ROCPD * Update projects/rocprofiler-systems/source/lib/rocprof-sys/library/rocprofiler-sdk/counters.cpp * Update projects/rocprofiler-systems/source/lib/rocprof-sys/library/rocprofiler-sdk/counters.cpp * Fixed formatting Signed-off-by: David Galiffi <David.Galiffi@amd.com> --------- Signed-off-by: Marjan Antic <Marjan.Antic@amd.com> Co-authored-by: David Galiffi <David.Galiffi@amd.com> * Update VERSION to 1.2.1 * Update CHANGELOG.md * Add caching of category region for rocpd (#1420) * Add caching of category region Fix vaapi traces Remove region_with_name * Applied suggestions from code review * Add some simple rocpd testing Signed-off-by: David Galiffi <David.Galiffi@amd.com> * Adjust `rocpd_string` validation parameters * Adjust `rocm_marker_api` validation parameters * Update projects/rocprofiler-systems/CHANGELOG.md Co-authored-by: Pratik Basyal <pratik.basyal@amd.com> * Update projects/rocprofiler-systems/CHANGELOG.md Co-authored-by: Pratik Basyal <pratik.basyal@amd.com> * Update projects/rocprofiler-systems/CHANGELOG.md Co-authored-by: Pratik Basyal <pratik.basyal@amd.com> --------- Signed-off-by: Marjan Antic <Marjan.Antic@amd.com> Signed-off-by: David Galiffi <David.Galiffi@amd.com> Co-authored-by: Kian Cossettini <Kian.Cossettini@amd.com> Co-authored-by: marantic-amd <marantic@amd.com> Co-authored-by: Milan Radosavljevic <milan.radosavljevic@amd.com> Co-authored-by: Pratik Basyal <pratik.basyal@amd.com>

Add OMPT to ROCpd

d3a9b15

github-actions bot added the project: rocprofiler-systems label Sep 16, 2025

kcossett-amd changed the title ~~[rocprofiler-systems] Add OMPT callbacks to ROCpd~~ [rocprofiler-systems] [ROCpd] Add OMPT callbacks to ROCpd Sep 16, 2025

systems-assistant bot added the organization: ROCm label Sep 16, 2025

kcossett-amd added the WIP label Sep 16, 2025

kcossett-amd force-pushed the users/kcossett-amd/rocpd-ompt branch from 757d9da to d3a9b15 Compare September 16, 2025 18:23

kcossett-amd added 2 commits September 16, 2025 14:25

Use correct category

e70f59b

Added wrapper functions for future control

9926233

kcossett-amd force-pushed the users/kcossett-amd/rocpd-ompt branch from 1712c1a to 9926233 Compare September 17, 2025 12:58

kcossett-amd added 2 commits September 17, 2025 08:59

Formatting

cdd3212

Fix naming

9cc79af

kcossett-amd marked this pull request as ready for review September 17, 2025 13:21

kcossett-amd requested review from a team and jrmadsen as code owners September 17, 2025 13:21

Comment change

2cb7b97

kcossett-amd removed the WIP label Sep 17, 2025

kcossett-amd marked this pull request as draft September 17, 2025 14:56

Remove ompt_get_cb_args

ffba5db

kcossett-amd marked this pull request as ready for review September 17, 2025 15:13

marantic-amd reviewed Sep 18, 2025

View reviewed changes

projects/rocprofiler-systems/source/lib/core/trace_cache/sample_type.hpp Outdated Show resolved Hide resolved

kcossett-amd added 2 commits September 18, 2025 07:50

Switched to using region_sample for OMPT

94e22f8

Remove relic function

fa74e17

mradosav-amd reviewed Sep 19, 2025

View reviewed changes

kcossett-amd added 3 commits September 19, 2025 07:25

Remove get_use_rocpd that was used in this pr (one still remains)

6f12f01

Rename ompt_get_args_string and reuse in tool_tracing_callback_stop

d2308b1

Make lock init and destroy cb instant

9c13e25

kcossett-amd force-pushed the users/kcossett-amd/rocpd-ompt branch from f8b1c31 to 9c13e25 Compare September 21, 2025 22:49

[Prototype] ROCPD Name fix

ace8715

mradosav-amd reviewed Sep 22, 2025

View reviewed changes

projects/rocprofiler-systems/source/lib/core/trace_cache/metadata_registry.cpp Outdated Show resolved Hide resolved

kcossett-amd force-pushed the users/kcossett-amd/rocpd-ompt branch from f6e0a14 to ace8715 Compare September 22, 2025 12:44

kcossett-amd removed the WIP label Sep 25, 2025

mradosav-amd approved these changes Sep 29, 2025

View reviewed changes

projects/rocprofiler-systems/source/lib/core/trace_cache/metadata_registry.hpp Outdated Show resolved Hide resolved

projects/rocprofiler-systems/source/lib/core/trace_cache/metadata_registry.cpp Outdated Show resolved Hide resolved

kcossett-amd force-pushed the users/kcossett-amd/rocpd-ompt branch from a1eb46c to 4cca350 Compare September 29, 2025 13:35

kcossett-amd added 3 commits September 29, 2025 09:35

Improve readability

00070cd

Make ompt storage maps thread local

1372fe3

Merge branch 'develop' into users/kcossett-amd/rocpd-ompt

ef4b82d

kcossett-amd requested a review from mradosav-amd September 29, 2025 13:40

mradosav-amd approved these changes Sep 29, 2025

View reviewed changes

kcossett-amd requested a review from dgaliffiAMD September 29, 2025 14:59

dgaliffiAMD reviewed Oct 3, 2025

View reviewed changes

dgaliffiAMD requested changes Oct 3, 2025

View reviewed changes

kcossett-amd added 4 commits October 5, 2025 18:18

Part 1: Variable name fix, memory cleanup, and fixed asserts

f61f3ef

Part 2: Add comments

ef77830

Part 3: Add CI_THROW

d5b0088

Part 4: Formatting

f3d620d

kcossett-amd requested a review from dgaliffiAMD October 6, 2025 11:03

dgaliffiAMD reviewed Oct 6, 2025

View reviewed changes

projects/rocprofiler-systems/source/lib/core/trace_cache/metadata_registry.hpp Outdated Show resolved Hide resolved

kcossett-amd added 2 commits October 6, 2025 13:49

Part 5: Move #include to cpp

0dafa85

Merge branch 'develop' into users/kcossett-amd/rocpd-ompt

1a99122

dgaliffiAMD approved these changes Oct 6, 2025

View reviewed changes

Merge branch 'develop' into users/kcossett-amd/rocpd-ompt

9de089f

dgaliffiAMD merged commit 0c53a12 into ROCm:develop Oct 7, 2025
48 of 50 checks passed

dgaliffiAMD mentioned this pull request Oct 17, 2025

[7.1] Fix rocPD data collection #1439

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rocprofiler-systems] [ROCpd] Add OMPT callbacks to ROCpd#1016

[rocprofiler-systems] [ROCpd] Add OMPT callbacks to ROCpd#1016
dgaliffiAMD merged 35 commits intoROCm:developfrom
kcossett-amd:users/kcossett-amd/rocpd-ompt

kcossett-amd commented Sep 16, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgaliffiAMD left a comment

Uh oh!

Uh oh!

dgaliffiAMD commented Oct 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

kcossett-amd commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgaliffiAMD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dgaliffiAMD commented Oct 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kcossett-amd commented Sep 16, 2025 •

edited

Loading