[SYCL] Implement multiple tracing levels for SYCL_UR_TRACE

[callumfare]

This undoes some changes to tracing that were made when porting from PI to UR, and means that SYCL_UR_TRACE now has the same behavior as the SYCL_PI_TRACE option that it replaced.

Also only enable the UR tracing layer when SYCL_UR_TRACE is set to the appropriate level, rather than always enabling it when UR_LOG_TRACING is set - fixes #14926

[omarahmed1111]

@intel/dpcpp-doc-reviewers @intel/dpcpp-esimd-reviewers @intel/dpcpp-tools-reviewers @intel/sycl-graphs-reviewers please review, Thanks!

[sarnex]

lgtm minus a question

[sarnex]

I don't understand the difference between 2 and -1

[callumfare]

2 enables just the tracing of UR calls, but -1 enables tracing of the UR calls as well as the basic device/adapter discovery tracing, i.e. 1 and 2 combined (and implicitly also any future tracing levels that may be added).

[sarnex]

thanks, think we can add that explanation to the doc?

[callumfare]

I've updated the documentation now

[omarahmed1111]

@intel/dpcpp-doc-reviewers Please review, Thanks!

[omarahmed1111]

@intel/llvm-reviewers-runtime Please review, Thanks!

[aelovikov-intel]

To be honest, it doesn't make sense to me to have any UR traces inside SYCL RT. Is that bad naming or some bigger issue with tracing design?

[dm-vodopyanov]

@callumfare please follow https://github.com/intel/llvm/blob/sycl/sycl/doc/developer/ContributeToDPCPP.md#commit-message

[kbenzie]

To be honest, it doesn't make sense to me to have any UR traces inside SYCL RT. Is that bad naming or some bigger issue with tracing design?

This is restoring the behavior of SYCL_PI_TRACE. It arguably shouldn't have been used in all the places it was either.

[callumfare]

To be honest, it doesn't make sense to me to have any UR traces inside SYCL RT. Is that bad naming or some bigger issue with tracing design?

@aelovikov-intel The tracing is implemented in UR, but the environment variables to enable it are quite verbose:
UR_ENABLE_LAYERS=UR_LAYER_TRACING UR_LOG_TRACING=level:info;output:stdout;flush:info UR_LOG_LOADER=level:info;output:stdout;flush:info

The SYCL_UR_TRACE option is just a shortcut for this, and also provides some continuity from SYCL_PI_TRACE which was recently removed

[aelovikov-intel]

This is restoring the behavior of SYCL_PI_TRACE. It arguably shouldn't have been used in all the places it was either.

But previously plugins were part of this project, so while not ideal implementation-wise, still not leaking abstractions terribly. I don't see that being true now that UR is separate.

[aelovikov-intel]

The SYCL_UR_TRACE option is just a shortcut for this, and also provides some continuity from SYCL_PI_TRACE which was recently removed

Why can't that shortcut be provided at the UR side?

[callumfare]

Why can't that shortcut be provided at the UR side?

@aelovikov-intel It could, although from a SYCL user's perspective it makes sense for the option to start with the SYCL_ prefix, and having an option with that name in UR wouldn't be ideal. Also as Benie said above, some elements of the tracing are implemented in SYCL-RT, which does mean that the option has a misleading name, so something should probably change.

I'd rather get this merged asap to get rid of functional differences with SYCL_PI_TRACE, and then we could revisit the design of the tracing in future.

[aelovikov-intel]

Not a fan of it, but if it's needed to unblock folks...