Create logger wrapper around spdlog that can be easily reused in other libraries by vyasr · Pull Request #1722 · rapidsai/rmm

vyasr · 2024-11-06T22:05:02Z

Description

This PR defines a new way to produce a logger wrapping spdlog. The logger's interface is declared in a template header file that can be processed by CMake to produce an interface that may be customized for placement into any project. The new implementation uses the PImpl idiom to isolate the spdlog (and transitively, fmt) dependency from the public API of the logger. The implementation is defined in an impl header. A corresponding source template file is provided that simply includes this header. All of these files are wrapped in some CMake logic for producing a custom target for a given project.

rmm leverages this new logger by requesting the creation of a logger target and a corresponding implementation. This is a breaking change because consumers of rmm will need to link the new rmm_logger_impl target into their own libraries to get logging support. Once this gets merged, the plan is to move this implementation out of rmm into its own repository. At that point, the logger may also be used to completely replace logger implementations in cudf, raft, and cuml (as well as any other RAPIDS libraries that are aiming to provide their own logging implementation). Once everything in RAPIDS is migrated to using the new logger, we will update the way that it uses spdlog to completely hide all spdlog symbols, which solves a half dozen different problems for us when it comes to packaging (symbol collision issues, ABI compatibility, conda environment conflicts, bundling of headers into conda packages, etc).

Resolves #1709

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

harrism

Couple comments.

vyasr · 2024-11-07T17:44:16Z

I should note that a lot of the complexity here is from supporting the backwards compatibility modes. I wanted to get this ready now so that we could easily test all cases and flip the switch as needed, but if it becomes too cumbersome we could just wait for 25.02 and delete the compatibility paths before merging anything.

vyasr · 2024-11-26T20:23:34Z

/merge

It looks like while #1722 introduced usage of the modern target_link_libraries syntax it did not adjust all other calls because I wasn't setting up coverage usage locally or anywhere else in CI. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Robert Maynard (https://github.com/robertmaynard) URL: #1738

This fixes a handful of issues uncovered in downstream CI after #1722. The following are bugs introduced in #1722: - When using rmm from the build directory rather than an installation, the namespaced targets are not present so we must generate aliases. - The `RMM_LOGGING_ASSERT` macro is never used in rmm itself, so we didn't catch that it was still using the old version of the logger. While fixing the above, I also uncovered that building fmt in this environment unearths a gcc bug. The following are underlying issues uncovered by #1722: - spdlog's fmt CMake linkage is determined at build time. As a result, the conda package for spdlog is hardcoded to use fmt as a library (static or shared depends on what the `fmt::fmt` target winds up being when a consumer using spdlog finds fmt in CMake), which means that is propagated to all consumers of the librmm package via its CMake. This means that we often wind up with both fmt_header_only and fmt as link targets for many RAPIDS libraries. For now, this PR makes it so that if `rapids_cpm_find(spdlog)` does not find a copy of spdlog locally, the cloned version will use an external header-only fmt via rapids-cmake's logic, which ensures that packages like wheels do not export a libfmt or libspdlog dependency. However, in environments where `rapids_cpm_find(spdlog)` does find a preexisting package, we allow that package's fmt linkage to propagate. In conda environments, we know that this fmt linkage is to the library, so we keep fmt as part of rmm's runtime dependencies (by placing it in host and relying on the run export) so that libfmt is always available in environments using rmm. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #1739

This PR adapts to breaking changes in rmm in rapidsai/rmm#1722. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #336

This PR adapts to breaking changes in rmm in rapidsai/rmm#1722. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Mark Harris (https://github.com/harrism) - Nghia Truong (https://github.com/ttnghia) URL: #17451

jameslamb · 2024-11-27T17:12:38Z

-    if grep -E 'spdlog\:\:' < "${symbol_file}" \
-        | grep -v 'std\:\:_Destroy_aux'
-    then
+    if grep -E 'spdlog\:\:' < "${symbol_file}"; then


(reviewing after merge just to be sure I understand what's happening)

Love that hiding these symbols also reduced the complexity of these checks! 🎉

This PR adapts to breaking changes in rmm in rapidsai/rmm#1722. This PR is a breaking change because consumers of raft that use any functionality that touches rmm logging will need to link to the rmm::rmm_logger_impl target as well now. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #2513

This PR adapts to breaking changes in rmm in rapidsai/rmm#1722. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - Ben Frederickson (https://github.com/benfred) URL: #499

This PR adapts to breaking changes in rmm in rapidsai/rmm#1722. --------- Co-authored-by: Bradley Dice <[email protected]>

This PR adapts to breaking changes in rmm in rapidsai/rmm#1722. Authors: - Vyas Ramasubramani (https://github.com/vyasr) - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - James Lamb (https://github.com/jameslamb) - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Don Acosta (https://github.com/acostadon) URL: #4794

As of rapidsai/rmm#1722, rmm no longer depends on spdlog. However, some tests were depending on spdlog transitively through rmm. This PR adds spdlog as an explicit dependency to correct those failures. Authors: - Robert Maynard (https://github.com/robertmaynard) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: #733

These targets were added in rapidsai/rmm#1722 and must be propagated. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - Robert Maynard (https://github.com/robertmaynard) URL: #739

vyasr requested review from a team as code owners November 6, 2024 22:05

vyasr requested review from harrism and jrhemstad November 6, 2024 22:05

github-actions Bot added CMake Python Related to RMM Python API cpp Pertains to C++ code labels Nov 6, 2024

vyasr self-assigned this Nov 6, 2024

vyasr added non-breaking Non-breaking change improvement Improvement / enhancement to an existing function labels Nov 6, 2024

vyasr marked this pull request as draft November 6, 2024 22:47

harrism reviewed Nov 7, 2024

View reviewed changes

Comment thread include/rmm/mr/device/detail/arena.hpp Outdated

harrism reviewed Nov 7, 2024

View reviewed changes

Comment thread rapids_logger/logger.hpp.in Outdated

Comment thread rapids_logger/logger.hpp.in Outdated

harrism reviewed Nov 7, 2024

View reviewed changes

Comment thread include/rmm/mr/device/arena_memory_resource.hpp Outdated

robertmaynard requested changes Nov 7, 2024

View reviewed changes

Comment thread tests/CMakeLists.txt Outdated

Comment thread CMakeLists.txt Outdated

Comment thread rapids_logger/CMakeLists.txt Outdated

Comment thread rapids_logger/CMakeLists.txt Outdated

robertmaynard requested changes Nov 7, 2024

View reviewed changes

Comment thread CMakeLists.txt Outdated

vyasr added 12 commits November 7, 2024 18:16

First version of new logger

27b6cab

Make rmm backwards compat layers explicit

aa92779

Move logger to its own directory

812b2bc

Set up basics of file generation

956fa3f

Convert to a function

3a6d49b

Switch to using a target

c9ec22c

Make rmm compatibility and logging support controllable via CMake

2573e02

Hide impl one level down

9173108

Make rapids_logger a CMake package

d89bb82

Fix install rules

07dfcc5

Switch arena to delegating constructor for simplicity

8194997

Clean up some conditional compilation

8589c53

Add comments

a21dedc

robertmaynard approved these changes Nov 26, 2024

View reviewed changes

AyodeAwe approved these changes Nov 26, 2024

View reviewed changes

rapids-bot Bot merged commit 4cfa31f into rapidsai:branch-25.02 Nov 26, 2024

vyasr mentioned this pull request Nov 26, 2024

Use consistent signature for target_link_libraries #1738

Merged

3 tasks

vyasr deleted the feat/logger branch November 26, 2024 20:45

jlowe mentioned this pull request Nov 26, 2024

Fix cudf submodule sync NVIDIA/spark-rapids-jni#2629

Merged

vyasr mentioned this pull request Nov 27, 2024

Fix some logger issues #1739

Merged

3 tasks

bdice mentioned this pull request Nov 27, 2024

Various fixes to the manifest rapidsai/devcontainers#422

Merged

jameslamb reviewed Nov 27, 2024

View reviewed changes

vyasr mentioned this pull request Nov 27, 2024

Adapt to rmm logger changes rapidsai/cugraph#4794

Merged

raydouglass pushed a commit to rapidsai/cuml that referenced this pull request Dec 2, 2024

Adapt to rmm logger changes (#6147)

6f8a189

This PR adapts to breaking changes in rmm in rapidsai/rmm#1722. --------- Co-authored-by: Bradley Dice <[email protected]>

vyasr mentioned this pull request Dec 16, 2024

Correct nvbench + conda test to explicitly pull spdlog rapidsai/rapids-cmake#733

Merged

7 tasks

vyasr mentioned this pull request Jan 6, 2025

Add new global targets for rmm rapidsai/rapids-cmake#739

Merged

7 tasks

robertmaynard mentioned this pull request Jan 9, 2025

Adapt to rmm 25.02 logger changes dmlc/xgboost#11153

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create logger wrapper around spdlog that can be easily reused in other libraries#1722

Create logger wrapper around spdlog that can be easily reused in other libraries#1722
rapids-bot[bot] merged 122 commits intorapidsai:branch-25.02from
vyasr:feat/logger

vyasr commented Nov 6, 2024 •

edited

Loading

Uh oh!

Uh oh!

harrism left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vyasr commented Nov 7, 2024

Uh oh!

vyasr commented Nov 26, 2024

Uh oh!

jameslamb Nov 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

vyasr commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

Uh oh!

harrism left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vyasr commented Nov 7, 2024

Uh oh!

vyasr commented Nov 26, 2024

Uh oh!

jameslamb Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vyasr commented Nov 6, 2024 •

edited

Loading