Skip to content

Conversation

@alekstheod
Copy link
Collaborator

@alekstheod alekstheod commented Sep 19, 2025

Fix tsan warning in LiteralBase::Storage

[ RUN      ] PjrtCApiGpuTransferManagerTest.TransferRawDataToBufferIsSuccessful
2025-09-19 07:47:26.558345: I xla/service/service.cc:152] XLA service 0x724400141040 initialized for platform ROCM (this does not guarantee that XLA will be used). Devices:
2025-09-19 07:47:26.558475: I xla/service/service.cc:160]   StreamExecutor device (0): AMD Instinct MI300X, AMDGPU ISA version: gfx942:sramecc+:xnack-
==================
WARNING: ThreadSanitizer: data race (pid=884588)
  Write of size 8 at 0x724800144a00 by main thread:
    #0 xla::LiteralBase::Piece::Uninitialized& xla::LiteralBase::Piece::Storage::Emplace<xla::LiteralBase::Piece::Uninitialized>() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/literal.h:1135:17 (libxla_Slibliteral.so+0x172185) (BuildId: b22939089cba7f317f3d89fdbf9517a0)
    #1 xla::LiteralBase::Piece::DeallocateBuffers() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/literal.cc:642:14 (libxla_Slibliteral.so+0x172185)
    #2 xla::Literal::DeallocateBuffers()::$_0::operator()(xla::ShapeIndex const&, xla::LiteralBase::Piece*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/literal.cc:315:16 (libxla_Slibliteral.so+0x172185)
    #3 void xla::LiteralBase::Piece::ForEachMutableSubpiece<xla::Literal::DeallocateBuffers()::$_0>(xla::Literal::DeallocateBuffers()::$_0 const&)::'lambda'(xla::ShapeIndex const&, xla::LiteralBase::Piece*)::operator()(xla::ShapeIndex const&, xla::LiteralBase::Piece*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/literal.h:1004:20 (libxla_Slibliteral.so+0x172185)
    #4 absl::lts_20230802::Status xla::LiteralBase::Piece::ForEachMutableHelper<void xla::LiteralBase::Piece::ForEachMutableSubpiece<xla::Literal::DeallocateBuffers()::$_0>(xla::Literal::DeallocateBuffers()::$_0 const&)::'lambda'(xla::ShapeIndex const&, xla::LiteralBase::Piece*)>(xla::Literal::DeallocateBuffers()::$_0 const&, xla::LiteralBase::Piece*, xla::ShapeIndex*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/literal.h:1219:7 (libxla_Slibliteral.so+0x172185)
    #5 void xla::LiteralBase::Piece::ForEachMutableSubpiece<xla::Literal::DeallocateBuffers()::$_0>(xla::Literal::DeallocateBuffers()::$_0 const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/literal.h:1002:14 (libxla_Slibliteral.so+0x391d5) (BuildId: b22939089cba7f317f3d89fdbf9517a0)
    #6 xla::Literal::DeallocateBuffers() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/literal.cc:313:15 (libxla_Slibliteral.so+0x391d5)
    #7 xla::Literal::~Literal() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/literal.cc:310:23 (libxla_Slibliteral.so+0x390f7) (BuildId: b22939089cba7f317f3d89fdbf9517a0)
    #8 void __gnu_cxx::new_allocator<xla::Literal>::destroy<xla::Literal>(xla::Literal*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:168:10 (pjrt_c_api_gpu_test_gpu_amd_any+0x1660e7) (BuildId: 57c0cb495d1c121dc743dc8c5c8a4c54)
    #9 void std::allocator_traits<std::allocator<xla::Literal>>::destroy<xla::Literal>(std::allocator<xla::Literal>&, xla::Literal*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:535:8 (pjrt_c_api_gpu_test_gpu_amd_any+0x1660e7)
    #10 std::_Sp_counted_ptr_inplace<xla::Literal, std::allocator<xla::Literal>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:528:2 (pjrt_c_api_gpu_test_gpu_amd_any+0x1660e7)
    #11 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:168:6 (pjrt_c_api_gpu_test_gpu_amd_any+0x179570) (BuildId: 57c0cb495d1c121dc743dc8c5c8a4c54)
    #12 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:705:11 (pjrt_c_api_gpu_test_gpu_amd_any+0x179570)
    #13 std::__shared_ptr<xla::Literal, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1154:31 (pjrt_c_api_gpu_test_gpu_amd_any+0x179570)
    #14 pjrt::(anonymous namespace)::PjrtCApiGpuTransferManagerTest_TransferRawDataToBufferIsSuccessful_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/c/pjrt_c_api_gpu_test.cc:500:1 (pjrt_c_api_gpu_test_gpu_amd_any+0x179570)
    #15 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #16 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #17 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #18 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #19 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #20 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #21 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #23 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #25 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Previous read of size 8 at 0x724800144a00 by thread T201:
    #0 xla::LiteralBase::Piece::Storage::data() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/literal.h:1162:16 (libxla_Slibliteral.so+0x6ce99) (BuildId: b22939089cba7f317f3d89fdbf9517a0)
    #1 xla::LiteralBase::Piece::buffer() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/literal.cc:207:19 (libxla_Slibliteral.so+0x6ce99)
    #2 xla::MutableBorrowingLiteral::CopyPieceSubtree(xla::Shape const&, xla::LiteralBase::Piece const*, xla::LiteralBase::Piece*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/literal.cc:2765:57 (libxla_Slibliteral.so+0x6ce99)
    #3 xla::MutableBorrowingLiteral::MutableBorrowingLiteral(xla::MutableLiteralBase*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/literal.cc:2802:3 (libxla_Slibliteral.so+0x6d85e) (BuildId: b22939089cba7f317f3d89fdbf9517a0)
    #4 xla::PjRtStreamExecutorBuffer::ToLiteral(xla::MutableLiteralBase*)::$_0::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:1511:32 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xa655c) (BuildId: 93462a436feb3642fce408b930908a8e)
    #5 void std::__invoke_impl<void, xla::PjRtStreamExecutorBuffer::ToLiteral(xla::MutableLiteralBase*)::$_0&>(std::__invoke_other, xla::PjRtStreamExecutorBuffer::ToLiteral(xla::MutableLiteralBase*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xa655c)
    #6 std::enable_if<is_invocable_r_v<void, xla::PjRtStreamExecutorBuffer::ToLiteral(xla::MutableLiteralBase*)::$_0&>, void>::type std::__invoke_r<void, xla::PjRtStreamExecutorBuffer::ToLiteral(xla::MutableLiteralBase*)::$_0&>(xla::PjRtStreamExecutorBuffer::ToLiteral(xla::MutableLiteralBase*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xa655c)
    #7 std::_Function_handler<void (), xla::PjRtStreamExecutorBuffer::ToLiteral(xla::MutableLiteralBase*)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xa655c)
    #8 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Spjrt_Slibtracked_Udevice_Ubuffer.so+0x10bc3) (BuildId: 32f63c854c03662ea200315a32625608)
    #9 xla::BufferSequencingEvent::ExecuteOrAddToFutureTasks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()>)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/tracked_device_buffer.cc:145:5 (libxla_Spjrt_Slibtracked_Udevice_Ubuffer.so+0x10bc3)
    #10 void std::__invoke_impl<void, xla::BufferSequencingEvent::ExecuteOrAddToFutureTasks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()>)::$_0&>(std::__invoke_other, xla::BufferSequencingEvent::ExecuteOrAddToFutureTasks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibtracked_Udevice_Ubuffer.so+0x10bc3)
    #11 std::enable_if<is_invocable_r_v<void, xla::BufferSequencingEvent::ExecuteOrAddToFutureTasks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, xla::BufferSequencingEvent::ExecuteOrAddToFutureTasks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()>)::$_0&>(xla::BufferSequencingEvent::ExecuteOrAddToFutureTasks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 (libxla_Spjrt_Slibtracked_Udevice_Ubuffer.so+0x10bc3)
    #12 std::_Function_handler<void (), xla::BufferSequencingEvent::ExecuteOrAddToFutureTasks(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Spjrt_Slibtracked_Udevice_Ubuffer.so+0x10bc3)
    #13 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f2f3) (BuildId: 5fc8eadf684b8166a9e18dc6ac6459cf)
    #14 tsl::thread::EigenEnvironment::ExecuteTask(tsl::thread::EigenEnvironment::Task const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/threadpool.cc:109:5 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f2f3)
    #15 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:362:41 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f2f3)
    #16 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:78:68 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f111) (BuildId: 5fc8eadf684b8166a9e18dc6ac6459cf)
    #17 void std::__invoke_impl<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(std::__invoke_other, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f111)
    #18 std::enable_if<is_invocable_r_v<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>, void>::type std::__invoke_r<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f111)
    #19 std::_Function_handler<void (), Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f111)
    #20 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f0af) (BuildId: 5fc8eadf684b8166a9e18dc6ac6459cf)
    #21 tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/threadpool.cc:92:7 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f0af)
    #22 void std::__invoke_impl<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(std::__invoke_other, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2f0af)
    #23 std::__invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::__invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2efb1) (BuildId: 5fc8eadf684b8166a9e18dc6ac6459cf)
    #24 std::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2efb1)
    #25 void absl::lts_20230802::internal_any_invocable::InvokeR<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&, void>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2efb1)
    #26 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2efb1)
    #27 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04) (BuildId: 6beeb825755baa193422753e666d20a5)
    #28 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:95:5 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04)

  Location is heap block of size 384 at 0x724800144900 allocated by main thread:
    #0 aligned_alloc <null> (pjrt_c_api_gpu_test_gpu_amd_any+0xe25c0) (BuildId: 57c0cb495d1c121dc743dc8c5c8a4c54)
    #1 operator new(unsigned long, std::align_val_t) <null> (libstdc++.so.6+0xaea45) (BuildId: e37fe1a879783838de78cbc8c80621fa685d58a2)
    #2 pjrt::(anonymous namespace)::PjrtCApiGpuTransferManagerTest_TransferRawDataToBufferIsSuccessful_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/c/pjrt_c_api_gpu_test.cc:493:3 (pjrt_c_api_gpu_test_gpu_amd_any+0x178e76) (BuildId: 57c0cb495d1c121dc743dc8c5c8a4c54)
    #3 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #5 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #6 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #7 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #8 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #9 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #10 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #11 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #12 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #13 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Thread T201 (tid=884798, running) created by main thread at:
    #0 pthread_create <null> (pjrt_c_api_gpu_test_gpu_amd_any+0xe300b) (BuildId: 57c0cb495d1c121dc743dc8c5c8a4c54)
    #1 tsl::(anonymous namespace)::PThread::PThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:74:15 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691) (BuildId: 6beeb825755baa193422753e666d20a5)
    #2 tsl::(anonymous namespace)::PosixEnv::StartThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:139:16 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691)
    #3 tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/threadpool.cc:84:17 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2c2a8) (BuildId: 5fc8eadf684b8166a9e18dc6ac6459cf)
    #4 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:78:41 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2c2a8)
    #5 tsl::thread::ThreadPool::ThreadPool(tsl::Env*, tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int, bool, Eigen::Allocator*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/threadpool.cc:139:31 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2b8f6) (BuildId: 5fc8eadf684b8166a9e18dc6ac6459cf)
    #6 tsl::thread::ThreadPool::ThreadPool(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, int) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/threadpool.cc:114:7 (libxla_Stsl_Splatform_Sdefault_Slibenv.so+0x2b690) (BuildId: 5fc8eadf684b8166a9e18dc6ac6459cf)
    #7 xla::PjRtStreamExecutorClient::PjRtStreamExecutorClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, xla::LocalClient*, std::vector<std::unique_ptr<xla::PjRtStreamExecutorDevice, std::default_delete<xla::PjRtStreamExecutorDevice>>, std::allocator<std::unique_ptr<xla::PjRtStreamExecutorDevice, std::default_delete<xla::PjRtStreamExecutorDevice>>>>, int, std::vector<std::unique_ptr<xla::PjRtMemorySpace, std::default_delete<xla::PjRtMemorySpace>>, std::allocator<std::unique_ptr<xla::PjRtMemorySpace, std::default_delete<xla::PjRtMemorySpace>>>>, std::unique_ptr<stream_executor::DeviceMemoryAllocator, std::default_delete<stream_executor::DeviceMemoryAllocator>>, std::unique_ptr<tsl::Allocator, std::default_delete<tsl::Allocator>>, bool, std::unique_ptr<xla::gpu::GpuExecutableRunOptions, std::default_delete<xla::gpu::GpuExecutableRunOptions>>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:253:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x4dda3) (BuildId: 93462a436feb3642fce408b930908a8e)
    #8 xla::StreamExecutorGpuClient::StreamExecutorGpuClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, xla::LocalClient*, std::vector<std::unique_ptr<xla::PjRtStreamExecutorDevice, std::default_delete<xla::PjRtStreamExecutorDevice>>, std::allocator<std::unique_ptr<xla::PjRtStreamExecutorDevice, std::default_delete<xla::PjRtStreamExecutorDevice>>>>, int, std::unique_ptr<stream_executor::DeviceMemoryAllocator, std::default_delete<stream_executor::DeviceMemoryAllocator>>, std::unique_ptr<tsl::Allocator, std::default_delete<tsl::Allocator>>, bool, std::unique_ptr<xla::gpu::GpuExecutableRunOptions, std::default_delete<xla::gpu::GpuExecutableRunOptions>>, std::shared_ptr<xla::KeyValueStoreInterface>, std::shared_ptr<xla::GpuTopology const>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:542:7 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3298f) (BuildId: acc254d3e26cdc2f1a4370e4e5281eb2)
    #9 std::_MakeUniq<xla::StreamExecutorGpuClient>::__single_object std::make_unique<xla::StreamExecutorGpuClient, char const*&, xla::LocalClient*&, std::vector<std::unique_ptr<xla::PjRtStreamExecutorDevice, std::default_delete<xla::PjRtStreamExecutorDevice>>, std::allocator<std::unique_ptr<xla::PjRtStreamExecutorDevice, std::default_delete<xla::PjRtStreamExecutorDevice>>>>, int const&, std::unique_ptr<stream_executor::DeviceMemoryAllocator, std::default_delete<stream_executor::DeviceMemoryAllocator>>, std::unique_ptr<tsl::BFCAllocator, std::default_delete<tsl::BFCAllocator>>, bool const&, std::unique_ptr<xla::gpu::GpuExecutableRunOptions, std::default_delete<xla::gpu::GpuExecutableRunOptions>>, std::shared_ptr<xla::KeyValueStoreInterface>, std::shared_ptr<xla::GpuTopology const>>(char const*&, xla::LocalClient*&, std::vector<std::unique_ptr<xla::PjRtStreamExecutorDevice, std::default_delete<xla::PjRtStreamExecutorDevice>>, std::allocator<std::unique_ptr<xla::PjRtStreamExecutorDevice, std::default_delete<xla::PjRtStreamExecutorDevice>>>>&&, int const&, std::unique_ptr<stream_executor::DeviceMemoryAllocator, std::default_delete<stream_executor::DeviceMemoryAllocator>>&&, std::unique_ptr<tsl::BFCAllocator, std::default_delete<tsl::BFCAllocator>>&&, bool const&, std::unique_ptr<xla::gpu::GpuExecutableRunOptions, std::default_delete<xla::gpu::GpuExecutableRunOptions>>&&, std::shared_ptr<xla::KeyValueStoreInterface>&&, std::shared_ptr<xla::GpuTopology const>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x4126d) (BuildId: acc254d3e26cdc2f1a4370e4e5281eb2)
    #10 xla::GetStreamExecutorGpuClient(xla::GpuClientOptions const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:1374:38 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x403b0) (BuildId: acc254d3e26cdc2f1a4370e4e5281eb2)
    #11 pjrt::gpu_plugin::PJRT_Client_Create(PJRT_Client_Create_Args*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/c/pjrt_c_api_gpu_internal.cc:182:3 (libxla_Spjrt_Sc_Slibpjrt_Uc_Uapi_Ugpu_Uinternal.so+0x140ee) (BuildId: c85779539d45aee936131953bd668b74)
    #12 pjrt::(anonymous namespace)::CreateClient(PJRT_Api const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/c/pjrt_c_api_test_base.cc:59:23 (libxla_Spjrt_Sc_Slibpjrt_Uc_Uapi_Utest_Ubase.so+0x10b4e) (BuildId: a70cc50b690a47225c4adb2739d573d3)
    #13 pjrt::PjrtCApiTestBase::PjrtCApiTestBase(PJRT_Api const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/c/pjrt_c_api_test_base.cc:69:13 (libxla_Spjrt_Sc_Slibpjrt_Uc_Uapi_Utest_Ubase.so+0x10b4e)
    #14 pjrt::(anonymous namespace)::PjrtCApiGpuTest::PjrtCApiGpuTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/c/pjrt_c_api_gpu_test.cc:92:23 (pjrt_c_api_gpu_test_gpu_amd_any+0x177dbc) (BuildId: 57c0cb495d1c121dc743dc8c5c8a4c54)
    #15 pjrt::(anonymous namespace)::PjrtCApiGpuTransferManagerTest::PjrtCApiGpuTransferManagerTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/c/pjrt_c_api_gpu_test.cc:176:7 (pjrt_c_api_gpu_test_gpu_amd_any+0x177dbc)
    #16 pjrt::(anonymous namespace)::PjrtCApiGpuTransferManagerTest_TransferRawDataToBufferIsSuccessful_Test::PjrtCApiGpuTransferManagerTest_TransferRawDataToBufferIsSuccessful_Test() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/c/pjrt_c_api_gpu_test.cc:432:1 (pjrt_c_api_gpu_test_gpu_amd_any+0x177dbc)
    #17 testing::internal::TestFactoryImpl<pjrt::(anonymous namespace)::PjrtCApiGpuTransferManagerTest_TransferRawDataToBufferIsSuccessful_Test>::CreateTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/internal/gtest-internal.h:456:44 (pjrt_c_api_gpu_test_gpu_amd_any+0x177dbc)
    #18 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #19 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a)
    #20 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2875:22 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e1c) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #21 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #23 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #25 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #27 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

SUMMARY: ThreadSanitizer: data race /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/literal.h:1135:17 in xla::LiteralBase::Piece::Uninitialized& xla::LiteralBase::Piece::Storage::Emplace<xla::LiteralBase::Piece::Uninitialized>()

Copy link
Collaborator

@i-chaochen i-chaochen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm...this part is identical to upstream as well, I would suggest just to upstream this fix and see how google folks said https://github.com/openxla/xla/blob/main/xla/literal.h#L1107-L1110

@alekstheod alekstheod force-pushed the fix_tsan_warnings_in_literal_base branch from e00abc7 to 7daf010 Compare September 19, 2025 14:20
@i-chaochen
Copy link
Collaborator

seems this is rocm-specific issue, we might need to further investigate it.

@alekstheod
Copy link
Collaborator Author

Sequence diagram explaining the issue:
image

The main thread will deallocate the Literals while thread T201 is still executing and using them.

@alekstheod alekstheod force-pushed the fix_tsan_warnings_in_literal_base branch 8 times, most recently from 3ad7ef4 to 0a10f24 Compare September 23, 2025 08:52
@alekstheod alekstheod force-pushed the fix_tsan_warnings_in_literal_base branch 4 times, most recently from 8531e14 to 755582c Compare September 24, 2025 13:11
@alekstheod alekstheod force-pushed the fix_tsan_warnings_in_literal_base branch from 755582c to 2609a66 Compare September 25, 2025 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants