Skip to content

Fix pjrt stream executor tsan issue #363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: rocm-jaxlib-v0.6.0
Choose a base branch
from
Open

Fix pjrt stream executor tsan issue #363

wants to merge 2 commits into from
+6 −5

Conversation

@alekstheod
Copy link
Collaborator

@alekstheod alekstheod commented Sep 19, 2025 •
edited
Loading 

[----------] Global test environment set-up.
[----------] 1 test from HloTest
[ RUN      ] HloTest.HloTest
2025-09-19 10:23:34.523662: I xla/service/service.cc:152] XLA service 0x724400006f40 initialized for platform ROCM (this does not guarantee that XLA will be used). Devices:
2025-09-19 10:23:34.523893: I xla/service/service.cc:160]   StreamExecutor device (0): AMD Instinct MI300X, AMDGPU ISA version: gfx942:sramecc+:xnack-
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1758277414.565907 1017489 se_gpu_pjrt_client.cc:999] Using BFC allocator.
I0000 00:00:1758277414.566053 1017489 gpu_helpers.cc:136] XLA backend allocating 16491332239 bytes on device 0 for BFCAllocator.
I0000 00:00:1758277414.566106 1017489 gpu_helpers.cc:177] XLA backend will use up to 189650320752 bytes on device 0 for CollectiveBFCAllocator.
xla/tests/fuzz/rand_000041.hlo
/* This file is autogenerated! Please don't edit! */
HloModule xla_computation_unknown, entry_computation_layout={(pred[8,6]{1,0})->(pred[8,6]{1,0})}

ENTRY main.3 {
  Arg_0.1 = pred[8,6]{1,0} parameter(0)
  ROOT tuple.2 = (pred[8,6]{1,0}) tuple(Arg_0.1)
}


2025-09-19 10:23:42.622357: I xla/service/gpu/llvm_gpu_backend/amdgpu_backend.cc:157] HSACO file cache in '/tmp' is enabled for LLVM modules with bitcode size >= 65536 bytes
W0000 00:00:1758277422.635718 1017489 stream_executor.h:344] Not enough memory to allocate 16491332096 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635791 1017489 stream_executor.h:344] Not enough memory to allocate 14842198016 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635806 1017489 stream_executor.h:344] Not enough memory to allocate 13357977600 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635818 1017489 stream_executor.h:344] Not enough memory to allocate 12022179840 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635828 1017489 stream_executor.h:344] Not enough memory to allocate 10819961856 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635837 1017489 stream_executor.h:344] Not enough memory to allocate 9737965568 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635846 1017489 stream_executor.h:344] Not enough memory to allocate 8764169216 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635855 1017489 stream_executor.h:344] Not enough memory to allocate 7887752192 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635864 1017489 stream_executor.h:344] Not enough memory to allocate 7098976768 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635873 1017489 stream_executor.h:344] Not enough memory to allocate 6389079040 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635887 1017489 stream_executor.h:344] Not enough memory to allocate 5750171136 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635896 1017489 stream_executor.h:344] Not enough memory to allocate 5175153664 on device 0 within provided limit.  limit=4404019200]
W0000 00:00:1758277422.635905 1017489 stream_executor.h:344] Not enough memory to allocate 4657638400 on device 0 within provided limit.  limit=4404019200]
==================
WARNING: ThreadSanitizer: data race (pid=1017489)
  Read of size 8 at 0x720c001197f0 by thread T6:
    #0 __gnu_cxx::__normal_iterator<absl::lts_20230802::AnyInvocable<void () &&>*, std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>>::__normal_iterator(absl::lts_20230802::AnyInvocable<void () &&>* const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:1028:20 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb150c) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::begin() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:812:16 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb150c)
    #2 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb150c)
    #3 void std::__invoke_impl<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(std::__invoke_other, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb150c)
    #4 std::__invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::__invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb150c)
    #5 std::invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb150c)
    #6 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb150c)
    #7 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb150c)
    #8 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #9 xla::WorkerThread::WorkLoop() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:57:5 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc)
    #10 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:63 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #11 void std::__invoke_impl<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(std::__invoke_other, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #12 std::__invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::__invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #13 std::invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #14 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&, void>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #15 void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #16 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04) (BuildId: 6beeb825755baa193422753e666d20a5)
    #17 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:95:5 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04)

  Previous write of size 8 at 0x720c001197f0 by main thread:
    #0 std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_impl_data::_Vector_impl_data(std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_impl_data&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:103:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_impl::_Vector_impl(std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_impl&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:144:36 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #2 std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_base(std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:308:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #3 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::vector(std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:572:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #4 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::$_1($_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #5 void absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::InitializeRemoteManager<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, 0>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:634:34 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #6 void absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::InitializeStorage<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, 0>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:595:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #7 void absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::Initialize<(absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::TargetType)3, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:573:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #8 absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::CoreImpl<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(absl::lts_20230802::internal_any_invocable::TypedConversionConstruct<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&>, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:469:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #9 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::Impl<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(absl::lts_20230802::internal_any_invocable::ConversionConstruct, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #10 absl::lts_20230802::AnyInvocable<void () &&>::AnyInvocable<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/any_invocable.h:186:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #11 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7561d)
    #12 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #13 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #14 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #15 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #16 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #18 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #20 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #21 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #23 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #28 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #29 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #30 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #31 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Location is heap block of size 48 at 0x720c001197f0 allocated by main thread:
    #0 malloc <null> (rand_000041.hlo_test_gpu_amd_any+0xbf50c) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 operator new(unsigned long) <null> (libstdc++.so.6+0xae98b) (BuildId: e37fe1a879783838de78cbc8c80621fa685d58a2)
    #2 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #3 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #4 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #5 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #6 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #7 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #8 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #9 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #10 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #13 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #14 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #15 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #16 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #19 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #20 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #21 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Thread T6 (tid=1017792, running) created by main thread at:
    #0 pthread_create <null> (rand_000041.hlo_test_gpu_amd_any+0xc0e6b) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 tsl::(anonymous namespace)::PThread::PThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:74:15 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691) (BuildId: 6beeb825755baa193422753e666d20a5)
    #2 tsl::(anonymous namespace)::PosixEnv::StartThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:139:16 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691)
    #3 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:12 (libxla_Spjrt_Slibworker_Uthread.so+0x2f03) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #4 std::_MakeUniq<xla::WorkerThread>::__single_object std::make_unique<xla::WorkerThread, tsl::Env*, char const (&) [16]>(tsl::Env*&&, char const (&) [16]) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xe33a) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #5 xla::LocalDeviceState::LocalDeviceState(stream_executor::StreamExecutor*, xla::LocalClient*, xla::LocalDeviceState::AllocationModel, int, bool, bool, int, std::optional<xla::LocalDeviceState::StreamOptions>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/local_device_state.cc:122:7 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xd4c0) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #6 std::_MakeUniq<xla::LocalDeviceState>::__single_object std::make_unique<xla::LocalDeviceState, stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel, int, bool, bool>(stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel&&, int&&, bool&&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35) (BuildId: ca04fde0f1b1bd292e56fb0b668bb930)
    #7 xla::(anonymous namespace)::BuildLocalDeviceStates(xla::LocalClient*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:966:9 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #8 xla::GetStreamExecutorGpuClient(xla::GpuClientOptions const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:1337:3 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #9 xla::(anonymous namespace)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/pjrt_gpu_client_registry.cc:33:32 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c) (BuildId: 043dd621e5330e6517632fc603217b8c)
    #10 absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> std::__invoke_impl<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #11 std::enable_if<is_invocable_r_v<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>, absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>>::type std::__invoke_r<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #12 std::_Function_handler<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> (), xla::(anonymous namespace)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #13 std::function<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5) (BuildId: 2159a5772edbd729fc7fe8982f4b31a7)
    #14 xla::(anonymous namespace)::GetHloRunnerForTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:39:56 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #15 xla::HloPjRtTestBase::HloPjRtTestBase(xla::HloPjRtTestBaseOptions) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:55:33 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #16 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::HloRunnerAgnosticReferenceMixin<>(std::unique_ptr<xla::HloRunnerInterface, std::default_delete<xla::HloRunnerInterface>>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:74:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::HloPjRtInterpreterReferenceMixin<xla::HloPjRtTestBase>::HloPjRtInterpreterReferenceMixin<>() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_pjrt_interpreter_reference_mixin.h:38:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d)
    #18 xla::(anonymous namespace)::HloTest::HloTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:36:7 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::(anonymous namespace)::HloTest_HloTest_Test::HloTest_HloTest_Test() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:38:1 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #20 testing::internal::TestFactoryImpl<xla::(anonymous namespace)::HloTest_HloTest_Test>::CreateTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/internal/gtest-internal.h:456:44 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #21 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a)
    #23 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2875:22 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e1c) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #28 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #29 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #30 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

SUMMARY: ThreadSanitizer: data race /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 in xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::operator()()
==================
==================
WARNING: ThreadSanitizer: data race (pid=1017489)
  Read of size 8 at 0x720c001197f8 by thread T6:
    #0 __gnu_cxx::__normal_iterator<absl::lts_20230802::AnyInvocable<void () &&>*, std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>>::__normal_iterator(absl::lts_20230802::AnyInvocable<void () &&>* const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_iterator.h:1028:20 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb151b) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::end() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:830:16 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb151b)
    #2 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb151b)
    #3 void std::__invoke_impl<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(std::__invoke_other, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb151b)
    #4 std::__invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::__invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb151b)
    #5 std::invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb151b)
    #6 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb151b)
    #7 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb151b)
    #8 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #9 xla::WorkerThread::WorkLoop() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:57:5 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc)
    #10 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:63 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #11 void std::__invoke_impl<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(std::__invoke_other, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #12 std::__invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::__invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #13 std::invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #14 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&, void>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #15 void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #16 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04) (BuildId: 6beeb825755baa193422753e666d20a5)
    #17 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:95:5 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04)

  Previous write of size 8 at 0x720c001197f8 by main thread:
    #0 std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_impl_data::_Vector_impl_data(std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_impl_data&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:103:28 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_impl::_Vector_impl(std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_impl&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:144:36 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #2 std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_Vector_base(std::_Vector_base<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:308:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #3 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::vector(std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:572:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #4 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::$_1($_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #5 void absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::InitializeRemoteManager<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, 0>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:634:34 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #6 void absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::InitializeStorage<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, 0>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:595:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #7 void absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::Initialize<(absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::TargetType)3, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:573:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #8 absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::CoreImpl<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(absl::lts_20230802::internal_any_invocable::TypedConversionConstruct<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&>, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:469:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #9 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::Impl<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(absl::lts_20230802::internal_any_invocable::ConversionConstruct, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #10 absl::lts_20230802::AnyInvocable<void () &&>::AnyInvocable<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/any_invocable.h:186:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #11 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x75629)
    #12 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #13 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #14 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #15 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #16 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #18 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #20 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #21 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #23 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #28 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #29 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #30 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #31 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Location is heap block of size 48 at 0x720c001197f0 allocated by main thread:
    #0 malloc <null> (rand_000041.hlo_test_gpu_amd_any+0xbf50c) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 operator new(unsigned long) <null> (libstdc++.so.6+0xae98b) (BuildId: e37fe1a879783838de78cbc8c80621fa685d58a2)
    #2 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #3 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #4 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #5 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #6 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #7 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #8 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #9 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #10 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #13 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #14 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #15 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #16 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #19 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #20 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #21 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Thread T6 (tid=1017792, running) created by main thread at:
    #0 pthread_create <null> (rand_000041.hlo_test_gpu_amd_any+0xc0e6b) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 tsl::(anonymous namespace)::PThread::PThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:74:15 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691) (BuildId: 6beeb825755baa193422753e666d20a5)
    #2 tsl::(anonymous namespace)::PosixEnv::StartThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:139:16 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691)
    #3 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:12 (libxla_Spjrt_Slibworker_Uthread.so+0x2f03) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #4 std::_MakeUniq<xla::WorkerThread>::__single_object std::make_unique<xla::WorkerThread, tsl::Env*, char const (&) [16]>(tsl::Env*&&, char const (&) [16]) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xe33a) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #5 xla::LocalDeviceState::LocalDeviceState(stream_executor::StreamExecutor*, xla::LocalClient*, xla::LocalDeviceState::AllocationModel, int, bool, bool, int, std::optional<xla::LocalDeviceState::StreamOptions>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/local_device_state.cc:122:7 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xd4c0) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #6 std::_MakeUniq<xla::LocalDeviceState>::__single_object std::make_unique<xla::LocalDeviceState, stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel, int, bool, bool>(stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel&&, int&&, bool&&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35) (BuildId: ca04fde0f1b1bd292e56fb0b668bb930)
    #7 xla::(anonymous namespace)::BuildLocalDeviceStates(xla::LocalClient*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:966:9 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #8 xla::GetStreamExecutorGpuClient(xla::GpuClientOptions const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:1337:3 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #9 xla::(anonymous namespace)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/pjrt_gpu_client_registry.cc:33:32 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c) (BuildId: 043dd621e5330e6517632fc603217b8c)
    #10 absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> std::__invoke_impl<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #11 std::enable_if<is_invocable_r_v<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>, absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>>::type std::__invoke_r<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #12 std::_Function_handler<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> (), xla::(anonymous namespace)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #13 std::function<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5) (BuildId: 2159a5772edbd729fc7fe8982f4b31a7)
    #14 xla::(anonymous namespace)::GetHloRunnerForTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:39:56 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #15 xla::HloPjRtTestBase::HloPjRtTestBase(xla::HloPjRtTestBaseOptions) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:55:33 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #16 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::HloRunnerAgnosticReferenceMixin<>(std::unique_ptr<xla::HloRunnerInterface, std::default_delete<xla::HloRunnerInterface>>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:74:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::HloPjRtInterpreterReferenceMixin<xla::HloPjRtTestBase>::HloPjRtInterpreterReferenceMixin<>() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_pjrt_interpreter_reference_mixin.h:38:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d)
    #18 xla::(anonymous namespace)::HloTest::HloTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:36:7 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::(anonymous namespace)::HloTest_HloTest_Test::HloTest_HloTest_Test() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:38:1 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #20 testing::internal::TestFactoryImpl<xla::(anonymous namespace)::HloTest_HloTest_Test>::CreateTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/internal/gtest-internal.h:456:44 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #21 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a)
    #23 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2875:22 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e1c) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #28 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #29 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #30 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

SUMMARY: ThreadSanitizer: data race /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 in xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::operator()()
==================
==================
WARNING: ThreadSanitizer: data race (pid=1017489)
  Read of size 8 at 0x72100020fe98 by thread T6:
    #0 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::ExtractInvoker() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #2 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #3 void std::__invoke_impl<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(std::__invoke_other, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #4 std::__invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::__invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #5 std::invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #6 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #7 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #8 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #9 xla::WorkerThread::WorkLoop() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:57:5 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc)
    #10 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:63 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #11 void std::__invoke_impl<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(std::__invoke_other, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #12 std::__invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::__invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #13 std::invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #14 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&, void>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #15 void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #16 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04) (BuildId: 6beeb825755baa193422753e666d20a5)
    #17 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:95:5 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04)

  Previous write of size 8 at 0x72100020fe98 by main thread:
    #0 absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::CoreImpl(absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:482:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::Impl(absl::lts_20230802::internal_any_invocable::Impl<void () &&>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:400:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #2 absl::lts_20230802::AnyInvocable<void () &&>::AnyInvocable(absl::lts_20230802::AnyInvocable<void () &&>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/any_invocable.h:176:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #3 void __gnu_cxx::new_allocator<absl::lts_20230802::AnyInvocable<void () &&>>::construct<absl::lts_20230802::AnyInvocable<void () &&>, absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:162:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #4 void std::allocator_traits<std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::construct<absl::lts_20230802::AnyInvocable<void () &&>, absl::lts_20230802::AnyInvocable<void () &&>>(std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>&, absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:516:8 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #5 void std::__relocate_object_a<absl::lts_20230802::AnyInvocable<void () &&>, absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>(absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_uninitialized.h:994:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #6 absl::lts_20230802::AnyInvocable<void () &&>* std::__relocate_a_1<absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>(absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_uninitialized.h:1032:2 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #7 absl::lts_20230802::AnyInvocable<void () &&>* std::__relocate_a<absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>(absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_uninitialized.h:1046:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #8 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_S_do_relocate(absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>&, std::integral_constant<bool, true>) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:456:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #9 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_S_relocate(absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>*, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:469:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #10 void std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_M_realloc_insert<absl::lts_20230802::AnyInvocable<void () &&>>(__gnu_cxx::__normal_iterator<absl::lts_20230802::AnyInvocable<void () &&>*, std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>>, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:461:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0c2f)
    #11 absl::lts_20230802::AnyInvocable<void () &&>& std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::emplace_back<absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:121:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #12 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::push_back(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1204:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #13 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2886:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #14 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #15 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #16 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #17 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #18 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #20 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #21 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #22 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #23 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #25 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #28 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #29 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #30 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #31 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #32 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #33 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Location is heap block of size 64 at 0x72100020fe80 allocated by main thread:
    #0 malloc <null> (rand_000041.hlo_test_gpu_amd_any+0xbf50c) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 operator new(unsigned long) <null> (libstdc++.so.6+0xae98b) (BuildId: e37fe1a879783838de78cbc8c80621fa685d58a2)
    #2 absl::lts_20230802::AnyInvocable<void () &&>& std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::emplace_back<absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:121:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #3 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::push_back(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1204:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #4 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2886:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #5 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #6 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #7 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #8 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #9 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #10 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #11 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #12 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #13 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #14 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #15 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #16 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #17 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #18 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #19 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #20 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #21 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #22 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #23 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #24 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Thread T6 (tid=1017792, running) created by main thread at:
    #0 pthread_create <null> (rand_000041.hlo_test_gpu_amd_any+0xc0e6b) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 tsl::(anonymous namespace)::PThread::PThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:74:15 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691) (BuildId: 6beeb825755baa193422753e666d20a5)
    #2 tsl::(anonymous namespace)::PosixEnv::StartThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:139:16 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691)
    #3 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:12 (libxla_Spjrt_Slibworker_Uthread.so+0x2f03) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #4 std::_MakeUniq<xla::WorkerThread>::__single_object std::make_unique<xla::WorkerThread, tsl::Env*, char const (&) [16]>(tsl::Env*&&, char const (&) [16]) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xe33a) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #5 xla::LocalDeviceState::LocalDeviceState(stream_executor::StreamExecutor*, xla::LocalClient*, xla::LocalDeviceState::AllocationModel, int, bool, bool, int, std::optional<xla::LocalDeviceState::StreamOptions>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/local_device_state.cc:122:7 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xd4c0) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #6 std::_MakeUniq<xla::LocalDeviceState>::__single_object std::make_unique<xla::LocalDeviceState, stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel, int, bool, bool>(stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel&&, int&&, bool&&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35) (BuildId: ca04fde0f1b1bd292e56fb0b668bb930)
    #7 xla::(anonymous namespace)::BuildLocalDeviceStates(xla::LocalClient*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:966:9 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #8 xla::GetStreamExecutorGpuClient(xla::GpuClientOptions const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:1337:3 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #9 xla::(anonymous namespace)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/pjrt_gpu_client_registry.cc:33:32 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c) (BuildId: 043dd621e5330e6517632fc603217b8c)
    #10 absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> std::__invoke_impl<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #11 std::enable_if<is_invocable_r_v<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>, absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>>::type std::__invoke_r<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #12 std::_Function_handler<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> (), xla::(anonymous namespace)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #13 std::function<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5) (BuildId: 2159a5772edbd729fc7fe8982f4b31a7)
    #14 xla::(anonymous namespace)::GetHloRunnerForTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:39:56 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #15 xla::HloPjRtTestBase::HloPjRtTestBase(xla::HloPjRtTestBaseOptions) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:55:33 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #16 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::HloRunnerAgnosticReferenceMixin<>(std::unique_ptr<xla::HloRunnerInterface, std::default_delete<xla::HloRunnerInterface>>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:74:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::HloPjRtInterpreterReferenceMixin<xla::HloPjRtTestBase>::HloPjRtInterpreterReferenceMixin<>() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_pjrt_interpreter_reference_mixin.h:38:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d)
    #18 xla::(anonymous namespace)::HloTest::HloTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:36:7 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::(anonymous namespace)::HloTest_HloTest_Test::HloTest_HloTest_Test() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:38:1 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #20 testing::internal::TestFactoryImpl<xla::(anonymous namespace)::HloTest_HloTest_Test>::CreateTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/internal/gtest-internal.h:456:44 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #21 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a)
    #23 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2875:22 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e1c) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #28 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #29 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #30 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

SUMMARY: ThreadSanitizer: data race /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 in absl::lts_20230802::internal_any_invocable::Impl<void () &&>::ExtractInvoker()
==================
==================
WARNING: ThreadSanitizer: data race (pid=1017489)
  Read of size 8 at 0x72100020feb8 by thread T6:
    #0 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::ExtractInvoker() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #2 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #3 void std::__invoke_impl<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(std::__invoke_other, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #4 std::__invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::__invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #5 std::invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #6 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #7 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb153b)
    #8 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #9 xla::WorkerThread::WorkLoop() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:57:5 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc)
    #10 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:63 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #11 void std::__invoke_impl<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(std::__invoke_other, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #12 std::__invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::__invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #13 std::invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #14 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&, void>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #15 void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #16 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04) (BuildId: 6beeb825755baa193422753e666d20a5)
    #17 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:95:5 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04)

  Previous write of size 8 at 0x72100020feb8 by main thread:
    #0 absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::CoreImpl(absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:482:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0bab) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::Impl(absl::lts_20230802::internal_any_invocable::Impl<void () &&>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:400:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0bab)
    #2 absl::lts_20230802::AnyInvocable<void () &&>::AnyInvocable(absl::lts_20230802::AnyInvocable<void () &&>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/any_invocable.h:176:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0bab)
    #3 void __gnu_cxx::new_allocator<absl::lts_20230802::AnyInvocable<void () &&>>::construct<absl::lts_20230802::AnyInvocable<void () &&>, absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:162:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0bab)
    #4 void std::allocator_traits<std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::construct<absl::lts_20230802::AnyInvocable<void () &&>, absl::lts_20230802::AnyInvocable<void () &&>>(std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>&, absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:516:8 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0bab)
    #5 void std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_M_realloc_insert<absl::lts_20230802::AnyInvocable<void () &&>>(__gnu_cxx::__normal_iterator<absl::lts_20230802::AnyInvocable<void () &&>*, std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>>, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:449:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0bab)
    #6 absl::lts_20230802::AnyInvocable<void () &&>& std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::emplace_back<absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:121:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #7 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::push_back(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1204:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #8 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2886:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #9 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #10 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #11 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #12 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #13 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #14 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #15 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #16 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #18 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #19 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #20 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #21 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #23 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #26 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #28 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Location is heap block of size 64 at 0x72100020fe80 allocated by main thread:
    #0 malloc <null> (rand_000041.hlo_test_gpu_amd_any+0xbf50c) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 operator new(unsigned long) <null> (libstdc++.so.6+0xae98b) (BuildId: e37fe1a879783838de78cbc8c80621fa685d58a2)
    #2 absl::lts_20230802::AnyInvocable<void () &&>& std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::emplace_back<absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:121:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #3 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::push_back(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1204:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #4 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2886:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #5 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #6 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #7 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #8 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #9 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #10 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #11 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #12 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #13 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #14 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #15 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #16 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #17 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #18 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #19 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #20 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #21 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #22 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #23 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #24 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Thread T6 (tid=1017792, running) created by main thread at:
    #0 pthread_create <null> (rand_000041.hlo_test_gpu_amd_any+0xc0e6b) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 tsl::(anonymous namespace)::PThread::PThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:74:15 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691) (BuildId: 6beeb825755baa193422753e666d20a5)
    #2 tsl::(anonymous namespace)::PosixEnv::StartThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:139:16 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691)
    #3 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:12 (libxla_Spjrt_Slibworker_Uthread.so+0x2f03) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #4 std::_MakeUniq<xla::WorkerThread>::__single_object std::make_unique<xla::WorkerThread, tsl::Env*, char const (&) [16]>(tsl::Env*&&, char const (&) [16]) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xe33a) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #5 xla::LocalDeviceState::LocalDeviceState(stream_executor::StreamExecutor*, xla::LocalClient*, xla::LocalDeviceState::AllocationModel, int, bool, bool, int, std::optional<xla::LocalDeviceState::StreamOptions>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/local_device_state.cc:122:7 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xd4c0) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #6 std::_MakeUniq<xla::LocalDeviceState>::__single_object std::make_unique<xla::LocalDeviceState, stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel, int, bool, bool>(stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel&&, int&&, bool&&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35) (BuildId: ca04fde0f1b1bd292e56fb0b668bb930)
    #7 xla::(anonymous namespace)::BuildLocalDeviceStates(xla::LocalClient*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:966:9 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #8 xla::GetStreamExecutorGpuClient(xla::GpuClientOptions const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:1337:3 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #9 xla::(anonymous namespace)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/pjrt_gpu_client_registry.cc:33:32 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c) (BuildId: 043dd621e5330e6517632fc603217b8c)
    #10 absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> std::__invoke_impl<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #11 std::enable_if<is_invocable_r_v<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>, absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>>::type std::__invoke_r<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #12 std::_Function_handler<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> (), xla::(anonymous namespace)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #13 std::function<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5) (BuildId: 2159a5772edbd729fc7fe8982f4b31a7)
    #14 xla::(anonymous namespace)::GetHloRunnerForTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:39:56 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #15 xla::HloPjRtTestBase::HloPjRtTestBase(xla::HloPjRtTestBaseOptions) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:55:33 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #16 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::HloRunnerAgnosticReferenceMixin<>(std::unique_ptr<xla::HloRunnerInterface, std::default_delete<xla::HloRunnerInterface>>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:74:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::HloPjRtInterpreterReferenceMixin<xla::HloPjRtTestBase>::HloPjRtInterpreterReferenceMixin<>() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_pjrt_interpreter_reference_mixin.h:38:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d)
    #18 xla::(anonymous namespace)::HloTest::HloTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:36:7 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::(anonymous namespace)::HloTest_HloTest_Test::HloTest_HloTest_Test() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:38:1 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #20 testing::internal::TestFactoryImpl<xla::(anonymous namespace)::HloTest_HloTest_Test>::CreateTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/internal/gtest-internal.h:456:44 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #21 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a)
    #23 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2875:22 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e1c) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #28 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #29 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #30 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

SUMMARY: ThreadSanitizer: data race /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 in absl::lts_20230802::internal_any_invocable::Impl<void () &&>::ExtractInvoker()
==================
==================
WARNING: ThreadSanitizer: data race (pid=1017489)
  Read of size 8 at 0x72100020fea0 by thread T6:
    #0 tsl::RCReference<tsl::AsyncValue>::operator->() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/ref_count.h:179:12 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 void tsl::AsyncValueRef<absl::lts_20230802::Status>::emplace<absl::lts_20230802::Status>(absl::lts_20230802::Status&&) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value_ref.h:322:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #2 void xla::internal::PjRtFutureBase<absl::lts_20230802::Status, false>::Promise::emplace<absl::lts_20230802::Status>(absl::lts_20230802::Status&&) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/pjrt/pjrt_future.h:227:25 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #3 xla::PjRtFuture<void>::Promise::Set(absl::lts_20230802::Status) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/pjrt/pjrt_future.h:475:22 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #4 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2887:60 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #5 void std::__invoke_impl<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>(std::__invoke_other, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #6 std::__invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>::type std::__invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #7 std::invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>::type std::invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #8 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #9 void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1419)
    #10 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #11 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #12 void std::__invoke_impl<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(std::__invoke_other, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #13 std::__invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::__invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #14 std::invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #15 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #16 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #17 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #18 xla::WorkerThread::WorkLoop() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:57:5 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc)
    #19 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:63 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #20 void std::__invoke_impl<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(std::__invoke_other, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #21 std::__invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::__invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #22 std::invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #23 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&, void>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #24 void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #25 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04) (BuildId: 6beeb825755baa193422753e666d20a5)
    #26 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:95:5 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04)

  Previous write of size 8 at 0x72100020fea0 by main thread:
    #0 tsl::RCReference<tsl::AsyncValue>::RCReference(tsl::RCReference<tsl::AsyncValue>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/ref_count.h:127:47 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb14ab) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 tsl::AsyncValueRef<absl::lts_20230802::Status>::AsyncValueRef(tsl::AsyncValueRef<absl::lts_20230802::Status>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value_ref.h:146:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb14ab)
    #2 xla::internal::PjRtFutureBase<absl::lts_20230802::Status, false>::Promise::Promise(xla::internal::PjRtFutureBase<absl::lts_20230802::Status, false>::Promise&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/pjrt/pjrt_future.h:212:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb14ab)
    #3 xla::PjRtFuture<void>::Promise::Promise(xla::PjRtFuture<void>::Promise&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/pjrt/pjrt_future.h:464:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb14ab)
    #4 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0::$_0($_0&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2887:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb14ab)
    #5 void absl::lts_20230802::internal_any_invocable::LocalManagerNontrivial<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>(absl::lts_20230802::internal_any_invocable::FunctionToCall, absl::lts_20230802::internal_any_invocable::TypeErasedState*, absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:288:48 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb14ab)
    #6 absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>::CoreImpl(absl::lts_20230802::internal_any_invocable::CoreImpl<false, void>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:481:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0b8d) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #7 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::Impl(absl::lts_20230802::internal_any_invocable::Impl<void () &&>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:400:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0b8d)
    #8 absl::lts_20230802::AnyInvocable<void () &&>::AnyInvocable(absl::lts_20230802::AnyInvocable<void () &&>&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/any_invocable.h:176:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0b8d)
    #9 void __gnu_cxx::new_allocator<absl::lts_20230802::AnyInvocable<void () &&>>::construct<absl::lts_20230802::AnyInvocable<void () &&>, absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:162:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0b8d)
    #10 void std::allocator_traits<std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::construct<absl::lts_20230802::AnyInvocable<void () &&>, absl::lts_20230802::AnyInvocable<void () &&>>(std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>&, absl::lts_20230802::AnyInvocable<void () &&>*, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:516:8 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0b8d)
    #11 void std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::_M_realloc_insert<absl::lts_20230802::AnyInvocable<void () &&>>(__gnu_cxx::__normal_iterator<absl::lts_20230802::AnyInvocable<void () &&>*, std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>>, absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:449:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb0b8d)
    #12 absl::lts_20230802::AnyInvocable<void () &&>& std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::emplace_back<absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:121:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #13 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::push_back(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1204:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #14 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2886:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #15 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #16 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #17 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #18 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #19 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #20 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #21 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #22 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #23 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #24 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #26 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #28 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #29 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #30 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #31 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #32 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #33 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #34 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Location is heap block of size 64 at 0x72100020fe80 allocated by main thread:
    #0 malloc <null> (rand_000041.hlo_test_gpu_amd_any+0xbf50c) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 operator new(unsigned long) <null> (libstdc++.so.6+0xae98b) (BuildId: e37fe1a879783838de78cbc8c80621fa685d58a2)
    #2 absl::lts_20230802::AnyInvocable<void () &&>& std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::emplace_back<absl::lts_20230802::AnyInvocable<void () &&>>(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:121:4 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #3 std::vector<absl::lts_20230802::AnyInvocable<void () &&>, std::allocator<absl::lts_20230802::AnyInvocable<void () &&>>>::push_back(absl::lts_20230802::AnyInvocable<void () &&>&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1204:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #4 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2886:23 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7553a)
    #5 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #6 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #7 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #8 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #9 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #10 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #11 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #12 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #13 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #14 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #15 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #16 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #17 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #18 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #19 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #20 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #21 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #22 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #23 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #24 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Thread T6 (tid=1017792, running) created by main thread at:
    #0 pthread_create <null> (rand_000041.hlo_test_gpu_amd_any+0xc0e6b) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 tsl::(anonymous namespace)::PThread::PThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:74:15 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691) (BuildId: 6beeb825755baa193422753e666d20a5)
    #2 tsl::(anonymous namespace)::PosixEnv::StartThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:139:16 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691)
    #3 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:12 (libxla_Spjrt_Slibworker_Uthread.so+0x2f03) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #4 std::_MakeUniq<xla::WorkerThread>::__single_object std::make_unique<xla::WorkerThread, tsl::Env*, char const (&) [16]>(tsl::Env*&&, char const (&) [16]) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xe33a) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #5 xla::LocalDeviceState::LocalDeviceState(stream_executor::StreamExecutor*, xla::LocalClient*, xla::LocalDeviceState::AllocationModel, int, bool, bool, int, std::optional<xla::LocalDeviceState::StreamOptions>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/local_device_state.cc:122:7 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xd4c0) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #6 std::_MakeUniq<xla::LocalDeviceState>::__single_object std::make_unique<xla::LocalDeviceState, stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel, int, bool, bool>(stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel&&, int&&, bool&&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35) (BuildId: ca04fde0f1b1bd292e56fb0b668bb930)
    #7 xla::(anonymous namespace)::BuildLocalDeviceStates(xla::LocalClient*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:966:9 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #8 xla::GetStreamExecutorGpuClient(xla::GpuClientOptions const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:1337:3 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #9 xla::(anonymous namespace)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/pjrt_gpu_client_registry.cc:33:32 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c) (BuildId: 043dd621e5330e6517632fc603217b8c)
    #10 absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> std::__invoke_impl<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #11 std::enable_if<is_invocable_r_v<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>, absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>>::type std::__invoke_r<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #12 std::_Function_handler<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> (), xla::(anonymous namespace)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #13 std::function<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5) (BuildId: 2159a5772edbd729fc7fe8982f4b31a7)
    #14 xla::(anonymous namespace)::GetHloRunnerForTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:39:56 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #15 xla::HloPjRtTestBase::HloPjRtTestBase(xla::HloPjRtTestBaseOptions) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:55:33 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #16 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::HloRunnerAgnosticReferenceMixin<>(std::unique_ptr<xla::HloRunnerInterface, std::default_delete<xla::HloRunnerInterface>>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:74:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::HloPjRtInterpreterReferenceMixin<xla::HloPjRtTestBase>::HloPjRtInterpreterReferenceMixin<>() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_pjrt_interpreter_reference_mixin.h:38:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d)
    #18 xla::(anonymous namespace)::HloTest::HloTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:36:7 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::(anonymous namespace)::HloTest_HloTest_Test::HloTest_HloTest_Test() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:38:1 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #20 testing::internal::TestFactoryImpl<xla::(anonymous namespace)::HloTest_HloTest_Test>::CreateTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/internal/gtest-internal.h:456:44 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #21 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a)
    #23 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2875:22 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e1c) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #28 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #29 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #30 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

SUMMARY: ThreadSanitizer: data race /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/ref_count.h:179:12 in tsl::RCReference<tsl::AsyncValue>::operator->() const
==================
==================
WARNING: ThreadSanitizer: data race (pid=1017489)
  Atomic write of size 8 at 0x722000012808 by thread T6:
    #0 std::atomic<tsl::AsyncValue::WaitersAndState>::exchange(tsl::AsyncValue::WaitersAndState, std::memory_order) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:303:2 (libxla_Stsl_Sconcurrency_Slibasync_Uvalue.so+0x54ab) (BuildId: 17a156856d29248deba8f833232b4829)
    #1 tsl::AsyncValue::NotifyAvailable(tsl::AsyncValue::State) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/concurrency/async_value.cc:64:47 (libxla_Stsl_Sconcurrency_Slibasync_Uvalue.so+0x54ab)
    #2 void tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>::emplace<absl::lts_20230802::Status>(absl::lts_20230802::Status&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value.h:628:5 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #3 void tsl::AsyncValue::emplace<absl::lts_20230802::Status, absl::lts_20230802::Status>(absl::lts_20230802::Status&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value.h:980:56 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #4 void tsl::AsyncValueRef<absl::lts_20230802::Status>::emplace<absl::lts_20230802::Status>(absl::lts_20230802::Status&&) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value_ref.h:322:13 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #5 void xla::internal::PjRtFutureBase<absl::lts_20230802::Status, false>::Promise::emplace<absl::lts_20230802::Status>(absl::lts_20230802::Status&&) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/pjrt/pjrt_future.h:227:25 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #6 xla::PjRtFuture<void>::Promise::Set(absl::lts_20230802::Status) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/pjrt/pjrt_future.h:475:22 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #7 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2887:60 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #8 void std::__invoke_impl<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>(std::__invoke_other, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #9 std::__invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>::type std::__invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #10 std::invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>::type std::invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #11 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #12 void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_0&&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb144b)
    #13 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #14 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2889:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #15 void std::__invoke_impl<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(std::__invoke_other, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #16 std::__invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::__invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #17 std::invoke_result<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>::type std::invoke<xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #18 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1, void>(xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #19 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const::$_1&&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0xb1541)
    #20 absl::lts_20230802::internal_any_invocable::Impl<void () &&>::operator()() && /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:876:1 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #21 xla::WorkerThread::WorkLoop() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:57:5 (libxla_Spjrt_Slibworker_Uthread.so+0x34bc)
    #22 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:63 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #23 void std::__invoke_impl<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(std::__invoke_other, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #24 std::__invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::__invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #25 std::invoke_result<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>::type std::invoke<xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:97:14 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #26 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&, void>(xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #27 void absl::lts_20230802::internal_any_invocable::LocalInvoker<false, void, xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (libxla_Spjrt_Slibworker_Uthread.so+0x3d01)
    #28 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04) (BuildId: 6beeb825755baa193422753e666d20a5)
    #29 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:95:5 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x11d04)

  Previous write of size 8 at 0x722000012808 by main thread:
    #0 std::atomic<tsl::AsyncValue::WaitersAndState>::atomic(tsl::AsyncValue::WaitersAndState) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:231:44 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7523c) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #1 tsl::AsyncValue::AsyncValue<absl::lts_20230802::Status>(tsl::AsyncValue::Kind, tsl::AsyncValue::State, bool, tsl::AsyncValue::TypeTag<absl::lts_20230802::Status>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value.h:292:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7523c)
    #2 tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>::ConcreteAsyncValue(tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>::UnconstructedPayload) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value.h:568:9 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7523c)
    #3 tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>* tsl::internal::PlacementConstruct<tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>, tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>::UnconstructedPayload>(void*, tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>::UnconstructedPayload&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value_ref.h:1056:20 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7523c)
    #4 tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>* tsl::internal::AllocateAndConstruct<tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>, tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>::UnconstructedPayload>(tsl::internal::ConcreteAsyncValue<absl::lts_20230802::Status>::UnconstructedPayload&&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value_ref.h:1062:10 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7523c)
    #5 tsl::AsyncValueRef<absl::lts_20230802::Status> tsl::MakeUnconstructedAsyncValueRef<absl::lts_20230802::Status>() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tsl/concurrency/async_value_ref.h:1079:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7523c)
    #6 xla::PjRtFuture<void>::CreatePromise() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/pjrt/pjrt_future.h:485:20 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7523c)
    #7 xla::PjRtStreamExecutorLoadedExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, bool, xla::PjRtDevice*) const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:2884:20 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7523c)
    #8 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #9 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #10 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #11 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #12 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #13 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #14 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #15 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #16 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #18 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #19 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #20 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #21 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #23 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #25 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #27 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Location is heap block of size 128 at 0x722000012800 allocated by main thread:
    #0 aligned_alloc <null> (rand_000041.hlo_test_gpu_amd_any+0xc0420) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 operator new(unsigned long, std::align_val_t) <null> (libstdc++.so.6+0xaea45) (BuildId: e37fe1a879783838de78cbc8c80621fa685d58a2)
    #2 xla::PjRtStreamExecutorLoadedExecutable::ExecuteSharded(absl::lts_20230802::Span<xla::PjRtBuffer* const>, xla::PjRtDevice*, xla::ExecuteOptions const&, std::optional<xla::PjRtFuture<void>>&, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/pjrt_stream_executor_client.cc:3073:7 (libxla_Spjrt_Slibpjrt_Ustream_Uexecutor_Uclient.so+0x7aa14) (BuildId: 3f7e785dbffebb47e4da86c9961147bd)
    #3 xla::HloRunnerPjRt::ExecuteWithDeviceBuffers(xla::PjRtLoadedExecutable*, xla::ExecuteOptions const&, std::vector<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>, std::allocator<std::unique_ptr<xla::PjRtBuffer, std::default_delete<xla::PjRtBuffer>>>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:400:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x202e2) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #4 xla::HloRunnerPjRt::ExecuteWithExecutable(xla::OpaqueExecutable*, absl::lts_20230802::Span<xla::Literal const* const>, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:458:3 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x21a6a) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #5 xla::HloRunnerPjRt::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool, xla::ExecutionProfile*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/service/hlo_runner_pjrt.cc:379:10 (libxla_Sservice_Slibhlo_Urunner_Upjrt.so+0x1fa08) (BuildId: 10b96d38733bff4b1d593636cb4a2b29)
    #6 xla::HloRunnerInterface::Execute(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, bool) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/service/hlo_runner_interface.h:244:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #7 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompareInternal(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, bool, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:241:5 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x13808)
    #8 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, absl::lts_20230802::Span<xla::Literal const* const>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:96:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1318c) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #9 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::RunAndCompare(std::unique_ptr<xla::HloModule, std::default_delete<xla::HloModule>>, std::optional<xla::ErrorSpec> const&, std::function<void (xla::HloModule*)> const&, std::function<void (xla::HloModule*)> const&, std::optional<long>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:142:12 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x11d35) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #10 xla::(anonymous namespace)::HloTest_HloTest_Test::TestBody() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:48:3 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf956) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #11 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #12 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f19a)
    #13 testing::Test::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2739:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x7f00f) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #14 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2885:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e39) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #15 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #16 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #19 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #20 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #21 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

  Thread T6 (tid=1017792, running) created by main thread at:
    #0 pthread_create <null> (rand_000041.hlo_test_gpu_amd_any+0xc0e6b) (BuildId: 4052c2b776e4ea792c7ebb55a9e448cd)
    #1 tsl::(anonymous namespace)::PThread::PThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:74:15 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691) (BuildId: 6beeb825755baa193422753e666d20a5)
    #2 tsl::(anonymous namespace)::PosixEnv::StartThread(tsl::ThreadOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, absl::lts_20230802::AnyInvocable<void ()>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/platform/default/env.cc:139:16 (libxla_Stsl_Splatform_Sdefault_Slibenv_Uimpl.so+0x10691)
    #3 xla::WorkerThread::WorkerThread(tsl::Env*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/worker_thread.cc:29:12 (libxla_Spjrt_Slibworker_Uthread.so+0x2f03) (BuildId: ad1ac031e164b108ab83462f33a5360f)
    #4 std::_MakeUniq<xla::WorkerThread>::__single_object std::make_unique<xla::WorkerThread, tsl::Env*, char const (&) [16]>(tsl::Env*&&, char const (&) [16]) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xe33a) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #5 xla::LocalDeviceState::LocalDeviceState(stream_executor::StreamExecutor*, xla::LocalClient*, xla::LocalDeviceState::AllocationModel, int, bool, bool, int, std::optional<xla::LocalDeviceState::StreamOptions>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/local_device_state.cc:122:7 (libxla_Spjrt_Sliblocal_Udevice_Ustate.so+0xd4c0) (BuildId: 136538cc6c995586c0bd2d0b124ee2c1)
    #6 std::_MakeUniq<xla::LocalDeviceState>::__single_object std::make_unique<xla::LocalDeviceState, stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel, int, bool, bool>(stream_executor::StreamExecutor*&, xla::LocalClient*&, xla::LocalDeviceState::AllocationModel&&, int&&, bool&&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:962:34 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35) (BuildId: ca04fde0f1b1bd292e56fb0b668bb930)
    #7 xla::(anonymous namespace)::BuildLocalDeviceStates(xla::LocalClient*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:966:9 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #8 xla::GetStreamExecutorGpuClient(xla::GpuClientOptions const&) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/pjrt/gpu/se_gpu_pjrt_client.cc:1337:3 (libxla_Spjrt_Sgpu_Slibse_Ugpu_Upjrt_Uclient.so+0x3ed35)
    #9 xla::(anonymous namespace)::$_0::operator()() const /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/pjrt_gpu_client_registry.cc:33:32 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c) (BuildId: 043dd621e5330e6517632fc603217b8c)
    #10 absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> std::__invoke_impl<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #11 std::enable_if<is_invocable_r_v<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>, absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>>::type std::__invoke_r<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>>, xla::(anonymous namespace)::$_0&>(xla::(anonymous namespace)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #12 std::_Function_handler<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> (), xla::(anonymous namespace)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:290:9 (libxla_Stests_Slibpjrt_Ugpu_Uclient_Uregistry.so+0x9f4c)
    #13 std::function<absl::lts_20230802::StatusOr<std::unique_ptr<xla::PjRtClient, std::default_delete<xla::PjRtClient>>> ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:590:9 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5) (BuildId: 2159a5772edbd729fc7fe8982f4b31a7)
    #14 xla::(anonymous namespace)::GetHloRunnerForTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:39:56 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #15 xla::HloPjRtTestBase::HloPjRtTestBase(xla::HloPjRtTestBaseOptions) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/hlo_pjrt_test_base.cc:55:33 (libxla_Stests_Slibhlo_Upjrt_Utest_Ubase.so+0x93d5)
    #16 xla::HloRunnerAgnosticReferenceMixin<xla::HloPjRtTestBase>::HloRunnerAgnosticReferenceMixin<>(std::unique_ptr<xla::HloRunnerInterface, std::default_delete<xla::HloRunnerInterface>>) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_runner_agnostic_reference_mixin.h:74:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #17 xla::HloPjRtInterpreterReferenceMixin<xla::HloPjRtTestBase>::HloPjRtInterpreterReferenceMixin<>() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/./xla/tests/hlo_pjrt_interpreter_reference_mixin.h:38:9 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0x1000d)
    #18 xla::(anonymous namespace)::HloTest::HloTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:36:7 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4) (BuildId: 32f7c28c0836e5585d16d1d1abea171e)
    #19 xla::(anonymous namespace)::HloTest_HloTest_Test::HloTest_HloTest_Test() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tests/fuzz/hlo_test_lib.cc:38:1 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #20 testing::internal::TestFactoryImpl<xla::(anonymous namespace)::HloTest_HloTest_Test>::CreateTest() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/internal/gtest-internal.h:456:44 (libxla_Stests_Sfuzz_Slibhlo_Utest_Ulib.so+0xf1d4)
    #21 testing::Test* testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #22 testing::Test* testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::TestFactoryBase, testing::Test*>(testing::internal::TestFactoryBase*, testing::Test* (testing::internal::TestFactoryBase::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x8138a)
    #23 testing::TestInfo::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2875:22 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x80e1c) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #24 testing::TestSuite::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:3063:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x82ab4) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #25 testing::internal::UnitTestImpl::RunAllTests() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:6054:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9e9b6) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #26 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2664:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #27 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:2700:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d8fa)
    #28 testing::UnitTest::Run() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/src/gtest.cc:5594:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x9d681) (BuildId: 808b8a1db15382517b1a68f57e94094c)
    #29 RUN_ALL_TESTS() /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googletest/include/gtest/gtest.h:2334:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7) (BuildId: 92298fb67d5b450298ac67813241a877)
    #30 main /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x1bc7)

SUMMARY: ThreadSanitizer: data race /root/.cache/bazel/_bazel_root/f367074f9120c6f1a67d35844ac058a3/execroot/xla/xla/tsl/concurrency/async_value.cc:64:47 in tsl::AsyncValue::NotifyAvailable(tsl::AsyncValue::State)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alekstheod