🧑‍🌾 Segmentation fault in rclpy `test_client` and `test_service`

[Crola1702]

Generated by Generative AI

Using Claude 4

Operating System:

Ubuntu Noble

ROS version or commit hash:

Rolling

RMW implementation (if applicable):

rmw_connextdds

RMW Configuration (if applicable):

No response

Client library (if applicable):

rclpy

'ros2 doctor --report' output

ros2 doctor --report

<COPY OUTPUT HERE>

Steps to reproduce issue

1. Run a build in Rolling Nightly Connext
2. See test regression fail

Expected behavior

Test should pass

Actual behavior

Test is failing

Additional information

Reference builds:

• https://build.ros2.org/job/Rci__nightly-connext_ubuntu_noble_amd64/537/

Test regressions:

• rclpy.test_client.xunit.missing_result
• rclpy.test_service.xunit.missing_result

Flaky ratio:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
Rci__nightly-connext_ubuntu_noble_amd64	2025-09-26	2025-09-16	12	10	83.33

Log output:

Fatal Python error: Segmentation fault

Thread 0x00007902f6b20080 (most recent call first):
  File "/tmp/ws/src/ros2/rclpy/rclpy/rclpy/service.py", line 130 in destroy
  File "/tmp/ws/src/ros2/rclpy/rclpy/rclpy/node.py", line 1969 in destroy_service
  File "/tmp/ws/src/ros2/rclpy/rclpy/test/test_client.py", line 151 in test_concurrent_calls_to_service

Root Cause Location

• File: rclpy/service.py
• Line: 130 in destroy() method
• Context: Called from node.py:1969 in destroy_service()
• Test: test_client.py:151 in test_concurrent_calls_to_service

[sloretz]

Attempting to reproduce with:

source /opt/rti.com/rti_connext_dds-7.3.0/resource/scripts/rtisetenv_x64Linux4gcc7.3.0.bash
colcon build --packages-up-to rclpy --packages-ignore-regex .*cyclonedds.* rmw_fastrtps.* .*zenoh*. --cmake-args -DCMAKE_BUILD_TYPE=Debug

Followed by

colcon test --packages-select rclpy --packages-ignore-regex .*cyclonedds.* rmw_fastrtps.* .*zenoh*. --ctest-args -R test_client --retest-until-fail 1000

I see test_client timeout (not segfault), and it also times out on CI https://build.ros2.org/job/Rci__nightly-connext_ubuntu_noble_amd64/544 .

Attempting to get more info by running pytest directly.

Looks like test_on_new_response_callback is hanging.

rclpy/rclpy/test/test_client.py

Lines 299 to 319 in b99fc95

	def test_on_new_response_callback(self) -> None:
	def _service(request, response):
	return response
	with self.node.create_client(Empty, '/service') as cli:
	with self.node.create_service(Empty, '/service', _service):
	executor = rclpy.executors.SingleThreadedExecutor(context=self.context)
	try:
	self.assertTrue(cli.wait_for_service(timeout_sec=20))
	executor.add_node(self.node)
	cb = Mock()
	cli.handle.set_on_new_response_callback(cb)
	cb.assert_not_called()
	cli.call_async(Empty.Request())
	executor.spin_once(0)
	cb.assert_called_once_with(1)
	cli.handle.clear_on_new_response_callback()
	cli.call_async(Empty.Request())
	executor.spin_once(0)
	cb.assert_called_once()
	finally:
	executor.shutdown()

ros@5eb8ebc2a380:/workspaces/ros2$ PYTHONPATH=$PYTHONPATH:/workspaces/ros2/build/rclpy pytest /workspaces/ros2/src/ros2/rclpy/rclpy/test/test_client.py -s -vv
================================================================= test session starts ==================================================================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /workspaces/ros2/src/ros2/rclpy
configfile: pytest.ini
plugins: launch-testing-3.9.2, ament-xmllint-0.20.2, ament-pep257-0.20.2, ament-mypy-0.20.2, ament-flake8-0.20.2, ament-copyright-0.20.2, ament-lint-0.20.2, repeat-0.9.3, rerunfailures-12.0, cov-4.1.0, colcon-core-0.20.0, timeout-2.2.0, mock-3.12.0
collected 13 items                                                                                                                                     

src/ros2/rclpy/rclpy/test/test_client.py::TestClient::test_concurrent_calls_to_service RTI Connext DDS Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User license@rti.com For non-production use only.
Expires on 00-jan-00
Please contact support@rti.com with any questions or comments.
PASSED
src/ros2/rclpy/rclpy/test/test_client.py::TestClient::test_different_type_raises PASSED
src/ros2/rclpy/rclpy/test/test_client.py::TestClient::test_get_service_name PASSED
src/ros2/rclpy/rclpy/test/test_client.py::TestClient::test_get_service_name_after_remapping PASSED
src/ros2/rclpy/rclpy/test/test_client.py::TestClient::test_logger_name_is_equal_to_node_name PASSED
src/ros2/rclpy/rclpy/test/test_client.py::TestClient::test_on_new_response_callback 

Seems like it hangs waiting for the service to shut down

#!/usr/bin/env python3

from unittest.mock import Mock

import rclpy
from test_msgs.srv import Empty

def _service(request, response):
            return response

context = rclpy.context.Context()
rclpy.init(context=context)
node = rclpy.create_node('TestClient', context=context)
        
with node.create_client(Empty, '/service') as cli:
    with node.create_service(Empty, '/service', _service):
        executor = rclpy.executors.SingleThreadedExecutor(context=context)
        try:
            print("Waiting for service")
            assert cli.wait_for_service(timeout_sec=20)
            print("Adding node")
            executor.add_node(node)
            cb = Mock()
            cli.handle.set_on_new_response_callback(cb)
            cb.assert_not_called()
            
            print("calling client")
            cli.call_async(Empty.Request())
            print("Waiting for executor to handle client call")
            executor.spin_once(0)
            cb.assert_called_once_with(1)
            cli.handle.clear_on_new_response_callback()

            print("calling client again")
            cli.call_async(Empty.Request())
            print("Waiting on executor again")
            executor.spin_once(0)
            cb.assert_called_once()
        finally:
            print("Shutting down executor")
            executor.shutdown()
            print("Executor is shut down")
    print("Service is shut down")
print("Node is shut down")

ros@5eb8ebc2a380:/workspaces/ros2$ ./rclpy1508.py 
RTI Connext DDS Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User license@rti.com For non-production use only.
Expires on 00-jan-00
Please contact support@rti.com with any questions or comments.
Waiting for service
Adding node
calling client
Waiting for executor to handle client call
Shutting down executor
Executor is shut down

Seems Connextdds is stuck waiting to take a semaphore

(gdb) bt
#0  futex_wait (private=0, expected=2, futex_word=0x40f33528) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x40f33528, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007fc6b267a101 in lll_mutex_lock_optimized (mutex=0x40f33528) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x40f33528) at ./nptl/pthread_mutex_lock.c:93
#4  0x00007fc6af10ff33 in RTIOsapiSemaphore_takeWithLogging (me=0x40f33520, blockDurationIn=0x0, loggingAvailable=1)
    at /rti/jenkins/workspace/connextdds/release7.3.0.0/x64Linux4gcc7.3.0/src/osapi.1.0/srcC/concurrency/Semaphore.c:765
#5  0x00007fc6af111773 in RTIOsapiSemaphore_take (me=0x40f33520, blockDurationIn=0x0)
    at /rti/jenkins/workspace/connextdds/release7.3.0.0/x64Linux4gcc7.3.0/src/osapi.1.0/srcC/concurrency/Semaphore.c:1392
#6  0x00007fc6af21cd0a in REDAWorker_enterExclusiveArea (worker=0x4098aaa0, failStatus=0x0, exclusiveArea=0x40f334c0)
    at /rti/jenkins/workspace/connextdds/release7.3.0.0/x64Linux4gcc7.3.0/src/reda.1.0/srcC/worker/Worker.c:1189
#7  0x00007fc6af210788 in REDACursor_modifyReadWriteArea (c=0x40f3bb60, fr=0x0)
    at /rti/jenkins/workspace/connextdds/release7.3.0.0/x64Linux4gcc7.3.0/src/reda.1.0/srcC/table/Cursor.c:425
#8  0x00007fc6af86a402 in PRESPsService_destroyLocalEndpoint (me=0x40d12140, failReason=0x7fff229128b4, group=0x40edce68, endpoint=0x41913700, 
    worker=0x4098aaa0) at /rti/jenkins/workspace/connextdds/release7.3.0.0/x64Linux4gcc7.3.0/src/pres.1.0/srcC/psService/PsService.c:2648
#9  0x00007fc6af713e88 in PRESParticipant_destroyLocalEndpoint (me=0x40d13050, failReason=0x7fff229128b4, group=0x40edce68, endpoint=0x41913700, 
    worker=0x4098aaa0) at /rti/jenkins/workspace/connextdds/release7.3.0.0/x64Linux4gcc7.3.0/src/pres.1.0/srcC/participant/Participant.c:9449
#10 0x00007fc6b0501176 in DDS_DataReader_deleteI (reader=0x41913978)
    at /rti/jenkins/workspace/connextdds/release7.3.0.0/x64Linux4gcc7.3.0/src/dds_c.1.0/srcC/subscription/DataReader.c:5311
#11 0x00007fc6b052d62c in DDS_Subscriber_delete_datareader (self=0x40edd160, reader=0x41913978)
    at /rti/jenkins/workspace/connextdds/release7.3.0.0/x64Linux4gcc7.3.0/src/dds_c.1.0/srcC/subscription/Subscriber.c:1399
#12 0x00007fc6b0d53ecb in RMW_Connext_Subscriber::finalize (this=0x4199d770)
    at /workspaces/ros2/src/ros2/rmw_connextdds/rmw_connextdds_common/src/common/rmw_impl.cpp:1539
#13 0x00007fc6b0d5d189 in RMW_Connext_Service::finalize (this=0x416954a0)
    at /workspaces/ros2/src/ros2/rmw_connextdds/rmw_connextdds_common/src/common/rmw_impl.cpp:3342
#14 0x00007fc6b0d7ed89 in rmw_api_connextdds_destroy_service (node=0x40dabb10, service=0x41976340)
    at /workspaces/ros2/src/ros2/rmw_connextdds/rmw_connextdds_common/src/common/rmw_service.cpp:529
#15 0x00007fc6b163f7fc in rmw_destroy_service (node=0x40dabb10, service=0x41976340)
    at /workspaces/ros2/src/ros2/rmw_connextdds/rmw_connextdds/src/rmw_api_impl_ndds.cpp:686
#16 0x00007fc6b168b1b0 in rcl_service_fini (service=0x41975900, node=0x40ac0230) at /workspaces/ros2/src/ros2/rcl/rcl/src/rcl/service.c:261
#17 0x00007fc6b148e978 in operator() (__closure=0x419025b0, service=0x41975900) at /workspaces/ros2/src/ros2/rclpy/rclpy/src/rclpy/service.cpp:72
#18 0x00007fc6b1491965 in std::_Sp_counted_deleter<rcl_service_s*, rclpy::Service::Service(rclpy::Node&, pybind11::object, const std::string&, pybind11::object)::<lambda(rcl_service_t*)>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose(void) (this=0x419025a0)
    at /usr/include/c++/13/bits/shared_ptr_base.h:527
#19 0x00007fc6b12d5aad in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x419025a0)
--Type <RET> for more, q to quit, c to continue without paging--c
    at /usr/include/c++/13/bits/shared_ptr_base.h:346
#20 0x00007fc6b12f0133 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7fff22916a18, __in_chrg=<optimized out>)
    at /usr/include/c++/13/bits/shared_ptr_base.h:1071
#21 0x00007fc6b1410dda in std::__shared_ptr<rcl_service_s, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7fff22916a10, __in_chrg=<optimized out>)
    at /usr/include/c++/13/bits/shared_ptr_base.h:1524
#22 0x00007fc6b1491b6c in std::__shared_ptr<rcl_service_s, (__gnu_cxx::_Lock_policy)2>::reset (this=0x41904680)
    at /usr/include/c++/13/bits/shared_ptr_base.h:1642
#23 0x00007fc6b148e8f2 in rclpy::Service::destroy (this=0x419045c0) at /workspaces/ros2/src/ros2/rclpy/rclpy/src/rclpy/service.cpp:45
#24 0x00007fc6b13a0176 in rclpy::Destroyable::destroy_when_not_in_use (this=0x419045c0)
    at /workspaces/ros2/src/ros2/rclpy/rclpy/src/rclpy/destroyable.cpp:71
#25 0x00007fc6b13a1675 in pybind11::cpp_function::cpp_function<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}::operator()(rclpy::Destroyable*) const (__closure=0x405226e8, c=0x419045c0) at /usr/include/pybind11/pybind11.h:111
#26 0x00007fc6b13a306b in pybind11::detail::argument_loader<rclpy::Destroyable*>::call_impl<void, pybind11::cpp_function::cpp_function<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}&, 0ul, pybind11::detail::void_type>(pybind11::cpp_function::cpp_function<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}&, std::integer_sequence<unsigned long, 0ul>, pybind11::detail::void_type&&) && (this=0x7fff22916b70, f=...) at /usr/include/pybind11/cast.h:1480
#27 0x00007fc6b13a2922 in pybind11::detail::argument_loader<rclpy::Destroyable*>::call<void, pybind11::detail::void_type, pybind11::cpp_function::cpp_function<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}&>(pybind11::cpp_function::cpp_function<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}&) && (this=0x7fff22916b70, f=...)
    at /usr/include/pybind11/cast.h:1454
#28 0x00007fc6b13a2372 in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}, void, rclpy::Destroyable*, pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(pybind11::cpp_function::initialize<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}&&, void (*)(rclpy::Destroyable*), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(pybind11::detail::function_call&)#1}::operator()(pybind11::detail::function_call&) const (__closure=0x0, call=...) at /usr/include/pybind11/pybind11.h:254
#29 0x00007fc6b13a23da in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}, void, rclpy::Destroyable*, pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(pybind11::cpp_function::initialize<void, rclpy::Destroyable, , pybind11::name, pybind11::is_method, pybind11::sibling, char [74]>(void (rclpy::Destroyable::*)(), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(rclpy::Destroyable*)#1}&&, void (*)(rclpy::Destroyable*), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [74])::{lambda(pybind11::detail::function_call&)#1}::_FUN(pybind11::detail::function_call&) () at /usr/include/pybind11/pybind11.h:224
#30 0x00007fc6b12e8228 in pybind11::cpp_function::dispatcher (self=0x7fc6b185bf00, args_in=0x7fc6b220c640, kwargs_in=0x0)
    at /usr/include/pybind11/pybind11.h:946
#31 0x0000000000581a6f in ?? ()
#32 0x00000000005492f5 in _PyObject_MakeTpCall ()
#33 0x00000000005d68bf in _PyEval_EvalFrameDefault ()
#34 0x000000000054cf04 in ?? ()
#35 0x0000000000549cf5 in PyObject_Vectorcall ()
#36 0x00000000005dcebf in _PyEval_EvalFrameDefault ()
#37 0x00000000005d4dab in PyEval_EvalCode ()
#38 0x0000000000607fc2 in ?? ()
#39 0x00000000006b4393 in ?? ()
#40 0x00000000006b40fa in _PyRun_SimpleFileObject ()
#41 0x00000000006b3f2f in _PyRun_AnyFileObject ()
#42 0x00000000006bbf45 in Py_RunMain ()
#43 0x00000000006bba2d in Py_BytesMain ()
#44 0x00007fc6b26041ca in __libc_start_call_main (main=main@entry=0x518bd0, argc=argc@entry=2, argv=argv@entry=0x7fff229177c8)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#45 0x00007fc6b260428b in __libc_start_main_impl (main=0x518bd0, argc=2, argv=0x7fff229177c8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fff229177b8) at ../csu/libc-start.c:360
#46 0x0000000000656a35 in _start ()

And that mutex seems to be zero'd - is this a use-after-free bug?

(gdb) f 4
#4  ___pthread_cond_clockwait64 (abstime=<optimized out>, clockid=<optimized out>, mutex=0x40a988f0, cond=0x40a98918) at ./nptl/pthread_cond_wait.c:691
691     in ./nptl/pthread_cond_wait.c
(gdb) p *mutex
$5 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
  __size = '\000' <repeats 12 times>, "\001", '\000' <repeats 26 times>, __align = 0}

[ros-discourse]

This issue has been mentioned on Open Robotics Discourse. There might be relevant details there:

https://discourse.openrobotics.org/t/ros-pmc-minutes-for-september-30-2025/50379/1