Bag recorder sometimes loses/skips data on bag split

[RMichaelSwan]

Description

The bag recorder seems to skip recording some data when large bag file splits (~4-5 GB) are saved. This behavior does not seem to occur for smaller split sizes. It is important to note that this occurred when using an external hard drive for storage.

Expected Behavior

Bag recorder would not lose data it is able to capture.

Actual Behavior

Data is missing for 1 minute every ~10 minutes (time it takes for us to record 4.7GB on our robot). In previous recording tests with smaller splits (~50MB) this does not occur. The bag recorder node does not give any error in the logs.

To Reproduce

1. Record bag with large split (e.g. 5GB)
2. After 1 or more splits occur, end recording
3. Review data timestamps, look for gaps

System (please complete the following information)

• OS: Ubuntu Noble
• ROS 2 Distro: Jazzy
• Install Method: APT
• Version: 0.26.6 (latest on Jazzy apt as of 2025-04-02)

Additional context

We were directing the recorder to save to an external hard drive like this one which has a USB 3.1 Gen 2 link rate. My guess is that the file writer took long enough that it held up the recorder from continuing to capture data. I suppose that this kind of issue could be partially alleviated by the proposal from #2007 though a more robust solution would be preferred if possible. Additional error messaging if this is not logically resolveable without changing settings/environment would be helpful .

[MichaelOrlov]

Hi @RMichaelSwan are you using MCAP storage plugin for recording?
If a messages loss happened on the Rosbag2 recorder side, you should see the warning message lke

Cache buffers lost messages per topic: 
Total lost: 

At the end of the recording or after the each bag split.
Form issue description, it looks like the disk too slow when dumping messages from the Rosbag2 inner cache.
Very litle we can do in this case. We already have double buffering scheme in the recorder. That allows us to decouple transport layer from the storage. I can recommend playing around with the inner Rosbag2 recorder cache size.
This is a --max-cache-size CLI option which is in bytes. Also I would recommend increasing DDS QoS queue depth via QoS override CLI option. Please refer to the https://github.com/ros2/rosbag2/blob/rolling/README.md#overriding-qos-profiles for details.

[RMichaelSwan]

This is with the MCAP storage plugin, yes. I do not see any warnings at all actually.

Do you have any suggestions on improving visibility? We did a very long run (like 24 hours) and only noticed the issue once we started analyzing the data, seeing minute long gaps periodically.

[MichaelOrlov]

@RMichaelSwan, it looks like you are losing messages on the transport layer when we are doing a bag split.
In the current situation, the band-aid would be increasing DDS QoS queue depth via QoS override CLI option as I have mentioned above.

As regards.

Do you have any suggestions on improving visibility?

On rolling, I have added logging when messages are lost on the transport layer in the scope of the #2039. The PR is not backportable by itself.
However, it is possible to add runtime output to the debug log when messages are lost on the transport layer.
Please refer to the a2585eb#diff-2ab554d1da9e067d9df2acd83b59c815430dbe754ac2b9e91e12dab74e2ead99R622-R627 part for details.

[RMichaelSwan]

I find it strange that the transport layer would be missing messages when all the messages are on the loopback (lo) adapter. I guess what is happening is the io-bound file writer is taking so long that the buffers fill up on both ends and we end up losing messages. Does the rosbag recorder write to file as it goes or is it in memory until a split is called? If the former, I wonder if this is happening because local storage is being used and then a move operation is being called on the completed file?

Seems like solutions are to either:

1. Increase QoS queue depth and max cache size
2. Reduce bag split size until no data is lost
3. Only write bags to faster local disk and then run a script/node to periodically move bag pieces over to external disk.

Will the recorder have any issues if not all the MCAP parts exist in the recorder target directory when the recording is ended?

Thanks so much for all your help.

[MichaelOrlov]

Well, to be honest, our bag split is a bit suboptimal.
When we need to split bag file we dumping the cache to the file and closing the current file then openning a new one. And this is happening in the write_message(serialized_message) operation.
As you can see, during this operation, the write_message(serialized_message) can take longer than usual, and the subscriber's DDS queue can potentially overflow. As a result, messages will drop on the transport layer.
That is why I advised you to increase the DDS queue depth on the Rosbag2 recorder side.