Timestamping is now handled by respective base classes (src / sink) + new raw nodes

[AiVerisimilitude]

Timestamp propagation (configurable) + raw metadata src/sink (rosrawsrc, rosrawsink) + base-class timestamping refactor.

TL;DR

• New configurable timestamping on sinks:
  • timestamp-mode = ros-offset (default) | reference | pts
  • timestamp-conversion-mode = none (default) | ntp2unix
• New raw metadata bridge:
  • rosrawsrc and rosrawsink using gst_msgs/StreamStampedData
• Sources (rosimagesrc, rostextsrc) now stamp buffers via a shared helper that can also attach reference timestamp meta (attach-reference-timestamp).
• Sources allow overriding the caps of timestamps for the reference-timestamp metadata via time-caps

Motivation

1. End-to-end timestamp propagation. Depending on pipeline topology (RTSP/ONVIF, local sources, or synthetic inputs), the authoritative timestamp may live in different places: ROS headers, PTS, or GstReferenceTimestampMeta. Making this configurable lets pipelines pick the right truth source.
2. Raw metadata transport. We often need to move binary payloads (e.g., ONVIF XML, detection blobs) beside video. rosrawsrc/sink provide a clean, typed way to move these bytes between ROS and GStreamer, keeping the door open for sync with video.

---

What’s in this MR

1) Sink-side timestamping is configurable (base class)

New properties on all ROS sinks (via RosBaseSink):

• timestamp-mode (ros-offset | reference | pts)
  • ros-offset (default): previous behavior (ROS clock → PTS adjusted by base time and ROS clock offset).
  • reference: take the time from GstReferenceTimestampMeta (if present).
  • pts: use the buffer PTS verbatim.
• timestamp-conversion-mode (none | ntp2unix)
  • Optional post-selection conversion (e.g., convert NTP→UNIX).

Applies uniformly to rosimagesink, rostextsink, and the new rosrawsink.

2) Source-side stamping & optional reference meta attachment (base class)

New property on all ROS sources (via RosBaseSrc):

• attach-reference-timestamp (bool, default false)
  • When true, the source attaches GstReferenceTimestampMeta with caps timestamp/x-unix and the message time.
  • All stamping is routed through a new helper set_timestamps(...) so sources don’t duplicate logic.

rosimagesrc and rostextsrc updated to use set_timestamps(...). rostextsrc (strings have no header) now timestamps using node->now() and can optionally attach reference meta when requested.

3) New raw metadata bridge

Elements:

• rosrawsrc (caps: application/x-onvif-metadata, format=xml, type=metadata)
  • Subscribes to a ROS raw byte stream and produces GstBuffers.
• rosrawsink (caps: application/x-onvif-metadata)
  • Consumes GstBuffers and publishes a ROS raw byte stream.

Message type:

• Introduces gst_msgs/StreamStampedData

  std_msgs/Header header
  uint8[] data

• rosrawsrc/sink use gst_msgs::msg::StreamStampedData (with header timestamps).

---

How it fits together (common scenarios)

A) RTSP/ONVIF pipeline using reference timestamps

• Source side (optional): set attach-reference-timestamp=true on ros*src to attach timestamp/x-unix meta.
• Sink side: set timestamp-mode=reference.
• Conversion: if your reference meta is NTP-based (e.g., from RTSP elements), enable timestamp-conversion-mode=ntp2unix. If your reference meta is already UNIX (e.g., from ros*src with attach-reference-timestamp=true), keep timestamp-conversion-mode=none.

B) PTS-driven stamping (e.g., rtponvifparse put the authoritative clock in PTS)

• Sink side: timestamp-mode=pts, typically with timestamp-conversion-mode=ntp2unix.

C) Legacy/default behavior

• Do nothing; defaults keep previous behavior: timestamp-mode=ros-offset + timestamp-conversion-mode=none.

---

Example

RTSP server (video) with reference timestamps and metadata

 gst_rtsp_media_factory_set_launch(
      factory,
      "( "
      // Video transport
      "rosimagesrc ros-topic=/test/image attach-reference-timestamp=true time-caps=timestamp/x-unix "
      "! queue "
      "! videoconvert "
      "! videoscale "
      "! video/x-raw,format=I420 "
      "! x264enc tune=zerolatency key-int-max=1 intra-refresh=true "
      "! rtph264pay pt=96 "
      "! rtponviftimestamp use-reference-timestamps=true name=pay0 "
      // Metadata transport
      "rosrawsrc ros-topic=/test/detections_raw attach-reference-timestamp=true time-caps=timestamp/x-unix "
      "! queue "
      "! rtponvifmetadatapay "
      "! rtponviftimestamp use-reference-timestamps=true name=pay1 "
      ")"
    );

Client version

gst-launch-1.0 \
  rtspsrc location=rtsp://localhost:8554/test_video \
          onvif-mode=1 onvif-rate-control=0 latency=200 \
          buffer-mode=0 ntp-sync=false name=src \
  \
  src. ! application/x-rtp,media=application,encoding-name=VND.ONVIF.METADATA \
       ! queue \
       ! rtponvifparse \
       ! rtponvifmetadatadepay \
       ! queue \
       ! rosrawsink ros-topic=/received/meta_raw \
                     timestamp-mode=pts \
                     timestamp-conversion-mode=ntp2unix \
                     sync=false \
  \
  src. ! application/x-rtp,media=video \
       ! queue \
       ! rtponvifparse \
       ! rtph264depay \
       ! avdec_h264 \
       ! videoconvert \
       ! timeoverlay time-mode=0 show-times-as-dates=true \
                     datetime-format="%d.%m.%Y %H:%M:%S.%f" \
                     halignment=right valignment=bottom \
       ! rosimagesink ros-topic=/received/image \
                      timestamp-mode=pts \
                      timestamp-conversion-mode=ntp2unix \
                      sync=false

Compatibility & defaults

• Defaults preserve behavior: timestamp-mode=ros-offset, timestamp-conversion-mode=none, attach-reference-timestamp=false.
• GStreamer versions:
  • gst_buffer_add_reference_timestamp_meta works on 1.18+ (so sources can attach meta).
  • If you rely on rtspsrc to inject reference meta, the convenient property is 1.20+. Not required when sources attach meta themselves.
• Metadata transport with ONVIF timestamps requires gst-plugins-rs

[AiVerisimilitude]

I've updated the PR to include raw nodes used for metadata transport together with a new message type mostly because the string node's message does not have header / timestamp information.

[BrettRD]

This is great, I'll have to draw out some diagrams to get my head around the new timestamp options; the conversions between ntp and unix don't seem fully consistent between src and sink

string node's message does not have header

yes, those were intended to interact with text-to-speech and subtitle systems, and are probably due for revisiting. Your choice to introduce a stamped blob message is extremely welcome.
I like how your rawsink (configured for onvif meta) ties in with elements like onvifmetadataextractor to produce concurrent ROS messages in parallel with camera calibration messages

Is there any reason not to set the default rosrawsrc and rosrawsink caps to ANY?

Would it make sense to deduce the ntp/unix conversion mode from the caps of the GstReferenceTimestampMeta? ie "timestamp/x-ntp" vs "timestamp/x-unix"
The src could specify gst_caps_new_empty_simple("timestamp/x-rostime"); or such, and the docs already suggest that as a usable distinction. (I'm not fond of assuming rostime is unix-time because it interacts badly with gazebo)

I don't fully understand the utility of TIMESTAMP_MODE_PTS, it seems like this should be paired with a feature that drives the /clock topic from the pipeline time

[AiVerisimilitude]

Hi! Awesome, thanks for the review.

the conversions between ntp and unix don't seem fully consistent between src and sink

I agree. At the moment, the only reason why that option was added was because when using onvif-mode=1 the PTS of the receiver will be in NTP which requires a conversion to Unix when dumping it back to ROS. I will need to look up what happens if different clocks are used.

It is, however, implied that any timestamp coming from ROS is Unix so I could add a new parameter to override it.

Is there any reason not to set the default rosrawsrc and rosrawsink caps to ANY?

No, mostly convenience for the use-case presented, but there is no preference here. I will change it.

Would it make sense to deduce the ntp/unix conversion mode from the caps of the GstReferenceTimestampMeta?

I don't understand the use case here. Currently, the conversion is explicit, in the sense that if you know the used timestamps but want a different one, you can force a conversion of NTP -> Unix (no other currently added since I had no other use cases). But you don't have to convert, you can use the clocks as is, meaning if the reference timestamp is already in the type you want, you can just use "timestamp-mode=reference" and leave the conversion to none.

What I could see potentially being useful is setting the desired clock-end type and try to deduce the conversion if one is needed, but I do not know how feasible this is.

I don't fully understand the utility of TIMESTAMP_MODE_PTS

This was added because onvif-mode will set the PTS to the timestamps sent in the onvif extended headers :(. Don't know if there is a different way to tackle it.

[BrettRD]

Timestamps and clocking easy to get wrong in gstreamer, and they lead to subtle bugs. there are already too many timestamp config gotchas in this repo that can cause the pipeline to silently hang, and that's not fair on my users.

implied that any timestamp coming from ROS is Unix

It's generally unsafe to assume that rostime is unixtime because gazebo can override it using the /clock topic, and multi-agent systems don't automatically have useful NTP. If a ros timestamp is transported, the origin clock needs to be uniquely identified. I tend to do this by packing the robot hostname in the frame_id of the header

Would it make sense to deduce the ntp/unix conversion mode from the caps of the GstReferenceTimestampMeta?

I don't understand the use case here. Currently, the conversion is explicit, in the sense that if you know the used timestamps but want a different one, you can force a conversion of NTP -> Unix (no other currently added since I had no other use cases). But you don't have to convert, you can use the clocks as is, meaning if the reference timestamp is already in the type you want, you can just use "timestamp-mode=reference" and leave the conversion to none.

What I could see potentially being useful is setting the desired clock-end type and try to deduce the conversion if one is needed, but I do not know how feasible this is.

GstReferenceTimestampMeta struct suggests the use of the caps string to specify the nature of the clock
From the docs:

• timestamp/x-drivername-stream : for timestamps that are locally generated by some driver named drivername when generating the stream, e.g. based on a frame counter
• timestamp/x-ntp, host=pool.ntp.org, port=123: for timestamps based on a specific NTP server. Note that the host/port parameters might not always be given.
• timestamp/x-ptp, version=IEEE1588-2008, domain=1: for timestamps based on a given PTP clock.
• timestamp/x-unix: for timestamps based on the UNIX epoch according to the local clock.

This convention allows us to specify the epoch and protocol of the clock, including allowing us to specify that the timestamp is a raw rostime from a specific host by saying something like timestamp/x-rostime, frame_id="turtle_1" Then the sink can intelligently decode the timestamp either converting from ntp/unix time, or pasting a local rostime, or applying a clock offset as required.

I don't fully understand the utility of TIMESTAMP_MODE_PTS

onvif-mode will set the PTS to the timestamps sent in the onvif extended headers :(.
Don't know if there is a different way to tackle it.

Setting PTS to onvif timestamp requires that the pipeline time is driven by whatever is decoding the onvif stream, otherwise the pipeline will hang when a user adds a live sink.
A live sink will wait for pipeline clock to catch up to the presentation time stamp which might never happen.
You can use onvif time or rostime as pipeline time by providing a gstreamer clock object. There's an early feature tracking issue for wrapping the \clock topic into a gst clock object

[AiVerisimilitude]

I've added a new property to rosbasesrc.cpp for manually setting the clock caps and set the default to timestamp/x-rostime. This default doesn't play well with some other plugins (as they don't recognize the type and don't know how to work with it) so I will update the example use-case to include the override to timestamp/x-unix.

I've also set the default caps of rosrawsrc to ANY. No real impact here.

Timestamps and clocking easy to get wrong in gstreamer, and they lead to subtle bugs. there are already too many timestamp config gotchas in this repo that can cause the pipeline to silently hang, and that's not fair on my users.

Yep, totally agree, and I ran into this issue a fair bit of times while working with ONVIF since it sets the PTS on the receiver. Don't really have ideas here on how to improve the situation. My default is always to build from source and inspect / debug, but I understand that is not something others are comfortable doing. If you have any ideas on what helped overcome these issues, let me know and I can implement them.

It's generally unsafe to assume that rostime is unixtime because gazebo can override it using the /clock topic, and multi-agent systems don't automatically have useful NTP. If a ros timestamp is transported, the origin clock needs to be uniquely identified. I tend to do this by packing the robot hostname in the frame_id of the header

Hopefully this is addressed by the new property

This convention allows us to specify the epoch and protocol of the clock, including allowing us to specify that the timestamp is a raw rostime from a specific host by saying something like timestamp/x-rostime, frame_id="turtle_1" Then the sink can intelligently decode the timestamp either converting from ntp/unix time, or pasting a local rostime, or applying a clock offset as required.

Currently, I do not have a use case for a GstReferenceTimestampMeta where I read the same caps that I set. My use-case is destructive in this sense. For the "server" GstReferenceTimestampMeta is used as a buffer for rtponviftimestamp use-reference-timestamps and it requires a timestamp caps understandable by it. While on the "client", if I am using ONVIF, its not used since rtponvifparse will put the timestamps in the ONVIF extended headers into PTS. But if instead I don't use ONVIF and want to use re-constructed timestamps from the RTCP sender reports by using rtspsrc add-reference-timestamp-meta I can access the timestamp in the GstReferenceTimestampMeta but it looses the original caps as this is a re-constructed timestamp - not the one set in the server.

A live sink will wait for pipeline clock to catch up to the presentation time stamp which might never happen.

Indeed, this is an issue and that is why I need to use rtspsrc onvif-rate-control=0. Unfortunately, I am piggybacking off ONVIF while it was created for a different use-case which is playback - not live feed, but it's the best tool so far (in the GST environment) for sending exact timetstamps with both images and raw data, together with sending raw data :(.

[AiVerisimilitude]

@BrettRD Does this address your concerns or is there anything else you'd like changed?