[FEA] Unnecessary StreamSynchronization in reading hinders the pipeline establishment

[JigaoLuo]

Hi cuIO team,

I'm writing to report a performance issue I've discovered while working on building a pipeline for Parquet reading. I was struggling with this pipeline yesterday and also raised some questions in #18268 . Now, I've delved deeper into the problem and revisited my previous inquiries.

I have a setup with:

• NVIDIA A100-SXM4-40GB

The Problem with the Pipeline

One persistent issue I've noticed is the peculiar behavior of the pipeline. Sometimes, when reading Parquet files, the operations within each stream tend to cluster together, preventing the establishment of a stable I/O&compute pipeline.

Profiling and Experiments

To investigate this issue further, I conducted profiling. The main difference from my previous experiment in #18268 is that I used 3 threads/streams to better saturate the GPU. All other aspects of the experiment remained the same as in the previous issue.

I closely examined the interaction of the 3 threads in the profiling data. What I found particularly strange was the large, prominent part of the FileHandle::pread() call and the subsequent stream synchronization beneath it. This part of the code comes from KvikIO, and the function calls are highlighted in different colors for easier identification. For instance, in Thread 1, this section was quite noticeable:

I hypothesized that the long duration of this read operation was due to stream synchronization. So, I tried to identify other threads that were also performing synchronization at the same time. I discovered that Thread 3 was sorting data and synchronizing the default stream upon completion of the sorting operation:

The alignment of the stream synchronizations in Thread 1 and Thread 3 made me question why the FileHandle::pread() function needed to synchronize with the default stream.

Locating the Synchronization in the Code

Finally, I found that in both cuDF and KvikIO's pread implementation as FileHandle::pread, that sync_default_stream is default true to synchronize with the default stream. In my case, this meant that the I/O operation had to wait for the previous sorting operation to finish before it could start:

cudf/cpp/src/io/utilities/datasource.cpp

Line 113 in 79109d4

return _kvikio_handle.pread(dst, read_size, offset);

https://github.com/rapidsai/kvikio/blob/3adbe7e6435c728dc8a03aba97094519f2a299c5/cpp/include/kvikio/file_handle.hpp#L256

I plan to submit a simple PR to address this issue. However, I'm not certain if it will improve performance in all cases, as the behavior of the pipeline doesn't seem to be entirely deterministic. Thank you for your attention to this matter. I look forward to your feedback and suggestions.

[GregoryKimball]

Thank you @JigaoLuo for this investigation, and the previous results shared in #17873 and #18268 about pipelining in libcudf. We've also seen mixed results here, where usually pipelining works well in query workloads and some benchmarks, but not others.

I'll attach one example profile from our billion_rows example (code pointer) that shows very good pipelining.

02-csvpipe_256_4b.zip

However, we observe poor pipelining in the parquet_reader_multithread microbenchmark with FILEPATH io_type (code pointer) and we haven't yet found a way to improve this case.

We believe that the task queue in KvikIO should already prevent the the IO from clogging up over multiple files, but for some reason we still observe cases where multiple threads are competing for IO at the same time. Perhaps the issue is that task submission is waiting on a sync, and then all threads put interleaved tasks onto the queue.

@JigaoLuo it's possible you may have identified the root cause, and the profiling you shared in this comment looks great. I will ask @kingcrimsontianyu to weigh in on your proposed solution.

[kingcrimsontianyu]

May I know what version of CUDA runtime you were using?

[JigaoLuo]

May I know what version of CUDA runtime you were using?

I have the conda cudf_dev as cudf provides.

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0