Async DirectIO model loading on Linux

[JTischbein]

Implements Direct I/O (uncached) file reading on Linux to improve model loading performance by bypassing the page cache. This is especially beneficial for large model files.

While mmap is fast on loading the same model multiple times, uncached read provides consistent model loading times at the speed of the sequential disk read speed. On DGX Spark loading GPT-OSS-120B-MXFP4 using mmap takes ~110s, in the following loads ~67s. With these changes it takes consistently ~10.5s. The speedup depends on the model size, the disk read speed and for sequential loading the available RAM.

I would propose to set uncached reads as default, Windows already has async uncached IO (PR)

[ggerganov]

This results in a huge load speedup on DGX Spark and also at the end of the program leaves the memory in state free instead of buff/cache.

Currently, the implementation is gated behind defined(__linux__). Is this functionality generally supported across all linux platforms? If I am reading this correctly, it boils down to having O_DIRECT support for open().

Also, do we expect this change to also have effect on non-DGX Spark systems?

[lemmi]

On my strix halo machine with btrfs, this is strictly worse than master or with mmap. mmap shows the highest throughput while loading the model (~6GByte/s), master is around 3GByte/s and this patch is 2GByte/s.

[ehoogeveen-medweb]

IIRC with Strix Halo and ROCm/HIP, loading a model into memory reserved for the GPU using mmap has a major performance issue, hanging basically indefinitely for larger models. Given that reserving memory for the GPU also means having less RAM available to the CPU, it would be great if this DirectIO doesn't have that issue as it would make ROCm/HIP more viable for larger models. Vulkan doesn't have this issue.

[JTischbein]

@ggerganov I have added a fallback open() in case O_DIRECT is not available. O_DIRECT is supported on Linux since 2.4.10.

In my tests the first (cold) load time improved with every system configuration (PCIe4.0/PCIe5.0 SSD, RTX5080/5090). On the second load mmap=true was faster again, but only in case the model fitted into the VRAM. Overall, with a fast disk, the load time is also near the cached load with mmap=true.

@lemmi Which disk are you using? And I assume the 6GB/s load with mmap=true was a loading from cache, not the first cold load? The difference between std::fread and read is odd, I will have a look into it.

[lemmi]

So, I ran a bunch of tests with vulkan backend to test #18012 and #18047 against master.

1. GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 /usr/bin/time -v build/bin/llama-cli -m ../models/gpt-oss-120b-GGUF/gpt-oss-120b-mxfp4.gguf -p "bla" -n 0 --single-turn --mmap
2. GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 /usr/bin/time -v build/bin/llama-cli -m ../models/gpt-oss-120b-GGUF/gpt-oss-120b-mxfp4.gguf -p "bla" -n 0 --single-turn --no-mmap
3. /usr/bin/time -v build/bin/llama-cli -m ../models/gpt-oss-120b-GGUF/gpt-oss-120b-mxfp4.gguf -p "bla" -n 0 --single-turn --no-mmap

Minisforum MS-S1 Max (AMD RYZEN AI MAX+ 395 w/ Radeon 8060S)
2x WD_BLACK SN850X HS 2000GB, RAID 0, btrfs
Kernel 6.18.0_1

Configuration	master (d6a1e18)	#18012	#18047	#18012 + #18047
GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 + --mmap	0:43.37	0:30.31	0:31.02	0:37.36
GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 + --no-mmap	0:17.53	0:36.17	Error (out‑of‑memory)	0:17.98
--no-mmap	0:17.57	0:36.42	Error (out‑of‑memory)	0:18.08

ddrescue	avg GB/s	time
buffered	6.3	0:09.57
direct I/O	1.9	31.72

The --mmap case is weird. It starts out with >6GB/s, then short pause, then depending on the PR it looks like the whole model is read again at 2-4GB/s.
With --no-mmap, the throughput is also always below 4GB/s and using direct I/O is the worst case. Direct I/O is a little tricky on a CoW FS, so maybe it's not a very optimized path.
(Ideally vulkan could just use mmaped files, but i have no idea, whether that's possible)

(EDIT: of course I was a good boy and ran echo 3 > /proc/sys/vm/drop_caches between tests)

[JTischbein]

Thank you for testing this @lemmi ! Looking at your numbers it seems like read() with 18012 + 18047 is falling back to buffered IO, leading to similar performance as on master with fread(). I have implementing a filesystem check to decide whether to use read() with O_DIRECT or to use fopen()/fread(). Does it still make sense when read() and fread() perform equally?

Is --mmap on a warm start quicker than --no-mmap on your machine @lemmi ?

[ggerganov]

Changing this to false by default, results in a huge slowdown on MacOS with default arguments:

time ./bin/llama-completion -m ../models/gpt-oss-120b/ggml-model-mxfp4.gguf -p "hello" -n 1 -no-cnv

# master
real	0m4.648s

# PR
real	0m17.957s

Not sure what is the best way to handle this. If we keep it true, then linux users would not get the benefit of Direct IO. If we switch to false, Mac users will take the hit.

[JTischbein]

Would it be OK to set mmap depending on the platform?

[ggerganov]

We don't have such precedent atm for any of the parameters in common, so I would say it's not ideal.

[JTischbein]

I have on M4 Pro with GPT-OSS-20B on cold load --no-mmap: 4.168s --mmap: 6.3s. The warm load however takes with --mmap 2.1s (--no-mmap still ~4.1s).

Measured using time ./llama-cli -m /Users/jtischbein/Documents/models/openai_gpt-oss-20b-MXFP4.gguf --no-mmap -p "bla" -n 0 --single-turn and filesystem cache cleared using purge.

So the cold load time is still faster using --mmap, but unfortunately not as fast as on Linux.

[ggerganov]

We can do the following:

• Add new CLI argument --direct-io, -dio
• Description: "Use DirectIO if available. Takes precedence over --mmap"
• Keep use_mmap == true and use_direct_io == true
• On Mac, the internal implementation will determine that DIO is not available so it will fallback to mmap

Might want to do it in a separate PR as it would require changes in libllama API. This PR should keep use_mmap == true by default.

[JTischbein]

Sounds good

[JTischbein]

The commit removes the branching in the llama-model-loader.cpp and reduces the code duplications in llama-mmap.cpp. Now DirectIO is easier to integrate on Windows and Mac.

[ggerganov]

Let's restore use_mmap to true and we can merge.

[JTischbein]

I will file a PR which implements the argument use_direct_io later

[rankaiyx]

For NVMe drive, you can partition one disk into multiple sections, create a RAID0 array with mdadm, and use striping to automatically turn single-threaded I/O requests into parallel operations.

The kernel and NVMe's multi-queue feature handle this parallel processing without requiring any application changes.

This approach lets you reach the NVMe’s full bandwidth and benefits all applications.

[NeoZhangJianyu]

@JTischbein
This PR lead to issue: #18296
When add parameter "--no-mmap", there is an error to read file.

Could you fix it or revert as soon?

Thank you!

This code report the error:

} else {
            bool successful = false;
            while (!successful) {
                off_t ret = read(fd, ptr, len);

                if (ret == -1) {
                    if (errno == EINTR) {
                        continue;  // Interrupted by signal, retry
                    }
                    throw std::runtime_error(format("read error: %s", strerror(errno)));
                }
                if (ret == 0) {
                    throw std::runtime_error("unexpectedly reached end of file");
                }

                successful = true;
            }
        }

[JTischbein]

@NeoZhangJianyu There is a fix for Vulkan now in #18467 by using a buffer from the host. Could you try the same in the SYCL backend? Thank you

[NeoZhangJianyu]

@NeoZhangJianyu There is a fix for Vulkan now in #18467 by using a buffer from the host. Could you try the same in the SYCL backend? Thank you
@JTischbein @jeffbolznv

#18467 only support --mmap and --no-mmap, not support -dio and -ndio.
I test it with SYCL backend.

--mmap is quicker than --no-mmap.
Both work wells.

Thank you!

[JTischbein]

@NeoZhangJianyu -dio is enabled implicitly in the master branch when applying --no-mmap.

The performance of --mmap and --no-mmap heavily depends on the platform. When launching Llama.cpp a model the first time since booting the machine --no-mmap with direct io is usually faster. On the second launch --mmap is often faster, but only in case the model is still in the filesystem cache (means CPU RAM is large enough and the filesystem cache did not get dropped due to other processes requesting too much memory).

Great to hear the PR also fixes the loading with the SYCL backend, thanks for testing!

[NeoZhangJianyu]

The PR #18467 or it's base just fix the crash issue.
It can restore the performance of load model (--no-mmap) of earlier version of #18012.

[NeoZhangJianyu]

@NeoZhangJianyu There is a fix for Vulkan now in #18467 by using a buffer from the host. Could you try the same in the SYCL backend? Thank you

@JTischbein
The similar solution of Vulkan now in #18467 can't support SYCL:

SYCL could use buffer with host_ptr like Vulkan. But SYCL backend use USM (Unified Shared Memory ) to manage the memory. It's better than buffer.
But USM can't support the host ptr like Vulkan.

Thank you!