ROCKernels: using queue pool causes performance regression

[pxl-th]

Observed with RX 6700 XT.

MWE:

using KernelAbstractions
using ROCKernels
using AMDGPU

# function ROCKernels.next_queue() <- uncomment to disable queue pooling
#     AMDGPU.default_queue()
# end

@kernel function memset!(x, v)
    i = @index(Global)
    x[i] = v
end

function main()
    dev = ROCDevice()
    x = ROCArray{Float32}(undef, 1024 * 1024)
    v = 1f0

    @time wait(memset!(dev)(x, v; ndrange=length(x)))
    @time for _ in 1:10_000
        wait(memset!(dev)(x, v; ndrange=length(x)))
    end

    return nothing
end
main()

• With queue pooling:

  9.458379 seconds (14.68 M allocations: 967.041 MiB, 3.37% gc time, 3.64% compilation time)
 13.647801 seconds (2.31 M allocations: 81.017 MiB, 0.36% gc time)

• Without, using only one default queue:

  9.409779 seconds (14.68 M allocations: 967.038 MiB, 3.32% gc time, 3.48% compilation time)
  3.316558 seconds (2.30 M allocations: 80.710 MiB, 1.47% gc time)

[vchuravy]

Would be interesting to see where we are spending the time. What does a profile say?

Generally #317 will fix this by removing the event system and moving towards a task based concurrency model.

[jpsamaroo]

I don't see a significant difference on my APU.

[pxl-th]

I think that you don't even need to use those queues, just having them aroung is enough to cause the slowdown.
You can see this, by allocating queues at the beginnig queues = [ROCQueue(dev) for i in 1:16] and disabling pooling.

Most of the time is spent on wait.

[pxl-th]

From "Oversubscribed Command Queues in GPUs" AMD paper:

With HSA user mode queuing, an application can create multiple user mode queues (HSA queues) to launch kernels.
However, it is challenging for the GPU to locate all outstanding works on these queues when it can only monitor a fixed number of queues at a time. Therefore, the GPU periodically swaps different queues into the available hardware queue slots (hereafter referred as hardware queues) and processes ready tasks on the monitored queues. To ensure all queues are eventually serviced, the hardware will swap in and out queues at regular intervals, called the scheduling quantum.
When the number of allocated HSA queues is greater than the number of hardware queues, the GPU wastes significant time rotating between all allocated queues in search of ready tasks. Furthermore, barrier packets that handle task dependencies can block queues while waiting for prior tasks to complete.

So from manually increasing a number of queues in a pool, I've noticed that Navi 2 can handle 6 queues.
At 7 queues performance starts to degrade.
And it is not enough to have idle queues, you have to cycle between them to "confuse" the scheduler:

The hardware scheduler maps and unmaps queues from the queue list to the hardware list stored inside the packet processor at each scheduling quantum.
The number of entries in the hardware list is limited and each entry in the hardware list corresponds to the HSA queue mapped to a hardware queue.
The packet processor monitors these hardware queues, processes both kernel dispatch and barrier HSA packets, resolves dependencies expressed by barrier packets and forwards ready tasks to the dispatcher.

[jpsamaroo]

We should try to figure this number out for all the GPUs we have access to and add a query in AMDGPU.jl that returns the max number of hardware queues; we can then do our best to stay under this limit, and otherwise print a warning from AMDGPU.jl when more are allocated.

[vchuravy]

Hopefully fixed by JuliaGPU/AMDGPU.jl#398 since it drastically reduces the number of queues used.

[luraess]

From our experience testing priority queue pooling on MI250x, we concluded that using 4 queues (2 for high and 2 for low priority) was giving the best performance. Digging more, @utkinis found that deep in HSA source, there is a deprecated enum field setting HSA_AGENT_INFO_QUEUES_MAX = 12 that could be queried from AMDGPU.get_info but is not really useful as such... and we did still not find a way to query the number of HW queues...