XDMA driver timeout in poll mode , can't work properly

[LNC0831]

Recently I'm working to get low latency in xdma tranfer, when I load the driver in interrupt mode, I always get 30us latency per transfer(32 byte), that means a bit long for my project. So I try to load the driver in poll mode, In XDMA/linux-kernel/test, I run that:
sudo ./load_driver.sh 4
and terminal returns:

interrupt_selection 4.
xdma                   86016  0
Loading driver...insmod xdma.ko poll_mode=1 ...

The Kernel module installed correctly and the xmda devices were recognized.
DONE

after this, In linux-kernel/tools, I run that:
sudo ./dma_to_device -a 0x44a00000 -v
and it takes about 10 seconds then terminal returns:

dev /dev/xdma0_h2c_0, addr 0x44a00000, aperture 0x0, size 0x20, offset 0x0, count 1
host buffer 0x1020 = 0x5620bae8f000
/dev/xdma0_h2c_0, write 0x20 @ 0x44a00000 failed -1.
write file: Unknown error 512

That issue was struggling and I have read other issues but it seems that no evidence had been given.
I will be appreciate if someone can help or share hints on the similar question.

My working environment:

1. Ubuntu 18.04.6
2. Kernel Version: 5.3.0-28-generic
3. dma_ip_driver Version: 2022.1.5

[dmitrym1]

Hi @LNC0831. Have you tried alanbl's patch set?

[LNC0831]

Hi @LNC0831. Have you tried alanbl's patch set?

Glad to receive your reply, I'm still try to figure out the root problem.I have read some artical on ZipCPU and there is a possibility that It's the AXI-Lite peripherals OR the AXI Smartconnect IP caused this problem. I've googled the alanbl's patch, and I did't find something about XDMA driver. Sorry for my unfamiliar with that, Could you please provide a link or upload a zip for me? After all,thank you for your reply.

[dmitrym1]

See the link #240
For AXI bus diagnostics I recommend using System ILA with Protocol Checker enabled, or standalone Protocol Checker IP, set a trigger on error detecton signal and you'll get a snapshot of bus state at the moment when the issue was detected. It will also give you an error code. But first try the aforementioned patch, there is a chance it will fix your problem. If it does not, see what "dmesg" tells you.

[LNC0831]

See the link #240 For AXI bus diagnostics I recommend using System ILA with Protocol Checker enabled, or standalone Protocol Checker IP, set a trigger on error detecton signal and you'll get a snapshot of bus state at the moment when the issue was detected. It will also give you an error code. But first try the aforementioned patch, there is a chance it will fix your problem. If it does not, see what "dmesg" tells you.

Thank you for your sincere help, I'll try that patch and will share the results or maybe solutions later 👍

[vvsvvsvvs]

Hi @LNC0831. Have you tried alanbl's patch set?

The problem is not in the thread lock. Seems it is in the polling thread.
So the patch set cannot help. x86 is OK, but ARM is not.
engine_service_wb_monitor() shows desc_wb == 0, expected_wb == 1. Thus providing the 10sec timeout.

For example, ARM CPU (RK3588) has IOMMU. And must translate addr. correctly (coherent dma allocation). But memory block at
engine->poll_mode_addr_virt is always 0 (all 8 entires).

[pigstimemachine]

Hi @LNC0831. Have you tried alanbl's patch set?

The problem is not in the thread lock. Seems it is in the polling thread. So the patch set cannot help. x86 is OK, but ARM is not. engine_service_wb_monitor() shows desc_wb == 0, expected_wb == 1. Thus providing the 10sec timeout.

For example, ARM CPU (RK3588) has IOMMU. And must translate addr. correctly (coherent dma allocation). But memory block at engine->poll_mode_addr_virt is always 0 (all 8 entires).

That means Poll_mode is unfeasible plan in the ARM linux system?

[dmitrym1]

That means Poll_mode is unfeasible plan in the ARM linux system?
It definitely works on aarch64 (iMX8M) and I don't see why it would not on 32-bit ARM. RK3588 is aarch64 too btw, so the problem must be somewhere else but XDMA.

Poll mode uses all the same stuff as interrupt mode. It creates a kernel thread that reads status register periodically during transfer. Interrupt mode also reads status register, except that it does this on interrupt rather than in a loop, and it does not create a separate thread. Original Xilinx code had some bugs that affected one mode but not another, so that's where differences may come from, but with latest community patches these bugs were fixed.

I don't see how IOMMU can do anything here, at least directly. XDMA is not that kind of DMA that requires all sorts of fancy stuff. In XDMA, CPU is the bus master and CPU is the component that initiates the transfer. So you start a transaction from userspace, it goes into kernel where XDMA forms a TLP and sends it through PCIe. After that, the CPU can do nothing but wait for FPGA to report that transaction is complete. So either user logic in the FPGA may not let it to complete transaction, or PCIe itself may not be able to do a correct transfer. For example, the latter may be because PCIe driver negotiated a mode that it does not actually support, or the physical link may be too bad so all the data is corrupted.

But again, this can not be the cause of your problem if interrupt mode works for you and poll mode does not. If interrupt mode works stable for you, then all low level stuff works just fine

[pigstimemachine]

That means Poll_mode is unfeasible plan in the ARM linux system?
It definitely works on aarch64 (iMX8M) and I don't see why it would not on 32-bit ARM. RK3588 is aarch64 too btw, so the problem must be somewhere else but XDMA.

Poll mode uses all the same stuff as interrupt mode. It creates a kernel thread that reads status register periodically during transfer. Interrupt mode also reads status register, except that it does this on interrupt rather than in a loop, and it does not create a separate thread. Original Xilinx code had some bugs that affected one mode but not another, so that's where differences may come from, but with latest community patches these bugs were fixed.

I don't see how IOMMU can do anything here, at least directly. XDMA is not that kind of DMA that requires all sorts of fancy stuff. In XDMA, CPU is the bus master and CPU is the component that initiates the transfer. So you start a transaction from userspace, it goes into kernel where XDMA forms a TLP and sends it through PCIe. After that, the CPU can do nothing but wait for FPGA to report that transaction is complete. So either user logic in the FPGA may not let it to complete transaction, or PCIe itself may not be able to do a correct transfer. For example, the latter may be because PCIe driver negotiated a mode that it does not actually support, or the physical link may be too bad so all the data is corrupted.

But again, this can not be the cause of your problem if interrupt mode works for you and poll mode does not. If interrupt mode works stable for you, then all low level stuff works just fine

But it truly can not run by poll_mode ,I use the TI-5718 chip to test interrupt and poll_mode ,interrupt is not stable ,but it works , poll_mode always overtime about 10 seconds ,and cannot rcv anything .

[dmitrym1]

But it truly can not run by poll_mode ,I use the TI-5718 chip to test interrupt and poll_mode ,interrupt is not stable ,but it works , poll_mode always overtime about 10 seconds ,and cannot rcv anything .

I understand that poll_mode may not work for your chip. I can help you debugging that. Apply latest community patch to the XDMA and rebuild it with debug output enabled. Write a simple app that does only one certain transfer. See what dmesg tells you about XDMA transfer in poll mode and in interrupt mode. There will be significant difference in the output since one mode works and the other one does not. You'll need to figure out where the problem is, in transaction requester or in transaction completer. You may need to use printk to see variables states in various functions. See if you spot any obvious difference between two modes

[pigstimemachine]

But it truly can not run by poll_mode ,I use the TI-5718 chip to test interrupt and poll_mode ,interrupt is not stable ,but it works , poll_mode always overtime about 10 seconds ,and cannot rcv anything .

I understand that poll_mode may not work for your chip. I can help you debugging that. Apply latest community patch to the XDMA and rebuild it with debug output enabled. Write a simple app that does only one certain transfer. See what dmesg tells you about XDMA transfer in poll mode and in interrupt mode. There will be significant difference in the output since one mode works and the other one does not. You'll need to figure out where the problem is, in transaction requester or in transaction completer. You may need to use printk to see variables states in various functions. See if you spot any obvious difference between two modes

The Debug mode will make the 100% cpu，and cannot run normally, that is another bug too.

[Prandr]

@dmitrym1
I am sorry to say that you seem to have a completely wrong idea how XDMA works.
The CPU sets the transfer up and configures the IP Core. It the IP Core on FPGA who then takes over the management of the transfer, by sending DMA requests as TLP packets over PCIE. The whole point of XDMA is to offload DMA operations to hardware on FPGA

[Prandr]

@LNC0831 @zhaochengjie0405
I have reworked the XDMA driver and it should improve the latency, particularly in poll mode, because everything there is done in the same thread. Thus there are no context switches or core migrations. You can find it on my repository:
https://github.com/Prandr/dma_ip_drivers

[pigstimemachine]

@LNC0831 @zhaochengjie0405 I have reworked the XDMA driver and it should improve the latency, particularly in poll mode, because everything there is done in the same thread. Thus there are no context switches or core migrations. You can find it on my repository: https://github.com/Prandr/dma_ip_drivers

I always cannot run the poll mode , Do I need to change the FPGA config for the poll mode?

[Prandr]

@LNC0831 @zhaochengjie0405 I have reworked the XDMA driver and it should improve the latency, particularly in poll mode, because everything there is done in the same thread. Thus there are no context switches or core migrations. You can find it on my repository: https://github.com/Prandr/dma_ip_drivers

I always cannot run the poll mode , Do I need to change the FPGA config for the poll mode?

@zhaochengjie0405
Are you talking about my reworked version?
Looks like you didn't read the docs. Please read the Readme first .
In short you, need to sudo make POLLING=1 install and reinsert the driver to enable polling mode.
Post further discussion or any problems on my repo, please.

[t460p]

@LNC0831 How is this issue going,? I use poll mode , the Completed Descriptor Count in WB Address is 0. But interrupt is table