[Mellanox] Disable SSD NCQ on Mellanox platforms by volodymyrsamotiy · Pull Request #17567 · sonic-net/sonic-buildimage

volodymyrsamotiy · 2023-12-19T15:01:59Z

Why I did it

Based on some research some products might experience an occasional IO failures in the communication between CPU and SSD because of NCQ.
There seems to be a problem between some kernel versions and some SATA controllers.

Syslog error message examples:

Error "ata1: SError: { UnrecovData Handshk }" - "failed command: WRITE FPDMA QUEUED".
Error "ata1: SError: { RecovComm HostInt PHYRdyChg CommWake 10B8B DevExch }" - "failed command: READ FPDMA QUEUED".

Some vendors already disabled NCQ on their platforms in SONiC due to similar issue:

[Arista] Disable ATA NCQ for a few products #13739 [Arista] Disable ATA NCQ for a few products
[Arista] Disable SSD NCQ on DCS-7050CX3-32S #13964 [Arista] Disable SSD NCQ on DCS-7050CX3-32S

Also there are other discussions on Debian/Ubuntu forums about similar issues and it was suggested to disable NCQ:

https://askubuntu.com/questions/133946/are-these-sata-errors-dangerous

Work item tracking

Microsoft ADO (number only):

How I did it

Add a kernel parameter to tell libata to disable NCQ

How to verify it

Use FIO tool - fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4

Test results with NCQ enabled:

 READ: bw=128MiB/s (135MB/s), 128MiB/s-128MiB/s (135MB/s-135MB/s), io=247MiB (259MB), run=1924-1924msec
WRITE: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=253MiB (265MB), run=1924-1924msec
…
 READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=247MiB (259MB), run=1902-1902msec
WRITE: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=253MiB (265MB), run=1902-1902msec
…
 READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=247MiB (259MB), run=1919-1919msec
WRITE: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=253MiB (265MB), run=1919-1919msec

Test results with NCQ disabled:

 READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=247MiB (259MB), run=2354-2354msec
WRITE: bw=107MiB/s (113MB/s), 107MiB/s-107MiB/s (113MB/s-113MB/s), io=253MiB (265MB), run=2354-2354msec
…
 READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=247MiB (259MB), run=2349-2349msec
WRITE: bw=108MiB/s (113MB/s), 108MiB/s-108MiB/s (113MB/s-113MB/s), io=253MiB (265MB), run=2349-2349msec
…
 READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=247MiB (259MB), run=2349-2349msec
WRITE: bw=108MiB/s (113MB/s), 108MiB/s-108MiB/s (113MB/s-113MB/s), io=253MiB (265MB), run=2349-2349msec

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

mssonicbld · 2023-12-19T15:05:48Z

@StormLiangMS,@liushilongbuaa PR: #17567 is conflict with MS internal repo
Please complete the following PR by pushing fix commit to sonicbld/conflict_prefix/17567-fix
https://msazure.visualstudio.com/One/_git/Networking-acs-buildimage/pullrequest/9264755
Then comment "/azpw ms_conflict" to rerun PR checker.

bingwang-ms · 2023-12-19T16:52:01Z

@saiarcot895 Could you please help review? Thanks!

bingwang-ms · 2023-12-19T16:52:52Z

How about the SN2700 A1?

liushilongbuaa · 2023-12-20T02:40:35Z

/azpw ms_conflict

liat-grozovik

@volodymyrsamotiy you are missing SN5600 as well as SN2700-A1.
please go over the list of systems. BTW. this means you will have conflict with 2205 thus please update the missing systems and create a backport PR for 202205

StormLiangMS · 2024-01-03T01:10:05Z

@liat-grozovik are we good to go?

mssonicbld · 2024-01-03T17:21:08Z

@StormLiangMS,@liushilongbuaa PR: #17567 is conflict with MS internal repo
Please complete the following PR by pushing fix commit to sonicbld/conflict_prefix/17567-fix
https://msazure.visualstudio.com/One/_git/Networking-acs-buildimage/pullrequest/9303135
Then comment "/azpw ms_conflict" to rerun PR checker.

yxieca · 2024-01-04T01:40:47Z

@volodymyrsamotiy you are missing SN5600 as well as SN2700-A1. please go over the list of systems. BTW. this means you will have conflict with 2205 thus please update the missing systems and create a backport PR for 202205

@liat-grozovik can we move forward with this PR and handle 5600 and 2700-A1 with another PR?

yxieca · 2024-01-04T01:49:26Z

/azpw ms_conflict

mssonicbld · 2024-01-04T01:57:21Z

@StormLiangMS,@liushilongbuaa PR: #17567 is conflict with MS internal repo
Please complete the following PR by pushing fix commit to sonicbld/conflict_prefix/17567-fix
https://msazure.visualstudio.com/One/_git/Networking-acs-buildimage/pullrequest/9306301
Then comment "/azpw ms_conflict" to rerun PR checker.

liushilongbuaa · 2024-01-04T02:45:38Z

/azpw ms_conflict

volodymyrsamotiy · 2024-01-04T07:33:15Z

@volodymyrsamotiy you are missing SN5600 as well as SN2700-A1. please go over the list of systems. BTW. this means you will have conflict with 2205 thus please update the missing systems and create a backport PR for 202205

@liat-grozovik can we move forward with this PR and handle 5600 and 2700-A1 with another PR?

@yxieca, no need to handle 5600 and 2700-A1 with another PR, this PR was already updated, it has all the changes.

yxieca · 2024-01-04T17:05:11Z

@volodymyrsamotiy you are missing SN5600 as well as SN2700-A1. please go over the list of systems. BTW. this means you will have conflict with 2205 thus please update the missing systems and create a backport PR for 202205

@liat-grozovik can we move forward with this PR and handle 5600 and 2700-A1 with another PR?

@yxieca, no need to handle 5600 and 2700-A1 with another PR, this PR was already updated, it has all the changes.

@volodymyrsamotiy what is keeping this PR in draft mode? @liat-grozovik any other blocking issues?

bingwang-ms · 2024-01-09T21:19:53Z

@liat-grozovik Can we unblock this PR now?

bingwang-ms · 2024-01-09T21:20:38Z

Removed the label for 202205 branch as another PR has been opened to 202205 #17662

prgeor · 2024-01-10T14:37:35Z

@volodymyrsamotiy you are missing SN5600 as well as SN2700-A1. please go over the list of systems. BTW. this means you will have conflict with 2205 thus please update the missing systems and create a backport PR for 202205

@liat-grozovik i think this has been addressed so please check again

mssonicbld · 2024-01-29T16:34:38Z

@volodymyrsamotiy PR conflicts with 202311 branch

yxieca · 2024-01-31T01:38:23Z

@volodymyrsamotiy can you help create 202311 PR?

- Why I did it Based on some research some products might experience an occasional IO failures in the communication between CPU and SSD because of NCQ. There seems to be a problem between some kernel versions and some SATA controllers. Syslog error message examples: Error "ata1: SError: { UnrecovData Handshk }" - "failed command: WRITE FPDMA QUEUED". Error "ata1: SError: { RecovComm HostInt PHYRdyChg CommWake 10B8B DevExch }" - "failed command: READ FPDMA QUEUED". Some vendors already disabled NCQ on their platforms in SONiC due to similar issue: [Arista] Disable ATA NCQ for a few products sonic-net#13739 [Arista] Disable ATA NCQ for a few products [Arista] Disable SSD NCQ on DCS-7050CX3-32S sonic-net#13964 [Arista] Disable SSD NCQ on DCS-7050CX3-32S Also there are other discussions on Debian/Ubuntu forums about similar issues and it was suggested to disable NCQ: https://askubuntu.com/questions/133946/are-these-sata-errors-dangerous - How I did it Add a kernel parameter to tell libata to disable NCQ - How to verify it Use FIO tool - fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4

mssonicbld · 2024-01-31T14:26:17Z

Cherry-pick PR to 202305: #17960

StormLiangMS · 2024-01-31T14:26:23Z

ADO: 25853968

- Why I did it Based on some research some products might experience an occasional IO failures in the communication between CPU and SSD because of NCQ. There seems to be a problem between some kernel versions and some SATA controllers. Syslog error message examples: Error "ata1: SError: { UnrecovData Handshk }" - "failed command: WRITE FPDMA QUEUED". Error "ata1: SError: { RecovComm HostInt PHYRdyChg CommWake 10B8B DevExch }" - "failed command: READ FPDMA QUEUED". Some vendors already disabled NCQ on their platforms in SONiC due to similar issue: [Arista] Disable ATA NCQ for a few products #13739 [Arista] Disable ATA NCQ for a few products [Arista] Disable SSD NCQ on DCS-7050CX3-32S #13964 [Arista] Disable SSD NCQ on DCS-7050CX3-32S Also there are other discussions on Debian/Ubuntu forums about similar issues and it was suggested to disable NCQ: https://askubuntu.com/questions/133946/are-these-sata-errors-dangerous - How I did it Add a kernel parameter to tell libata to disable NCQ - How to verify it Use FIO tool - fio --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4

volodymyrsamotiy added Request for 202205 Branch Request for 202305 Branch labels Dec 19, 2023

moshemos added Issue for 202311 Request for 202311 Branch and removed Issue for 202311 labels Dec 19, 2023

prgeor added the NVIDIA label Dec 19, 2023

liat-grozovik requested review from prgeor and yxieca December 25, 2023 08:59

liat-grozovik requested changes Dec 25, 2023

View reviewed changes

[Mellanox] Disable SSD NCQ on Mellanox platforms

49e301e

volodymyrsamotiy force-pushed the noncq branch from 472b63f to 49e301e Compare January 3, 2024 17:15

volodymyrsamotiy mentioned this pull request Jan 3, 2024

[202205] [Mellanox] Disable SSD NCQ on Mellanox platforms #17662

Merged

12 tasks

volodymyrsamotiy marked this pull request as ready for review January 8, 2024 12:52

bingwang-ms removed the Request for 202205 Branch label Jan 9, 2024

prgeor approved these changes Jan 10, 2024

View reviewed changes

liat-grozovik approved these changes Jan 22, 2024

View reviewed changes

liat-grozovik merged commit f1d6655 into sonic-net:master Jan 28, 2024

yxieca added the Approved for 202311 Branch label Jan 29, 2024

mssonicbld added the Cherry Pick Conflict_202311 label Jan 29, 2024

StormLiangMS added the Approved for 202305 Branch label Jan 31, 2024

mssonicbld added the Created PR to 202305 Branch label Jan 31, 2024

mssonicbld mentioned this pull request Jan 31, 2024

[action] [PR:17567] [Mellanox] Disable SSD NCQ on Mellanox platforms #17960

Merged

12 tasks

mssonicbld added Included in 202305 Branch and removed Created PR to 202305 Branch labels Jan 31, 2024

volodymyrsamotiy mentioned this pull request Feb 5, 2024

[202311] [Mellanox] Disable SSD NCQ on Mellanox platforms #18040

Merged

12 tasks

yxieca added the Included in 202311 Branch label Feb 7, 2024

Conversation

volodymyrsamotiy commented Dec 19, 2023 • edited by liat-grozovik Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

mssonicbld commented Dec 19, 2023

Uh oh!

bingwang-ms commented Dec 19, 2023

Uh oh!

bingwang-ms commented Dec 19, 2023

Uh oh!

liushilongbuaa commented Dec 20, 2023

Uh oh!

liat-grozovik left a comment

Choose a reason for hiding this comment

Uh oh!

StormLiangMS commented Jan 3, 2024

Uh oh!

mssonicbld commented Jan 3, 2024

Uh oh!

yxieca commented Jan 4, 2024

Uh oh!

yxieca commented Jan 4, 2024

Uh oh!

mssonicbld commented Jan 4, 2024

Uh oh!

liushilongbuaa commented Jan 4, 2024

Uh oh!

volodymyrsamotiy commented Jan 4, 2024

Uh oh!

yxieca commented Jan 4, 2024

Uh oh!

bingwang-ms commented Jan 9, 2024

Uh oh!

bingwang-ms commented Jan 9, 2024

Uh oh!

prgeor commented Jan 10, 2024

Uh oh!

mssonicbld commented Jan 29, 2024

Uh oh!

yxieca commented Jan 31, 2024

Uh oh!

mssonicbld commented Jan 31, 2024

Uh oh!

StormLiangMS commented Jan 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

volodymyrsamotiy commented Dec 19, 2023 •

edited by liat-grozovik

Loading