Skip to content

[ASAN]Increasing switch create timeout for ASAN images#2777

Merged
qiluo-msft merged 5 commits intosonic-net:masterfrom
dgsudharsan:asan_timeout
May 30, 2023
Merged

[ASAN]Increasing switch create timeout for ASAN images#2777
qiluo-msft merged 5 commits intosonic-net:masterfrom
dgsudharsan:asan_timeout

Conversation

@dgsudharsan
Copy link
Copy Markdown
Collaborator

@dgsudharsan dgsudharsan commented May 15, 2023 •
edited
Loading

What I did
When running ASAN images with low CPU systems, due to the additional overhead associated with ASAN, it results in switch create timeout as seen in logs below. Hence increasing the switch create timeout to 2x for ASAN builds. e.g. For regular switch scenario, the timeout will be 120 seconds instead of 60 seconds.

Apr 18 20:23:46.749074 arc-switch1004 NOTICE swss#orchagent: :- create: request switch create with context 0
Apr 18 20:23:46.749074 arc-switch1004 NOTICE swss#orchagent: :- allocateNewSwitchObjectId: created SWITCH VID oid:0x21000000000000 for hwinfo: ''
Apr 18 20:24:46.816998 arc-switch1004 ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on getresponse
Apr 18 20:24:46.817243 arc-switch1004 ERR swss#orchagent: :- wait: failed to get response for getresponse
Apr 18 20:24:46.817405 arc-switch1004 ERR swss#orchagent: :- create: create status: SAI_STATUS_FAILURE
Apr 18 20:24:46.817552 arc-switch1004 ERR swss#orchagent: :- main: Failed to create a switch, rv:-1

Why I did it
To avoid timeout to create switch when testing ASAN builds with lower CPU systems

How I verified it
Loaded the build with changes and verified no issues are seen

Details if related

@dgsudharsan dgsudharsan requested a review from prsunny as a code owner May 15, 2023 18:19
prsunny
prsunny reviewed May 17, 2023
View reviewed changes
orchagent/main.cpp Outdated

auto delay_factor = 1;

#ifdef ASAN_ENABLED
Copy link
Copy Markdown
Collaborator

@prsunny prsunny May 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer not use #ifdefs in code. @xumia do you've any suggestions?

@prsunny prsunny requested a review from xumia May 17, 2023 23:34
prsunny
prsunny approved these changes May 19, 2023
View reviewed changes
Copy link
Copy Markdown
Collaborator

@prsunny prsunny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. @xumia to review as well.

@dgsudharsan
Copy link
Copy Markdown
Collaborator Author

dgsudharsan commented May 23, 2023

@xumia Can you please help to close the review?

@prsunny
Copy link
Copy Markdown
Collaborator

prsunny commented May 27, 2023

@dgsudharsan , can you please update the description as to what was the previous timeout and what is the new value in seconds?

@dgsudharsan
Copy link
Copy Markdown
Collaborator Author

dgsudharsan commented May 28, 2023

@dgsudharsan , can you please update the description as to what was the previous timeout and what is the new value in seconds?

Done

@qiluo-msft qiluo-msft merged commit 90b34d4 into sonic-net:master May 30, 2023
StormLiangMS pushed a commit that referenced this pull request Jun 10, 2023
@dgsudharsan @StormLiangMS
[ASAN]Increasing switch create timeout for ASAN images (#2777)
1e8eaa7 
**What I did**
When running ASAN images with low CPU systems, due to the additional overhead associated with ASAN, it results in switch create timeout as seen in logs below. Hence increasing the **switch create timeout to 2x for ASAN builds.** e.g. For regular switch scenario, the timeout will be 120 seconds  instead of 60 seconds.

```
Apr 18 20:23:46.749074 arc-switch1004 NOTICE swss#orchagent: :- create: request switch create with context 0
Apr 18 20:23:46.749074 arc-switch1004 NOTICE swss#orchagent: :- allocateNewSwitchObjectId: created SWITCH VID oid:0x21000000000000 for hwinfo: ''
Apr 18 20:24:46.816998 arc-switch1004 ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on getresponse
Apr 18 20:24:46.817243 arc-switch1004 ERR swss#orchagent: :- wait: failed to get response for getresponse
Apr 18 20:24:46.817405 arc-switch1004 ERR swss#orchagent: :- create: create status: SAI_STATUS_FAILURE
Apr 18 20:24:46.817552 arc-switch1004 ERR swss#orchagent: :- main: Failed to create a switch, rv:-1

```

**Why I did it**
To avoid timeout to create switch when testing ASAN builds with lower CPU systems

**How I verified it**
Loaded the build with changes and verified no issues are seen
theasianpianist pushed a commit to theasianpianist/sonic-swss that referenced this pull request Jul 20, 2023
@dgsudharsan @theasianpianist
[ASAN]Increasing switch create timeout for ASAN images (sonic-net#2777)
e33d2e1 
**What I did**
When running ASAN images with low CPU systems, due to the additional overhead associated with ASAN, it results in switch create timeout as seen in logs below. Hence increasing the **switch create timeout to 2x for ASAN builds.** e.g. For regular switch scenario, the timeout will be 120 seconds  instead of 60 seconds.

```
Apr 18 20:23:46.749074 arc-switch1004 NOTICE swss#orchagent: :- create: request switch create with context 0
Apr 18 20:23:46.749074 arc-switch1004 NOTICE swss#orchagent: :- allocateNewSwitchObjectId: created SWITCH VID oid:0x21000000000000 for hwinfo: ''
Apr 18 20:24:46.816998 arc-switch1004 ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on getresponse
Apr 18 20:24:46.817243 arc-switch1004 ERR swss#orchagent: :- wait: failed to get response for getresponse
Apr 18 20:24:46.817405 arc-switch1004 ERR swss#orchagent: :- create: create status: SAI_STATUS_FAILURE
Apr 18 20:24:46.817552 arc-switch1004 ERR swss#orchagent: :- main: Failed to create a switch, rv:-1

```

**Why I did it**
To avoid timeout to create switch when testing ASAN builds with lower CPU systems

**How I verified it**
Loaded the build with changes and verified no issues are seen
Janetxxx pushed a commit to Janetxxx/sonic-swss that referenced this pull request Nov 10, 2025
@dgsudharsan
[ASAN]Increasing switch create timeout for ASAN images (sonic-net#2777)
fc8ac23 
**What I did**
When running ASAN images with low CPU systems, due to the additional overhead associated with ASAN, it results in switch create timeout as seen in logs below. Hence increasing the **switch create timeout to 2x for ASAN builds.** e.g. For regular switch scenario, the timeout will be 120 seconds  instead of 60 seconds.

```
Apr 18 20:23:46.749074 arc-switch1004 NOTICE swss#orchagent: :- create: request switch create with context 0
Apr 18 20:23:46.749074 arc-switch1004 NOTICE swss#orchagent: :- allocateNewSwitchObjectId: created SWITCH VID oid:0x21000000000000 for hwinfo: ''
Apr 18 20:24:46.816998 arc-switch1004 ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on getresponse
Apr 18 20:24:46.817243 arc-switch1004 ERR swss#orchagent: :- wait: failed to get response for getresponse
Apr 18 20:24:46.817405 arc-switch1004 ERR swss#orchagent: :- create: create status: SAI_STATUS_FAILURE
Apr 18 20:24:46.817552 arc-switch1004 ERR swss#orchagent: :- main: Failed to create a switch, rv:-1

```

**Why I did it**
To avoid timeout to create switch when testing ASAN builds with lower CPU systems

**How I verified it**
Loaded the build with changes and verified no issues are seen
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@prsunny prsunny prsunny approved these changes

@oleksandrivantsiv oleksandrivantsiv oleksandrivantsiv approved these changes

@xumia xumia xumia approved these changes

Assignees

No one assigned

Labels

Included in 202211 Branch Request for 202211 Branch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@dgsudharsan @prsunny @oleksandrivantsiv @xumia @qiluo-msft @StormLiangMS