You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| 1.0 | 06/22/2021 | Vivek Reddy Karri | Auto Invocation of Techsupport, triggered by a core dump |
28
28
| 1.1 | TBD | Vivek Reddy Karri | Add the capability to Register/Deregister app extension to AUTO_TECHSUPPORT_FEATURE table |
29
29
| 2.0 | TBD | Vivek Reddy Karri | Extending Support for Kernel Dumps |
30
+
| 3.0 | 02/2022 | Stepan Blyshchak | Extending Support for memory usage threshold crossed |
30
31
31
32
## About this Manual
32
33
This document describes the details of the system which facilitates the auto techsupport invocation support in SONiC. The auto invocation is triggered when any process inside the docker crashes and a core dump is generated.
@@ -36,13 +37,20 @@ Currently, techsupport is run by invoking `show techsupport` either by orchestra
36
37
37
38
However if the techsupport invocation can be made event-driven based on core dump generation, that would definitely improve the debuggability. That is the overall idea behind this HLD. All the high-level requirements are summarized in the next section
38
39
40
+
Another use case is to gather more information about the system in case there is a memory usage threshold crossed.
41
+
SONiC dump generated after system reboots due to out of memory is not enough for debugging the issue
42
+
as all the information about processes and their mem usage, smaps (/proc/PID/smaps) is lost.
43
+
Once the system detects abnormal memory usage SONiC dump is generated automatically.
44
+
39
45
## 2. High Level Requirements
40
46
### Global Scope
41
47
* Techsupport invocation should also be made event-driven based on core dump generation.
42
48
* This is only applicable for the processes running inside the dockers. Does not apply for other processes.
43
49
* init_cfg.json will be enhanced to include the "CONFIG" required for this feature (described in section 4) and is enabled by default.
44
50
* To provide flexibility, a compile time flag "ENABLE_AUTO_TECH_SUPPORT" should be provided to enable/disable the "CONFIG" for this feature.
45
51
* Users should have the abiliity to enable/disable this capability through CLI.
52
+
* Techsupport invocation should also be made event-driven based on memory usage threshold crossing.
53
+
* The memory usage threshold should be configurable system-wise and per container.
46
54
47
55
### Configurable Params
48
56
* A configurable "rate_limit_interval" should be introduced to limit the number consecutive of techsupport invocations.
@@ -68,14 +76,59 @@ The naming format and compression is governed by the script `/usr/local/bin/core
68
76
69
77
Where `<comm>` value in the command name associated with a process. comm value of a running process can be read from `/proc/[pid]/comm` file
```${container}_mem_usage``` used memory by $container ("docker stats --no-stream --format {{.MemUsage}}" $container),
90
+
91
+
```${container}_mem_usage_threshold``` configured memory threshold for $container, (100 - ${container}available_mem_threshold),
92
+
93
+
```mem_free``` is the total minus mem usage, ```mem_free_threshold``` - mem free threshold.
94
+
95
+
the SONiC techsupport is automatically generated.
96
+
97
+
```mem_free_threshold``` is there to invoke dump when there is quite small amount of memory left that is needed to successfully execute "show techsupport". This is going to be 200 MB by default, as at least 80-90 MB takes "show techsupport" execution.
98
+
99
+
The check will be implemented as a script that is ran by monit periodically:
100
+
101
+
```
102
+
check program mem_checker with path "/usr/bin/mem_threshold_check"
103
+
if status != 0 for 10 times within 20 cycles then exec /usr/local/bin/mem_threshold_check_handler"
104
+
```
105
+
106
+
The action is going to be ran only once the mem_check script detects memory usage above threshold.
107
+
108
+
The "10 times within 20 cycles" part is kept in sync with mem_usage alert from sonic-host monit configuration.
109
+
It is possible to make those values configurable however, only the threshold value is considered to be configurable.
110
+
111
+
The rate limit as well as techsupport maximum limit is applicable to techsupport generated by memory check.
112
+
113
+
#### 202106 and older
114
+
115
+
To support thechsupport generation on memory leaks a simple rule to monit is added:
116
+
117
+
```
118
+
check system $HOST
119
+
if memory usage > 90% for 10 times within 20 cycles then exec /usr/bin/generate_dump
120
+
```
121
+
122
+
## 5. Schema Additions
72
123
73
124
### Config DB
74
125
75
126
#### AUTO_TECHSUPPORT Table
76
127
```
77
128
key = "AUTO_TECHSUPPORT|global"
78
129
state = "enabled" / "disabled" ; Enable this to make the Techsupport Invocation event driven based on core-dump generation
130
+
available_mem_threshold = 1*2DIGIT ; Memory threshold; 0 to disable techsupport invocation on mem leak.
131
+
min_available_mem = 1*5DIGIT ; Minimum free memory amount in MB when techsupport will be executed.
79
132
rate_limit_interval = 1*5DIGIT ; Minimum Time in seconds, between two successive techsupport invocations.
80
133
Manual Invocations will be considered as well in the calculation.
81
134
Configure 0 to explicitly disable
@@ -99,6 +152,7 @@ since = 1*32VCHAR; ; This limits the auto-invoke
99
152
```
100
153
key = feature name
101
154
state = "enabled" / "disabled" ; Enable auto techsupport invocation on the critical processes running inside this feature
155
+
available_mem_threshold = 1*2DIGIT ; Memory threshold; 0 to disable techsupport invocation on mem leak in this container.
102
156
rate_limit_interval = 1*5DIGIT ; Rate limit interval for the corresponding feature. Configure 0 to explicitly disable
@@ -374,7 +462,7 @@ Enhance the existing techsupport sonic-mgmt test with the following cases.
374
462
| 2 | Check if the techsupport cleanup is working as expected |
375
463
| 3 | Check if the global rate-& & per-process rate-limit-interval is working as expected |
376
464
| 4 | Check if the core-dump cleanup is working as expected |
377
-
465
+
| 5 | Check if the core-dump generated when reaching memory threshold |
378
466
## 8. SONiC-to-SONiC Upgrade Considerations
379
467
380
468
The default config required for auto_techsupport is present in the init_cfg.json. Therefore, when a clean installation of SONiC is performed, the configuration is found in the config DB and the feature is active.
@@ -391,6 +479,7 @@ Load this Example config provided below to enable the feature. Each of the field
391
479
"rate_limit_interval": "180",
392
480
"max_techsupport_limit": "10.0",
393
481
"max_core_limit": "5.0",
482
+
"available_mem_threashold": "10.0",
394
483
"since": "2 days ago"
395
484
}
396
485
},
@@ -461,4 +550,8 @@ Load this Example config provided below to enable the feature. Each of the field
461
550
}
462
551
}
463
552
}
553
+
```
554
+
555
+
# Open question
464
556
557
+
1. Is 10 % free memory/90 % used memory threshold a reasonable default?
0 commit comments