OOM killed when using the nuclei SDK with the standard templates #4756
Description
Nuclei version:
3.1.8
Current Behavior:
I run the nuclei SDK as part of a binary that is deployed in a linux container (alpine) with memory limits of 8GB and 16GB. I use the standard templates. In both cases it gets OOM killed. Here are the settings I specify.
Rate Limit: 150 per second
Exclude Severity []string{"info", "low"}
Template Concurrency 25
Host Concurrency 100
Scan Strategy "template-spray"
Network Timeout 10
Network Retries 2
Disable Host Errors true
Max Host Errors 15000
Probe Non Http Targets 0
Enable Code Templates 0
Stats true
I tried this with 115 and 380 hosts and both are having memory issues. What is causing the high memory utilization? I am saving the results from the nuclei scan in a list. Could the results be so large that they fill in the memory?
I run nuclei like this:
ne, err := nuclei.NewNucleiEngine(opts...)
if err != nil {
return err
}
defer ne.Close()
ne.LoadTargets(liveHosts, n.ProbeNonHttpTargets)
err = ne.LoadAllTemplates()
if err != nil {
return err
}
var results []*NucleiResult
err = ne.ExecuteWithCallback(func(event *output.ResultEvent) {
// Convert output.ResultEvent into NucleiResult ...
res := &NucleiResult{...}
results = append(results, res)
})
Expected Behavior:
The nuclei SDK should trivially handle scanning hosts with the above settings. It will be great to have an example of the SDK settings that match the default nuclei cli scan settings.
What would be the equivalent settings for the SDK?
nuclei -u example.com
Additionally what settings in the SDK control the memory utilization? It will be good to document those as well.
Steps To Reproduce:
Use the above settings and set up a scan. Watch it take a lot of memory over time. Better if you use 115 (or more) web sites.