Skip to content

Conversation

@Mzack9999
Copy link
Member

@Mzack9999 Mzack9999 commented Mar 5, 2024

Proposed changes

  • Introducing the memguardian component that actively monitor used memory. Activation is optional via export MEMGUARDIAN=1. If the used RAM is above the threshold (default 75% of available RAM). It reduces parallel threading of templates to either MEMGUARDIAN_THREADS if defined, or 1/5 of the total original threading. Also if MEMGUARDIAN_ALLOC is defined, it caps the maximum amount of buffers that can be allocated in the pool. When memory conditions are back to normal, the slow down mechanisms are automatically reverted.
  • collecting events within ScanContext is now optional, as it's not used anywhere and it was preventing gc to recycle large maps of data (and probably a co-cause of running nuclei with sdk may have gc problem #4836)

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Example

Closes #4756

$ MEMGUARDIAN=1 go run . -l 790targets.txt -jsc 110 -stats -tags fuzz,fuzzing,brute-force,bruteforce,brute,dos,local,code  -severity critical -rl 0
...
[INF] Templates loaded for current scan: 6
[INF] Executing 4 signed templates from projectdiscovery/nuclei-templates
[WRN] Executing 3 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 790
[0:00:06] | Templates: 6 | Hosts: 790 | RPS: 1796 | Matched: 0 | Errors: 0 | Requests: 11227/6737120 (0%)
...
[0:35:50] | Templates: 6 | Hosts: 790 | RPS: 686 | Matched: 0 | Errors: 37507 | Requests: 6417135/6737120 (95%)

@Mzack9999 Mzack9999 self-assigned this Mar 6, 2024
@Mzack9999 Mzack9999 added the Type: Enhancement Most issues will probably ask for additions or changes. label Mar 6, 2024
@Mzack9999 Mzack9999 marked this pull request as ready for review March 6, 2024 12:53
@Mzack9999 Mzack9999 requested a review from ehsandeep March 13, 2024 19:15
@ehsandeep ehsandeep requested a review from tarunKoyalwar March 13, 2024 21:40
@Mzack9999 Mzack9999 merged commit b7f76cf into dev Mar 14, 2024
@Mzack9999 Mzack9999 deleted the maint-memory branch March 14, 2024 22:39
@Mzack9999
Copy link
Member Author

Optional behavior - Merging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type: Enhancement Most issues will probably ask for additions or changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OOM killed when using the nuclei SDK with the standard templates

3 participants