Skip to content

Conversation

@tarunKoyalwar
Copy link
Member

@tarunKoyalwar tarunKoyalwar commented Jan 24, 2024

Proposed changes

  • Implement Response Chain
    • doesn't re-allocate when memory when following redirect chain instead reuses them
    • global http response buffers reuse
  • adds -jsc flag javascript concurrency with 120 default (i.e in parallel 120 flow executors and 120 javascript protocols exec can occur)
  • refactor javascript vm/compiler logic + remove unused code
  • overall improvements to memory and cpu usage
  • follow up issue for extended memory usage optimizations Extended Memory Usage Optimizations #4703

Observations

  • From testing it was observed that running nuclei on 1k targets with default concurrency settings runs without crash and requires 2vcpu and 3GB(depends on http response of target, but 3GB is good maximum to have)

ccf5d099-9942-4a57-a501-bf1daa703787

753e0ac0-e9a8-4328-95ee-b2a672f3dfdf

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

@tarunKoyalwar tarunKoyalwar changed the title use bytes.buffer and reuse buffers for response management memory leak fixes and optimizations Jan 25, 2024
Copy link
Member

@Mzack9999 Mzack9999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggesting minor spelling changes + question on failing tests

@tarunKoyalwar tarunKoyalwar marked this pull request as draft January 29, 2024 15:34
@tarunKoyalwar tarunKoyalwar self-assigned this Jan 29, 2024
@tarunKoyalwar
Copy link
Member Author

tarunKoyalwar commented Jan 29, 2024

TODO

  • above changes (related to js) were experimental but idea seems to have worked. need to do following before fix is ready
    • fix failing integration test (caused by http response chain)
    • remove dead code ( we don't use capture variables - not supported in templates)
    • finalize implementations
    • do some scans probably 5k targets + javascript templates to identify appropriate js concurrency from practical testing
    • add -jc -js-concurrency to control/limit max js vms or (in limit total inuse vm's) . this flag govern's both flow executor js and javascript protocol js vm

@tarunKoyalwar tarunKoyalwar marked this pull request as ready for review January 30, 2024 15:30
@ehsandeep ehsandeep requested a review from Mzack9999 January 30, 2024 16:15
@ehsandeep ehsandeep merged commit 5bd9d9e into dev Jan 30, 2024
@ehsandeep ehsandeep deleted the issue-4552-update2 branch January 30, 2024 20:29
@tarunKoyalwar tarunKoyalwar linked an issue Jan 31, 2024 that may be closed by this pull request
@rxerium
Copy link

rxerium commented Feb 10, 2024

Hey team, I'm still experiencing memory issues unfortunately (on version 3.1.10). I'm reverting to version 2.9.15 to resolve the issue and also note I'm running scans through docker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

-cloud-upload not working as expected

5 participants