Memory Bug

[TheTechromancer]

Memory usage gets really high during some scans. Until we get a debug.log for one of these scans, it's a mystery as to why.

@Amiremami

[TheTechromancer]

I was able to reproduce this with debug enabled. The culprit doesn't appear to be a module:

2024-08-14 23:20:49,341 [DEBUG] bbot.scanner scanner.py:1029 MODULE MEMORY USAGE:
2024-08-14 23:20:49,342 [DEBUG] bbot.scanner scanner.py:1029     - bucket_amazon: 2.50MB
2024-08-14 23:20:49,342 [DEBUG] bbot.scanner scanner.py:1029     - bucket_azure: 2.50MB
2024-08-14 23:20:49,342 [DEBUG] bbot.scanner scanner.py:1029     - bucket_firebase: 2.50MB
2024-08-14 23:20:49,342 [DEBUG] bbot.scanner scanner.py:1029     - bucket_google: 2.50MB
2024-08-14 23:20:49,342 [DEBUG] bbot.scanner scanner.py:1029     - json: 2.00MB
2024-08-14 23:20:49,342 [DEBUG] bbot.scanner scanner.py:1029     - dnsbrute_mutations: 933.06KB

Doing a second run with memray.

[TheTechromancer]

Memray reports pretty normal memory usage:

So it seems like either massdns/httpx is responsible, or else one of the engines is mibehaving. Running again with a ps monitor.

[TheTechromancer]

Looks like a memory leak in the web helper:

----------------------------------------
Timestamp: 2024-08-16 04:12:45
Top Memory Usage Processes:
PID USER %MEM MB COMMAND
459384 root 13.6% 1083.85 MB BBOT WebHelper 
459376 root 5.3% 428.63 MB /root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/bin/python -c import sys; from importlib import import_module; sys.argv = ['/root/.cache/pypoetry/virtualenvs/bbot-aKeTQkGo-py3.12/bin/bbot', '-t', 'dell.com', '-p', 'subdomain-enum', 'web-basic']; sys.exit(import_module('bbot.cli').main()) 
5710 root 1.4% 117.17 MB tmux 
459392 root 1.1% 87.38 MB BBOT DNSHelper 
459432 root 1.0% 80.30 MB bbot process pool 
459437 root 0.9% 72.26 MB bbot process pool 
459480 root 0.8% 71.32 MB bbot process pool 
459669 nobody 0.6% 50.22 MB massdns -r /root/.bbot/cache/25152d64f6205b681a2b170969063ff7030fd01c -s 1000 -t A -o J -q 
459670 root 0.5% 40.71 MB httpx -silent -json -include-response -threads 50 -timeout 5 -retries 1 -header User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.2151.97 -response-size-to-read 5242880 -r 1.1.1.1,1.0.0.1 
399 root 0.4% 37.25 MB /usr/bin/containerd 
----------------------------------------

[TheTechromancer]

The culprit seems to be httpx:

...although this might actually be an extremely large response, i.e. from wayback.

[TheTechromancer]

It really does look like a memory leak in httpx:

[TheTechromancer]

There is a discussion for a similar issue: encode/httpx#2785.

[TheTechromancer]

Fixed in #1667.