allow multiple instance of Suricata in PCAP processing mode via UNIX socket

[mmguero]

In v25.02.0 we implemented #457 which improved performance of PCAP processing by spinning up a single long-lived instance of Suricata and then submitting PCAPs to it via a UNIX socket. This massively helps in the "lots of smaller PCAPs" case.

I even set it up to support the SURICATA_AUTO_ANALYZE_PCAP_THREADS environment variable. I asked about it on the Suricata forums and didn't get much of a response, but in my testing I could see it affect CPU utilization somewhat and (somewhat prematurely, it would seem) decided it looked like it was working.

For some PCAP sets it does seem to have more of an effect than others, but on large PCAP sets (maybe that are more homogeneous? not 100% sure of what it does threading-wise within one PCAP) it still seemed to be basically just using one CPU.

Reading the Suricata documentation more carefully tells us (emphasis added):

You can add multiple files without waiting for each to be processed; they will be sequentially processed and the generated log/alert files will be put into the directory specified as second argument of the pcap-file command.

So... sequentially 😦 .

So, what we're probably going to do is have TWO environment variables for the suricata-offline configuration, something like:

• SURICATA_SOCKET_PROCESSES=x
• SURICATA_AUTO_ANALYZE_PCAP_THREADS=y

Where x is the number of independent suricata-offline socket-controllable processes, and y is the number of threads each one is given (today's SURICATA_AUTO_ANALYZE_PCAP_THREADS value).

Then, the pcap processor will do something like query the count of the PCAP files currently being processed by each suricata socket and always submit to the lowest one. This will allow us to more efficiently work through PCAP sets without getting backed up.

[mmguero]

• example config/suricata-offline.env:

...
# The number of Suricata processes for analyzing uploaded PCAP files allowed
#   to run concurrently
SURICATA_AUTO_ANALYZE_PCAP_PROCESSES=3
# The number of threads per Suricata process (0 means "default behavior")
SURICATA_AUTO_ANALYZE_PCAP_THREADS=2
...

• startup logs (with debug -v)

suricata-1  | 2025-07-09 18:10:17,107 INFO success: pcap-suricata entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
suricata-1  | 2025-07-09 18:10:17,107 INFO success: socket-suricata-1 entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
suricata-1  | 2025-07-09 18:10:17,107 INFO success: socket-suricata-2 entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
suricata-1  | Info: unix-manager: unix socket '/var/run/suricata/suricata-command-2.socket'
suricata-1  | Notice: threads: Threads created ->   Engine started.
suricata-1  | Info: unix-manager: unix socket '/var/run/suricata/suricata-command-1.socket'
suricata-1  | Notice: threads: Threads created ->   Engine started.
suricata-1  | 2025-07-09 18:10:46 INFO: pcap_suricata_processor.py:	subscribed to topic at 30441
suricata-1  | 2025-07-09 18:10:46 INFO: pcap_suricata_processor.py[1]:	started
suricata-1  | 2025-07-09 18:10:46 INFO: pcap_suricata_processor.py[1]:	Connected to suricata-command-1.socket
suricata-1  | 2025-07-09 18:10:46 INFO: pcap_suricata_processor.py[1]:	Connected to suricata-command-2.socket

• after submitting a bunch of PCAPs for processing

$ docker compose exec suricata bash -c "for FILE in /var/run/suricata/suricata-command-*.socket; do echo -n \"\$FILE: \"; suricatasc -c 'pcap-file-number' "\$FILE" | jq '.message'; done"
/var/run/suricata/suricata-command-1.socket: 17
/var/run/suricata/suricata-command-2.socket: 18
/var/run/suricata/suricata-command-3.socket: 18