running nuclei with sdk may have gc problem

[CyL0NG]

golang version: go1.21.1
platform: mac
nuclei version: v3.1.10
the following code will output:

0xc00088fd60, [0xc00ce126c0]
0xc00088fd60, [0xc00ce126c0 0xc0004598c0]
0xc00088fd60, [0xc00ce126c0 0xc0004598c0 0xc001428b40]
0xc00088fd60, [0xc00ce126c0 0xc0004598c0 0xc001428b40 0xc001428d80]
0xc00088fd60, [0xc00ce126c0 0xc0004598c0 0xc001428b40 0xc001428d80 0xc001428fc0]

the ret slice init every time in run function, it should not grow every time. When I put ret = nil at the end line in run function, it't output like this:

0xc000999900, [0xc006dbe6c0]
0xc00cafa468, [0xc004f92d80]
0xc007346540, [0xc004f92d80 0xc008b2a900]
0xc00518a040, [0xc004f92d80 0xc008b2a900 0xc008b2a6c0]
0xc00518a040, [0xc004f92d80 0xc008b2a900 0xc008b2a6c0 0xc008b2ab40]

it's like a gc problem, but use runtime.GC() is not working.

package main

import (
	"fmt"
	nuclei "github.com/projectdiscovery/nuclei/v3/lib"
	"github.com/projectdiscovery/nuclei/v3/pkg/output"
)

func run() {
	ret := make([]*output.ResultEvent, 0, 10)
	ne, err := nuclei.NewThreadSafeNucleiEngine()
	if err != nil {
		return
	}
	cb := func(event *output.ResultEvent) {
		ret = append(ret, event)
		fmt.Printf("%p, %v\n", ret, ret)
	}
	ne.GlobalResultCallback(cb)

	tf := nuclei.TemplateFilters{IDs: []string{"exposed-redis"}}
	opts := []nuclei.NucleiSDKOptions{
		nuclei.WithTemplateFilters(tf),
	}
	err = ne.ExecuteNucleiWithOpts([]string{"127.0.0.1:6379"}, opts...)
	if err != nil {
		fmt.Println("err: ", err)
	}
	ne.Close()
}

func main() {
	for i := 0; i < 5; i++ {
		run()
	}
}

[Mzack9999]

@CyL0NG nice catch - I could reproduce the behavior. It seems like that for some reason the output.ResultEvent remains hanging around, I'm investigating for a potential fix

[jak0ne]

Hello,

I have experienced the same issue when running single instance, should the cache be purged when calling ExecuteWithCallback as well?

package main

import (
	"fmt"
	nuclei "github.com/projectdiscovery/nuclei/v3/lib"
	"github.com/projectdiscovery/nuclei/v3/pkg/output"
)

func run() {
	ret := make([]*output.ResultEvent, 0, 10)

	ne, err := nuclei.NewNucleiEngine(
		nuclei.WithTemplateFilters(nuclei.TemplateFilters{
			IDs: []string{"git-config"},
		}),
	)
	if err != nil {
		return
	}
	defer ne.Close()

	ne.LoadTargets([]string{"http://127.0.0.1:8000"}, false)

	err = ne.ExecuteWithCallback(func(event *output.ResultEvent) {
		ret = append(ret, event)
		fmt.Printf("%p, %v\n", ret, ret)
	})
	if err != nil {
		fmt.Println("err: ", err)
	}
}

func main() {
	for i := 0; i < 5; i++ {
		run()
	}
}

Output:

0x14000cb18b0, [0x140049abd40]
0x14000cb18b0, [0x140049abd40 0x1400ce93b00]
0x14000cb18b0, [0x140049abd40 0x1400ce93b00 0x1400036d680]
0x14000cb18b0, [0x140049abd40 0x1400ce93b00 0x1400036d680 0x140009938c0]
0x14000cb18b0, [0x140049abd40 0x1400ce93b00 0x1400036d680 0x140009938c0 0x1400ce938c0]

[jak0ne]

Hey,

I tested after PR merged and the issue still remains in single instance. Should the cache also be purged when running non global callbacks?

Thanks!

[Mzack9999]

@jak0ne Thanks for the reporting this additional problem. I'm reopening the issue and will implement a follow up PR for what appears to be an issue at design level, as the cache should just hold abstracted representation usable concurrently by multiple instances, with the custom settings only applied on a per-runner basis.

[jak0ne]

Thanks @Mzack9999 ! I am experiencing a weird behavior and I'm not sure if it's because of this caching issue or a problem in my code.

Basically I run single instance nuclei engines against a single target on each iteration, and I'm getting callbacks from previous loop iterations. Here's a simplified version of the code:

package main

import (
	"fmt"
	nuclei "github.com/projectdiscovery/nuclei/v3/lib"
	"github.com/projectdiscovery/nuclei/v3/pkg/output"
)

func main() {

	done := make(chan struct{})

	targets := map[string]string{
		"127.0.0.1:8000": "tcp",
		"127.0.0.1:8001": "tcp",
		"http://127.0.0.1:8000": "http",
		"http://127.0.0.1:8001": "http",
	}

	go func(targets map[string]string) {
		for target, protocol := range targets {
			ne, _ := nuclei.NewNucleiEngine(nuclei.WithTemplateFilters(
				nuclei.TemplateFilters{
					ProtocolTypes: protocol,
					ExcludeTags: []string{"bruteforce", "dos", "fuzzing"},
				}),
			)

			ne.LoadTargets([]string{target}, false)

			_ = ne.ExecuteWithCallback(func(event *output.ResultEvent) {
				if target != event.URL {
					fmt.Printf("%s != %s\n", target, event.URL)
				}
			})

			ne.Close()
		}
		done <-struct{}{}
	}(targets)

	<-done
}

I expected each nuclei instance to be a clean slate on every loop iteration (no cached templates or results from previous ones). Is it a wrong assumption? Any suggestions here?

Thanks!

[ehsandeep]

@jak0ne see follow up ticket #4863 for additional information.