GC activities are extremely slow and it is working sequentially and each "marking" step takes 10 to 20 seconds.

[mfarooq-tech]

Bug report:

GC activities are extremely slow and it is working sequentially and each "marking" step takes 10 to 20 seconds.

{"level":"info","ts":"2021-11-10 23:31:10.953","caller":"client/client.go:224","msg":"leave from task result: true for taskId: 375fc2b79210bce0524323335c11d122af5347253c45dee6
2d8a6a2b090c186d, scheduler server node: dns:///x.y.z.100:8002, err:","peerId":"x.y.z.11-1-fa9a35f7-61c3-4928-99ff-0f74ee4bfbc9","errMsg":null}
{"level":"info","ts":"2021-11-10 23:31:10.953","caller":"daemon/daemon.go:121","msg":"step 4:leave task 375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d/x.y.z.
11-1-fa9a35f7-61c3-4928-99ff-0f74ee4bfbc9 state ok"}
{"level":"info","ts":"2021-11-10 23:31:10.953","caller":"storage/local_storage.go:369","msg":"task 375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d/x.y.z.11-1-
fa9a35f7-61c3-4928-99ff-0f74ee4bfbc9 will be reclaimed, marked","task":"375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d","peer":"x.y.z.11-1-fa9a35f7-61c3-4928
-99ff-0f74ee4bfbc9","component":"io.d7y.storage.v2.advance"}
{"level":"info","ts":"2021-11-10 23:31:23.780","caller":"client/client.go:224","msg":"leave from task result: true for taskId: 375fc2b79210bce0524323335c11d122af5347253c45dee6
2d8a6a2b090c186d, scheduler server node: dns:///x.y.z.100:8002, err:","peerId":"x.y.z.11-1-fee567a5-0706-4e03-b1bc-4c5087852c51","errMsg":null}
{"level":"info","ts":"2021-11-10 23:31:23.780","caller":"daemon/daemon.go:121","msg":"step 4:leave task 375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d/x.y.z.
11-1-fee567a5-0706-4e03-b1bc-4c5087852c51 state ok"}
{"level":"info","ts":"2021-11-10 23:31:23.780","caller":"storage/local_storage.go:369","msg":"task 375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d/x.y.z.11-1-
fee567a5-0706-4e03-b1bc-4c5087852c51 will be reclaimed, marked","task":"375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d","peer":"x.y.z.11-1-fee567a5-0706-4e03
-b1bc-4c5087852c51","component":"io.d7y.storage.v2.advance"}
{"level":"info","ts":"2021-11-10 23:31:33.702","caller":"client/client.go:224","msg":"leave from task result: true for taskId: 375fc2b79210bce0524323335c11d122af5347253c45dee6
2d8a6a2b090c186d, scheduler server node: dns:///x.y.z.100:8002, err:","peerId":"x.y.z.11-1-0945465c-bb96-460f-8e24-ad69bfe21a93","errMsg":null}
{"level":"info","ts":"2021-11-10 23:31:33.703","caller":"daemon/daemon.go:121","msg":"step 4:leave task 375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d/x.y.z.
11-1-0945465c-bb96-460f-8e24-ad69bfe21a93 state ok"}
{"level":"info","ts":"2021-11-10 23:31:33.703","caller":"storage/local_storage.go:369","msg":"task 375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d/x.y.z.11-1-
0945465c-bb96-460f-8e24-ad69bfe21a93 will be reclaimed, marked","task":"375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d","peer":"x.y.z.11-1-0945465c-bb96-460f
-8e24-ad69bfe21a93","component":"io.d7y.storage.v2.advance"}
{"level":"info","ts":"2021-11-10 23:31:45.779","caller":"client/client.go:224","msg":"leave from task result: true for taskId: 375fc2b79210bce0524323335c11d122af5347253c45dee6
2d8a6a2b090c186d, scheduler server node: dns:///x.y.z.100:8002, err:","peerId":"x.y.z.11-1-393c78b7-3d6f-418b-884d-f378b8284d50","errMsg":null}
{"level":"info","ts":"2021-11-10 23:31:45.779","caller":"daemon/daemon.go:121","msg":"step 4:leave task 375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d/x.y.z.
11-1-393c78b7-3d6f-418b-884d-f378b8284d50 state ok"}
{"level":"info","ts":"2021-11-10 23:31:45.779","caller":"storage/local_storage.go:369","msg":"task 375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d/x.y.z.11-1-
393c78b7-3d6f-418b-884d-f378b8284d50 will be reclaimed, marked","task":"375fc2b79210bce0524323335c11d122af5347253c45dee62d8a6a2b090c186d","peer":"x.y.z.11-1-393c78b7-3d6f-418b
-884d-f378b8284d50","component":"io.d7y.storage.v2.advance"}

Expected behavior:

How to reproduce it:

Environment:

• Dragonfly version: beta.1
• OS:
• Kernel (e.g. uname -a):
• Others:

Logs:
core.log.gz

[jim3ma]

When mark a peer task for gc, client will call scheduler's LeaveTask, it seems this call slowly.

[mfarooq-tech]

This slow step will be problematic when disk is almost full and gc is not cleaning fast enough.

[gaius-qi]

Please use latest version.