By checking the tokio-console dashboard of RN, we spotted an unexpected behavior of the future derivation-pipeline worker. Normal future should have very short sched time(waiting for the tokio runtime to execute it) and long Idle time, whereas the the future derivation-pipeline worker has a very long sched time but short Idle time, which means the future is starved to be executed.
Warn ID State Name Total Busy Sched Idle Polls Kind Location Fields-
! 1 80 IDLE 10m43s 173ms 10m29s 13s 294 task /app/crates/derivation-pipeline/src/lib.rs:153:9 size.bytes=632 target=tokio::task
We better check the rationale of this behavior.