PF_EXITING goto-local path can strand exiting tasks on nohz_full cpus

[root-drift-flux]

Summary

First time issue submitter here.

I've been hitting repeated runnable task stall watchdog ejections on a custom sched_ext scheduler under a wine/cargo workload, and I believe I've traced the enqueue-side trigger to the default PF_EXITING shortcut in do_enqueue_task() interacting badly with nohz_full iso cpus. Setting SCX_OPS_ENQ_EXITING and handling the PF_EXITING case explicitly in ops.enqueue with SCX_ENQ_PREEMPT empirically closes the window. Filing this for confirmation from maintainers — the enqueue-side race is clear-cut, but I don't have a complete model of the full dwell mechanism and would appreciate a sanity check before framing a patch.

The existing SCX_OPS_ENQ_EXITING kdoc describes a different failure mode (bpf_task_from_pid() lookup failure, RCU grace period stalls) and gives no indication that the default path is unsafe on nohz_full — that's the gap I'd like to close.

Environment

• Kernel: 6.19.11-zen (verified the relevant paths are byte-identical on torvalds/master HEAD, see below)
• Hardware: Ryzen 7 5700X (8c/16t, single CCD)
• Cmdline: nohz_full=1-7,9-15 rcu_nocbs=1-7,9-15 isolcpus=nohz,domain,managed_irq,1-7,9-15
• Scheduler: custom scx_cages — strict-priority 4-tier DSQ scheduler. Iso cores (1-7, 9-15) run only game/"wrapped" and "promoted" tiers via custom DSQs; HK cores (0, 8) run janitor. Iso cores never run janitor, so there is no SCX slack work that would force schedule() on an idle iso cpu.
• Workload that reproduces: Steam/wine game (Helldivers 2) running on iso cores via the wrapped tier, plus a concurrent cargo build --release -j16 under iso.slice/promoted. Reliably triggers within ~90s.

Symptom

sched_ext: cages: runnable task stall (wineserver[285798] failed to run for 36.288s)
sched_ext: cages: runnable task stall (cargo[265224]    failed to run for 31.656s)
sched_ext: cages: runnable task stall (rustc[278231]    failed to run for 32.683s)
sched_ext: cages: runnable task stall (i386-linux-gnu-[287471] failed to run for 31.970s)
sched_ext: cages: runnable task stall (cc1[290477]      failed to run for 30.371s)
sched_ext: cages: runnable task stall (opt cgu.0[289296] failed to run for 30.789s)

10+ ejections in a single session across wineserver, cargo, rustc, cc1, ld.lld, opt cgu (rustc codegen units). The scheduler is auto-restarted by its service supervisor after each eject.

Every stalled task is in the do_exit → do_group_exit → __x64_sys_exit_group family at eject time

Evidence from scx_dump_state

I enabled the scx exit-time debug dump (set exit_dump_len on the struct_ops shadow before load — required, scx_dump_state at kernel/sched/ext.c:4581 is a no-op with len=0). Sample from the wineserver[285798] eject:

CPU 2   : nr_run=1 flags=0x1 ops_qseq=89544392
          curr=swapper/2[0] class=idle_sched_class

  R wineserver[285798] -36288ms
      scx_state/flags=3/0x1 dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=0x8000000000000002
      dsq_vtime=0 slice=20000000 weight=100
      cpus=fefe no_mig=0

    do_exit+0x32a/0xa60
    do_group_exit+0x8b/0x90
    __x64_sys_exit_group+0x17/0x20
    x64_sys_call+0x15e0/0x1870
    do_syscall_64+0x73/0x290
    entry_SYSCALL_64_after_hwframe+0x76/0x7e

Event counters
--------------
SCX_EV_SELECT_CPU_FALLBACK:               26
SCX_EV_DISPATCH_LOCAL_DSQ_OFFLINE:         0
SCX_EV_DISPATCH_KEEP_LAST:                 0
SCX_EV_ENQ_SKIP_EXITING:                1009
SCX_EV_ENQ_SKIP_MIGRATION_DISABLED:        0
SCX_EV_REFILL_SLICE_DFL:                1012
SCX_EV_BYPASS_DURATION:              9754621
SCX_EV_BYPASS_DISPATCH:                    3
SCX_EV_BYPASS_ACTIVATE:                    1

Five facts from this:

1. dsq_id = 0x8000000000000002 = SCX_DSQ_LOCAL (= SCX_DSQ_FLAG_BUILTIN | 2, include/linux/sched/ext.h:59). Task is associated with a per-cpu local DSQ. Combined with walking rq->scx.runnable_list from CPU 2's rq, it's CPU 2's local DSQ specifically.

2. ops_state/qseq = 0/0 (SCX_OPSS_NONE). Tasks dispatched via ops.enqueue transition through SCX_OPSS_QUEUEING (set at ext.c:1393). NONE with qseq=0 means ops.enqueue was never called for this task on this enqueue — i.e. the goto-local shortcut was taken, bypassing the scheduler.

3. slice = 20000000 = 20ms = SCX_SLICE_DFL (include/linux/sched/ext.h:30). The scheduler's tier_slice() would have returned 5ms for the wrapped tier and 2ms for janitor. The slice here was set by refill_task_slice_dfl on the enqueue: label at ext.c:1435, which only fires when goto-local / goto-global / goto-bypass was taken. Matches the bypass conclusion from (2).

4. SCX_EV_ENQ_SKIP_EXITING = 1009. This counter is only incremented in one place — the PF_EXITING goto-local shortcut at ext.c:1375. 1009 PF_EXITING tasks were routed through it in the scheduler's lifetime window before the eject.

5. Every stalled task's stack ends in exit_group. Confirms the category.

Analysis (enqueue-side)

With SCX_OPS_ENQ_EXITING unset (default), do_enqueue_task() at kernel/sched/ext.c:1372-1377:

    /* see %SCX_OPS_ENQ_EXITING */
    if (!(sch->ops.flags & SCX_OPS_ENQ_EXITING) &&
        unlikely(p->flags & PF_EXITING)) {
        __scx_add_event(sch, SCX_EV_ENQ_SKIP_EXITING, 1);
        goto local;
    }

Flow: skip ops.enqueue, goto local, fall through to the enqueue: label at ext.c:1428, which does:

enqueue:
    touch_core_sched(rq, p);
    refill_task_slice_dfl(sch, p);
    dispatch_enqueue(sch, dsq, p, enq_flags);

enq_flags here is whatever the wake path passed in, typically ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK | ... — no SCX_ENQ_PREEMPT and no SCX_ENQ_HEAD.

dispatch_enqueue ends up calling local_dsq_post_enq() at ext.c:987:

static void local_dsq_post_enq(struct scx_dispatch_q *dsq, struct task_struct *p,
                               u64 enq_flags)
{
    struct rq *rq = container_of(dsq, struct rq, scx.local_dsq);
    bool preempt = false;

    if (rq->scx.flags & SCX_RQ_IN_BALANCE)
        return;

    if ((enq_flags & SCX_ENQ_PREEMPT) && p != rq->curr &&
        rq->curr->sched_class == &ext_sched_class) {
        rq->curr->scx.slice = 0;
        preempt = true;
    }

    if (preempt || sched_class_above(&ext_sched_class, rq->curr->sched_class))
        resched_curr(rq);
}

Case analysis on rq->curr at the moment of this call:

• rq->curr is ext-class (common — under SCX ownership of the iso cpu, curr is usually a game thread). (enq_flags & SCX_ENQ_PREEMPT) == 0, so the PREEMPT branch doesn't fire (preempt stays false). sched_class_above(ext_sched_class, ext_sched_class) == false. Neither condition holds. resched_curr is not called. The exiting task lands in the local DSQ silently; no TIF_NEED_RESCHED, no IPI.

• rq->curr is idle. sched_class_above(ext, idle) == true → resched_curr is called. Normal case; no stranding.

• rq->curr is RT/DL (higher class). sched_class_above(ext, rt) == false. Neither condition holds. No resched. (Less relevant on SCX-owned cpus, but the same shape exists.)

So whenever an exiting task is ttwu'd (e.g., wakes from a mutex in exit_mmap / release_task / seccomp filter release, which happens during do_exit cleanup), and the target cpu's curr is ext-class at that exact instant, the exiting task is enqueued without any resched attempt. This is what I believe the enqueue-side trigger is.

Open questions

This is the part I want to flag honestly and where I'd like maintainer input:

The enqueue-side race as described explains how the task lands on a local DSQ without a resched. It does not fully explain why the task stays there for 30+ seconds. Candidate continuations:

1. Curr's slice eventually expires → schedule() → pick_task_scx → balance_one → first_local_task picks the stranded task. At a 5-20ms slice this should bound the dwell at ~20ms, not 36s. So either:

   • Curr's slice doesn't decrement (nohz_full + SCX_RQ_CAN_STOP_TICK?), or
   • balance_one takes the SCX_RQ_BAL_KEEP path at ext.c:2186-2189 (prev still has slice, keep running prev), and prev's slice is somehow indefinite, or
   • Curr goes idle silently without calling schedule() (not possible — idle transition goes through schedule()), which means idle is entered via schedule(), which should have picked the stranded task via first_local_task.

2. The enqueue happens while the curr is in the middle of its run, then curr does enter schedule() later, but pick_task_scx somehow skips the stranded task. I don't have a plausible mechanism for this.

3. There's a subsequent race: curr hits its slice, enters schedule(), balance_one() runs, sets SCX_RQ_IN_BALANCE, and the cpu is mid-pick when the exiting-task enqueue arrives on a different rq lock window. In that case local_dsq_post_enq returns early at ext.c:998-999 (IN_BALANCE short-circuit). The IN_BALANCE path is designed to let the ongoing pick_task_scx pick up the new arrival via first_local_task, but if the race window closes after the first_local_task call inside balance_one — the newly-arrived task would miss this pick. I'm not sure whether the code handles that case.

I don't have proof that any specific continuation explains the 36-second dwell. What I do know is (a) the enqueue-side race as described is real and counter-confirmed, and (b) the empirical fix below unambiguously stops the stalls, across ~60 minutes of the same workload that previously stalled within 90 seconds. I'd appreciate a maintainer confirming or correcting the dwell mechanism.

Workaround

.flags = ... | SCX_OPS_ENQ_EXITING,

void BPF_STRUCT_OPS(my_enqueue, struct task_struct *p, u64 enq_flags)
{
    if (p->flags & PF_EXITING) {
        scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, default_slice,
                           enq_flags | SCX_ENQ_PREEMPT);
        return;
    }
    ...
}

SCX_OPS_ENQ_EXITING routes PF_EXITING tasks through ops.enqueue instead of the goto-local shortcut. SCX_ENQ_PREEMPT forces local_dsq_post_enq to take the first branch, which calls resched_curr regardless of curr's sched_class. Same destination as the kernel default (the task's current cpu's local DSQ — critical, because a PF_EXITING task's cpus_allowed is locked to its cgroup cpuset at fork time and must not be routed cross-domain), but the resched is guaranteed.

Validation:

• Before: cargo build --release -j16 under iso.slice/promoted + a game running on iso.slice/wrapped → 4 ejects in 90 seconds (opt cgu, cc1, lto cgu, i386-linux-gnu-).
• After (same binary, only change is SCX_OPS_ENQ_EXITING | SCX_ENQ_PREEMPT): same workload, 17.7s wallclock build, 110 cpu-seconds, zero ejects, SCX_EV_ENQ_SKIP_EXITING stays at 0 (kernel no longer takes the goto-local path for exiting tasks).

Anti-pattern that bit me during debug: I first tried routing PF_EXITING tasks to a separate "janitor" HK-only DSQ (CPUs 0, 8), thinking it would bypass nohz_full entirely. This made things worse because PF_EXITING tasks born under iso.slice have cpus_allowed = 0xfefe (iso mask). An HK-only DSQ drained only by HK cpus is unreachable for them — task_can_run_on_remote_rq() rejects the pick, and tasks pile up with no eligible consumer. 6+ stranded per iso cpu within 30 seconds. Don't reroute exiting tasks across cpu domains.

Mainline applicability

I verified the relevant functions are byte-identical on torvalds/linux master HEAD:

• local_dsq_post_enq body — identical
• The PF_EXITING goto-local shortcut (SCX_EV_ENQ_SKIP_EXITING / goto local) — identical
• The enqueue: label that does refill_task_slice_dfl + dispatch_enqueue — identical
• SCX_OPS_ENQ_EXITING kdoc comment in kernel/sched/ext_internal.h — same wording, same gap (mentions bpf_task_from_pid / RCU stalls as the motivation, does not mention nohz_full stranding)

So this is not a 6.19.x-only thing.

Relationship to SCX_ENQ_IMMED (for-7.1)

Tejun's SCX_ENQ_IMMED patch queued in the for-7.1 branch provides a general "never linger on local DSQ behind other tasks or on a cpu taken by a higher-priority class" primitive. From the public description:

"Once a task is dispatched with IMMED, it either gets on the CPU immediately and stays on it, or gets reenqueued back to the BPF scheduler. It will never linger on a local DSQ behind other tasks or on a CPU taken by a higher-priority class."

This is a caller opt-in flag on scx_bpf_dsq_insert. From what I can see in public searches I couldn't confirm whether the kernel-internal goto-local PF_EXITING path in for-7.1 was also updated to pass SCX_ENQ_IMMED. If not, then:

• Schedulers on 7.1 that haven't been updated to use SCX_ENQ_IMMED remain vulnerable to the same race.
• Schedulers on pre-7.1 kernels (anyone on current stable) have no way to avoid the goto-local path except SCX_OPS_ENQ_EXITING.

Either way the kdoc change below is valuable independently of SCX_ENQ_IMMED. A maintainer who has the for-7.1 tree open can tell me whether the goto-local path itself now passes IMMED, which would resolve the default case for 7.1+.

Proposed resolutions

Option A (low risk): kdoc patch

 /*
  * An exiting task may schedule after PF_EXITING is set. In such cases,
  * bpf_task_from_pid() may not be able to find the task and if the BPF
  * scheduler depends on pid lookup for dispatching, the task will be
  * lost leading to various issues including RCU grace period stalls.
  *
  * To mask this problem, by default, unhashed tasks are automatically
  * dispatched to the local DSQ on enqueue. If the BPF scheduler doesn't
  * depend on pid lookups and wants to handle these tasks directly, the
  * following flag can be used.
+ *
+ * Schedulers running on nohz_full cpus SHOULD set this flag. The default
+ * goto-local path passes the caller's enq_flags verbatim, which omits
+ * SCX_ENQ_PREEMPT. local_dsq_post_enq() only calls resched_curr() when
+ * PREEMPT is set or when the current sched_class is above ext_sched_class.
+ * Under SCX ownership of an iso cpu, rq->curr is typically ext-class, so
+ * the exiting task can be enqueued silently on an idle nohz_full cpu and
+ * strand on the local DSQ until the 30s runnable-task-stall watchdog
+ * ejects the scheduler. Setting this flag routes exiting tasks through
+ * ops.enqueue(), where the scheduler can insert with SCX_ENQ_PREEMPT
+ * (or, on 7.1+, SCX_ENQ_IMMED) to force resched.
  */
 SCX_OPS_ENQ_EXITING = 1LLU << 2,

Narrow, acknowledges the existing escape hatch, low-risk. I'm happy to send this as a proper patch with Signed-off-by if the wording works.

Option B (broader fix): force PREEMPT in the goto-local path

-    /* see %SCX_OPS_ENQ_EXITING */
-    if (!(sch->ops.flags & SCX_OPS_ENQ_EXITING) &&
-        unlikely(p->flags & PF_EXITING)) {
-        __scx_add_event(sch, SCX_EV_ENQ_SKIP_EXITING, 1);
-        goto local;
-    }
+    /* see %SCX_OPS_ENQ_EXITING */
+    if (!(sch->ops.flags & SCX_OPS_ENQ_EXITING) &&
+        unlikely(p->flags & PF_EXITING)) {
+        __scx_add_event(sch, SCX_EV_ENQ_SKIP_EXITING, 1);
+        /* Force resched on the target cpu — the default
+         * local_dsq_post_enq path skips resched_curr for
+         * ext-vs-ext wakes without PREEMPT, which can strand
+         * exiting tasks on idle nohz_full cpus. */
+        enq_flags |= SCX_ENQ_PREEMPT;
+        goto local;
+    }

This makes behavior safe by default and obsoletes one reason for schedulers to set SCX_OPS_ENQ_EXITING. Risk: changes default preemption semantics for all exiting-task enqueues on all schedulers (preempts a running ext task mid-slice to run a dying task). I'd defer to maintainers on whether that's acceptable.

What I'm asking for

1. Am I reading the enqueue-side race correctly? In particular, is there a reason the default goto-local path deliberately omits PREEMPT that I'm missing?
2. Is the dwell-time gap (open question above) worth chasing? Or is "it eventually strands" sufficient once we've established the missed-resched?
3. Is the kdoc patch (Option A) acceptable as framed? I'll send it as a formal patch if so.
4. On for-7.1: does the goto-local path itself pass SCX_ENQ_IMMED now? That would localize the problem to pre-7.1 stable kernels and tighten the doc patch scope.

Happy to provide more data, instrument further, or run experiments if anything is unclear.

Thanks.

[root-drift-flux]

---

Update

Follow-up after a few hours running the SCX_OPS_ENQ_EXITING | SCX_ENQ_PREEMPT workaround in production. Stalls resumed under a heavier real-world co-load, and the evidence empirically elevates case 3 from the "Open question" section above (the SCX_RQ_IN_BALANCE early-return) from speculation to the leading candidate.

Workload that reproduces

Same binary as the "after" validation above (md5 8d6c94a8fb316e289c4646173bd9bd15, the fix commit), but with the full real desktop load instead of the isolated cargo repro:

• Helldivers 2 on iso.slice/wrapped (wine + vkd3d-proton + nvidia 580)
• cargo build --release -j16 on iso.slice/promoted (rustc + cc1 + lld + lto)
• Firefox/Chromium on HK via janitor.slice (Background Work, TaskQueuePacedS, VizCompositor)
• pipewire + compositor + usual desktop background

9 runnable task stall ejects in ~2 hours (04:47–06:53 local), all 30–39s dwell. Victims span every class of task in the load:

sched_ext: cages: i386-linux-gnu-[287471]   failed to run for 31.970s
sched_ext: cages: ctrl-c[288326]            failed to run for 34.094s
sched_ext: cages: opt cgu.1[289296]         failed to run for 30.789s
sched_ext: cages: opt cgu.07[290025]        failed to run for 30.452s
sched_ext: cages: cc1[290477]               failed to run for 30.371s
sched_ext: cages: lto cgu.0.rcgu.[290544]   failed to run for 30.370s
sched_ext: cages: main[292324]              failed to run for 39.417s
sched_ext: cages: wineserver[301866]        failed to run for 33.320s
sched_ext: cages: vkd3d_queue[303435]       failed to run for 34.813s
sched_ext: cages: TaskQueuePacedS[305778]   failed to run for 30.277s
sched_ext: cages: Background Work[306578]   failed to run for 30.083s

The isolated "cargo build -j16 under iso.slice/promoted, zero ejects" validation I reported in the original post was too light — it did not capture the race. I'm flagging that explicitly because the prior validation is cited in the report above and readers should know the goalposts moved.

Dump evidence — different shape from the original bug

Sample from the Background Work[306578] eject (post-fix, so this is not the same path as the original report):

CPU 4   : nr_run=1 flags=0x1 cpu_rel=0 ops_qseq=95337109
          curr=swapper/4[0] class=idle_sched_class

  R Background Work[306578] -30083ms
      scx_state/flags=3/0x9 dsq_flags=0x0 ops_state/qseq=0/0
      sticky/holding_cpu=-1/-1 dsq_id=0x8000000000000002
      dsq_vtime=0 slice=2000000 weight=100
      cpus=fefe no_mig=0

    _nv049529rm+0x3b/0x80 [nvidia]
    ... (nvidia rm_cleanup_file_private chain) ...
    rm_cleanup_file_private+0x1db/0x200 [nvidia]
    __fput+0xfd/0x280
    task_work_run+0x9d/0xc0
    do_exit+0x315/0xa60
    do_group_exit+0x77/0x90
    get_signal+0x665/0x6f0
    ... (exit_to_user_mode_loop → syscall return) ...

Event counters
--------------
SCX_EV_SELECT_CPU_FALLBACK:           7072
SCX_EV_ENQ_SKIP_EXITING:                 0    <-- key delta
SCX_EV_REFILL_SLICE_DFL:                 8
SCX_EV_BYPASS_DURATION:           10691051

The wineserver[301866] eject (earlier in the same session) is identical in shape: stranded in dsq_id=0x8000000000000002, slice=2000000, cpus=fefe, curr=swapper/6 class=idle_sched_class, stack in do_exit → do_group_exit → __x64_sys_exit_group.

Five facts that distinguish this from the original bug:

1. SCX_EV_ENQ_SKIP_EXITING = 0 (was 1009 in the original report). The kernel's goto-local shortcut is no longer firing — SCX_OPS_ENQ_EXITING is active and PF_EXITING tasks are being routed through ops.enqueue as intended. The original fix is doing what it's supposed to do.

2. slice = 2000000 = 2ms. This matches slice_janitor_ns in my scheduler, which is the exact value my PF_EXITING handler in cages_enqueue passes to scx_bpf_dsq_insert. The kernel's default goto-local path would have set 20ms via refill_task_slice_dfl. So the slice value is a fingerprint that my ops.enqueue handler did run, and it did call scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, slice_janitor_ns, enq_flags | SCX_ENQ_PREEMPT) as intended.

3. ops_state/qseq = 0/0. Consistent with a task that was successfully inserted via ops.enqueue and had its ops state reset to NONE at the end of dispatch_enqueue (the SCX_ENQ_CLEAR_OPSS path is taken from direct_dispatch, but from ordinary ops.enqueue return it's cleared at ext.c:1120). Not evidence of a bypass this time.

4. curr = swapper/N, class = idle_sched_class, for the full ~30s window until the watchdog fired. The cpu was genuinely idle, not running an ext task that failed to yield.

5. Task is stranded alone in its local DSQ (nr_run=1), cpu is idle, watchdog ejects. SCX_ENQ_PREEMPT was on the insert. local_dsq_post_enq for idle-class curr should have fallen through to the sched_class_above(ext, idle) branch and called resched_curr regardless of PREEMPT. Yet the cpu sat in swapper for 30+ seconds.

Why I now think case 3 could be it.

The only code path in local_dsq_post_enq (ext.c:987-1009 as cited in the original post) that reaches this task state from the evidence is the SCX_RQ_IN_BALANCE early return:

if (rq->scx.flags & SCX_RQ_IN_BALANCE)
    return;

My reading (which is what option #3 in the original post was already groping toward) is:

• balance_scx runs pick_task_scx on cpu N (here: CPU 4 / CPU 6) with SCX_RQ_IN_BALANCE set.
• pick_task_scx checks rq->scx.local_dsq — empty at that instant — and calls ops.dispatch, which for my scheduler tries to scx_bpf_dsq_move_to_local from the tier DSQs. Empty. Returns without moving anything to local.
• While SCX_RQ_IN_BALANCE is still set, the exiting task's ttwu arrives on a remote rq (from whatever CPU X ran the wakeup for the nvidia/wine/chromium cleanup). set_task_cpu(p, N) was already done by select_task_rq_scx, so the enqueue ends up on cpu N's rq. ops.enqueue runs my PF_EXITING fast path, calls scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL, slice_janitor_ns, enq_flags | SCX_ENQ_PREEMPT). dispatch_enqueue → local_dsq_post_enq → SCX_RQ_IN_BALANCE → early return. PREEMPT is silently dropped. No resched_curr.
• balance_scx on cpu N completes its pick having already committed to "nothing to run" → returns false / picks idle. pick_next_task picks swapper. Cpu N goes idle.
• Nothing subsequently re-runs pick_task_scx on cpu N — the cpu is in nohz_full, is already in swapper/idle, no tick, no wakeup source. The task sits in local DSQ until the 30s watchdog fires.

Per-hypothesis fit:

• Matches case 3 from the original post exactly — the one I'd flagged as "I'm not sure whether the code handles that case."
• Explains why the fix was not sufficient: my workaround added SCX_ENQ_PREEMPT, but SCX_RQ_IN_BALANCE is checked before the PREEMPT branch in local_dsq_post_enq, so PREEMPT can never save this case.
• Explains the ~30s dwell: no slice decrement, no tick, no alternate scheduling point — only the scx watchdog itself.
• Consistent with curr=swapper at dump time: the cpu reached idle correctly via schedule() once, during balance, before our insert landed. After that, there's no reason for the cpu to leave idle.
• Consistent with the race being load-dependent: the original cargo-only validation workload has lighter exit churn and a smaller window for ops.enqueue to collide with SCX_RQ_IN_BALANCE on a to-be-idle iso cpu. Cargo + Helldivers 2 + chromium + wine + nvidia ramps the exit-churn × idle-target window intersection by a lot.

What I need maintainer input on

The original questions still stand, but priority has shifted:

1. Is the SCX_RQ_IN_BALANCE early-return actually safe under this shape? Specifically: when balance_scx → pick_task_scx has already checked first_local_task / local_dsq and committed to a pick (or to idle), and an ops.enqueue insert lands afterward but still within the IN_BALANCE window, is there a mechanism that ensures that late insert is noticed before the cpu reaches idle? If there is, I'm missing it. If not, this looks like a genuine kernel-side race, not a scheduler-side workaround gap.

2. Does the kdoc patch (Option A above) still close enough of the problem to be worth landing on its own? My view now is: no, not on its own, because even a scheduler that does everything Option A recommends (set SCX_OPS_ENQ_EXITING, use SCX_ENQ_PREEMPT on the direct-local insert) can still strand on this second race on a nohz_full cpu. Option A is still necessary — it closes the first race — but it should be paired with either:

   • (a) clearer docs that SCX_ENQ_PREEMPT is insufficient when SCX_RQ_IN_BALANCE can be set at insert time, and BPF schedulers should additionally scx_bpf_kick_cpu(target, SCX_KICK_PREEMPT) after the insert; or
   • (b) a kernel-side fix to the SCX_RQ_IN_BALANCE early-return that re-checks rq->scx.local_dsq at the tail of balance_scx / before pick_next_task commits to idle; or
   • (c) SCX_ENQ_IMMED being wired through the goto-local and the ops.enqueue caller path on 7.1+ in a way that forces a re-pick.

3. Does SCX_ENQ_IMMED on for-7.1 actually handle the IN_BALANCE race? The public description of IMMED ("never linger on a local DSQ behind other tasks or on a cpu taken by a higher-priority class") doesn't explicitly cover the "cpu went idle during the balance that observed an empty local_dsq" case. A maintainer with the for-7.1 tree open can confirm — this is the question I most want answered before proposing a broader patch.

What I'm trying next on my side

Pending maintainer input, the scheduler-side workaround I'll test is adding scx_bpf_kick_cpu(bpf_get_smp_processor_id(), SCX_KICK_PREEMPT) immediately after the scx_bpf_dsq_insert in the PF_EXITING and kthread fast paths. scx_bpf_kick_cpu goes through irq_work_queue_on and smp_send_reschedule, which is out-of-band with local_dsq_post_enq entirely and does not consult SCX_RQ_IN_BALANCE. If that closes the window empirically I'll report back with a counter-validation run on the same real co-load, not a lighter cargo-only repro.

I'd rather not paper over a kernel race with kick_cpu if the underlying SCX_RQ_IN_BALANCE behavior is actually intended (or already addressed in for-7.1), so I'd welcome a maintainer saying "yes, that's the right workaround, the kernel can't reasonably do better" or "no, the kernel should re-check, here's how" before I commit it.

Apologies for the length — I wanted to be precise about what changed and which parts of the original report I'm now retracting vs. leaving standing. The enqueue-side race in the original post (goto-local with no PREEMPT) is still real and the Option A doc patch is still valuable. It just isn't the whole story for nohz_full.

[galpt]

I've been hitting repeated runnable task stall watchdog

might be related to #1202

[htejun]

Thanks for the detailed dumps.

The posted dumps show curr == swapper and rq->scx.local_dsq.nr > 0
coexisted at the watchdog instant. They don't show the event sequence
that got the CPU into that state, so neither candidate theory in the
report is decided by the data captured so far. Rather than iterate on
theories, we need the event history leading into the violation.

We've hit a similar-shape problem on these paths before (IMMED latency in
sched_ext/for-7.1 this March). The root cause there wasn't in any initial
hypothesis list; it came from per-CPU ring-buffer instrumentation
narrowing a gap round by round. The methodology that worked is written up
as a protocol your agent can execute end-to-end:

https://gist.github.com/htejun/9df3cacec489dcffcb47a931e351f0fe

It covers: probe set, starter code (tree-independent ring + triggers),
discovery checklist per probe site (so the agent finds the right function
in your 6.19.x tree rather than blindly copying symbols), and a strict
OBSERVED/INFERRED reporting format.

Happy to help interpret dumps once you have ring data.

[root-drift-flux]

10-4. I will return with ring data. I got work today so it may be tomorrow or the next day until I can validate and return results. Thank you for the timely response.

[root-drift-flux]

Okay so I've run the protocol you posted and have returned with ringbuf logs. I reproduced the bug cleanly.

The only observation I've made so far: at the insert that finally unblocked a 30 second stalled task, local_dsq_post_enq took the SCX_RQ_IN_BALANCE early return and skipped resched_curr. H6 is alive. I need to save per-task insert context so the watchdog dump can show the origin's state. Next run I'll patch that in.

Here's a full dump of all traces.

scx-3530-round1.zip

Here's the report generated by my local agent using your protocol.

Round 1 — 2026-04-19

Setup

• Kernel: 6.19.11-zen1-dbg3530 (zen-sources 6.19.11 + sched_ext tracing patch per §6A/§6B)
• Debug patch: Part A (per-CPU 128-entry ring, 15 __weak noinline probes, dbg3530_check_invariant_and_dump at __schedule commit-to-idle, dbg3530_check_residence_and_dump at scx pick return, watchdog backstop call from check_rq_for_timeouts); Part B insertion points wired in dispatch_enqueue, local_dsq_post_enq, __resched_curr, smp_send_reschedule, do_idle, __schedule, and the scx pick path. Ring size 128, residence threshold armed at 25 s via debugfs (/sys/kernel/debug/scx_dbg3530_residence_ns). Primary invariant trigger and residence fallback both armed; watchdog backstop calls invariant check per-CPU under rq lock.
• Scheduler: scx_cages (custom 3-tier sched_ext BPF scheduler: iso cores 1-7,9-15 run wrapped + promoted tiers only; HK cores 0,8 run janitor tier; iso cores go idle rather than run janitor).
• Workload: stress-ng --fork 16 --exec 8 --signal 4, a /bin/true fork churn, and a cargo clean; cargo build --release loop, all placed in iso.slice/promoted. Per the reporter's repro class: fork + exit churn drives PF_EXITING rate.
• Runtime: single run 20260419-225532, 63 s total (scheduler attached 22:55:48, ejected 22:56:51 by scx runnable task stall watchdog). One trigger fire; zero primary-invariant fires.

OBSERVED

Facts from dump-20260419-225532-1.log and scx_cages-20260419-225532.log.

Reproduction matches §2 established facts. [scx_cages-20260419-225532.log L533, L584–L623]

• stress-ng-fork[790236] failed to run for 40.447 s; dsq_id = 0x8000000000000002 (SCX_DSQ_LOCAL); cpus_allowed = 0xfefe (iso mask, HK CPUs 0/8 excluded).
• stress-ng-fork[790240] stalled 30.208 s at the same dsq_id / cpus_allowed.
• Both tasks in the exit path: stack ends in exit_mmap → __mmput → do_exit (790236) and anon_vma_interval_tree_remove → free_pgtables → exit_mmap → __mmput (790240). PF_EXITING is set.

Residence fallback fired on pid 790240 at delta = 30.208 s. [dump-20260419-225532-1.log L13]

• DBG3530_TRIGGER why=residence cpu=790240 a=30208186497 ts=1917441453307 (recorded on ksoftirqd/3-44 [003]).
• Interpretation per §6A: a is delta = ktime_get_ns() - p->scx.dbg3530_insert_ts. p->scx.dbg3530_insert_ts was set by a local-DSQ insert 30.208 s earlier and never cleared (clear-on-pick path per §6B never ran for this task in that window).

Final pre-pick insert chain for pid 790240.

ts (ns)	CPU	ev	record	dump line
1917441450127	8	1 INSERT	pid=790240 a1=0x8000000000000003 a2=0x2	L1145
1917441453167	3	3 POST_ENQ	pid=790240 a1=0x2c (curr=ksoftirqd/3) a2=0x1	L524
1917441453197	3	12 PICK	pid=790240 a1=0xc0ee0 (picked_pid=790240) a2=0x1 (pre-pick nr=1)	L525
1917441453307	3	14 TRIGGER	why=residence delta=30.208 s	L13

Decode of POST_ENQ reason bitfield (per §5, a2=0x1):

• bit 0 (0x01) = 1 — SCX_RQ_IN_BALANCE early-return taken
• bit 1 (0x02) = 0 — PREEMPT branch not fired
• bit 2 (0x04) = 0 — sched_class_above(ext, curr->sched_class) fall-through not reached
• bit 3 (0x08) = 0 — resched_curr was not called
• bit 4 (0x10) = 0 — curr was not the idle task (curr = ksoftirqd/3, pid 44)

Primary invariant trigger fired zero times. [manifest-20260419-225532.txt: dump_count=1; no why=invariant string in any dump]

• The primary-trigger site (inside __schedule at commit-to-idle) fires when next == idle_task is picked; the backstop call at check_rq_for_timeouts (ext.c:2746) evaluates the same predicate under rq lock. Across 63 s of runtime neither path won the cmpxchg.
• Ring event spread: 387 DBG_IDLE_ENTER, 246 DBG_SCHEDULE_ENTER, 227 DBG_PICK, 52 DBG_SWITCH_TO_IDLE, 42 DBG_IDLE_EXIT, 185 DBG_INSERT_CTX, 289 DBG_INSERT, 219 DBG_RESCHED, 202 DBG_POST_ENQ. No DBG_TRIGGER (ev=14) entry in the ring (overwritten by subsequent events before dump_all_rings iterated the winning CPU's ring).

IPI-path events absent from ring. No DBG_SMP_SEND_RESCHED (ev=5), DBG_IPI_ENTER (ev=6), or DBG_IPI_EXIT (ev=7) entries in the dump. Consistent with §6B note that x86 sysvec IDTENTRY expansion has no clean inline site; IPI probes were not wired in this build.

Hypothesis ledger

ID	Hypothesis	Disposition	Citation
H1	Enqueue is silent: no resched_curr called for the stranded task's target CPU.	ALIVE (refined)	[dump-20260419-225532-1.log L524] POST_ENQ reason=0x1 — bit 3 (resched_curr called) CLEAR on the last-leg insert.
H2	resched_curr called but elides IPI via wake_idle_without_ipi (TIF_POLLING).	UNTESTED	No DBG_RESCHED event recorded for pid 790240 (ring wraparound).
H3	IPI issued but not delivered.	UNTESTED	No DBG_SMP_SEND_RESCHED recorded for pid 790240.
H4	IPI delivered but CPU fails to exit idle.	UNTESTED	No IPI probes wired (§6B).
H5	Balance / first_local_task doesn't pick the task.	DEAD	[L525] PICK at ts=1917441453197 returned pid=790240 at pre-pick nr=1. When first_local_task actually ran on CPU 3, it picked this task. The 30 s gap is not a pick-side bug.
H6	local_dsq_post_enq takes SCX_RQ_IN_BALANCE early-return, dropping PREEMPT and resched_curr.	ALIVE — direct observation on the final insert	[L524] POST_ENQ reason=0x1: bit 0 (IN_BALANCE taken) set, bit 3 (resched_curr) clear, all other bits clear. See GAP below — this is NOT yet an observation of the stall-ONSET insert.
H7	Long IRQ-disabled section on target CPU blocks TIF_NEED_RESCHED response.	UNTESTED	No DBG_GAP probes wired.
H8	PF_EXITING is incidental; bug affects any task.	DEAD	[scx_cages L604, L621 stacks] Both stranded tasks are in exit_mmap/__mmput; PF_EXITING is set on both. Matches §2 reporter exactly. This round provides no counterexample.

New hypotheses generated this round:

ID	Hypothesis	Decisive observable
H9	The stall-onset insert also took the IN_BALANCE early-return (generalizing H6 from the final pick-path insert to the onset).	POST_ENQ event with reason & 0x1 == 1 recorded on the target rq at the moment p->scx.dbg3530_insert_ts was stamped — i.e. 30 s before the PICK. Requires ring history > 30 s OR per-task context save, neither currently available.
H10	dsq_id = 0x8000000000000003 appearing on the final-leg INSERT is a distinct DSQ from 0x8000000000000002; the bug manifests on inserts into this alternate ID.	Enumerate which code paths produce dsq->id = 0x8000000000000003. In upstream sched_ext enum scx_dsq_id_flags, builtin value 3 is not defined; this id originates either from a tree-specific builtin or from a user-DSQ encoding this reviewer has not identified. Needs source grep of the dbg3530 tree.

Gap identification

• Target CPU: 3 (iso core, where pid 790240 was ultimately picked).
• Start event: p->scx.dbg3530_insert_ts was stamped at t = 1887233266810 ns (computed: trigger_ts − delta = 1917441453307 − 30208186497). No ring event for this moment; ring overwrote it.
• End event: [L524] DBG_POST_ENQ at ts=1917441453167 on CPU 3.
• Duration: 30.208 s.
• Ring coverage on CPU 3 at dump time: approx last 28 ms (128 entries × ~220 µs/event observed in tail window). Ring holds ~0.09 % of the stall window.
• What we can observe in the 28 ms before the final PICK: CPU 3 cycled between ksoftirqd/3 (pid 44) and kworker/3:1 (pid 783) — not idle — with SCHEDULE_ENTER/BALANCE_ENTER/PICK triples every ~700 µs. No idle state visits.

Next probes

Shape = narrow the 30-s gap / discriminate H6 vs H9. Three mutually compatible probes:

1. Per-task context save at INSERT and POST_ENQ. Add three fields to struct sched_ext_entity:

   u32 dbg3530_insert_curr_pid;
   u64 dbg3530_insert_flags;      /* from INSERT_CTX */
   u64 dbg3530_post_enq_reason;   /* from POST_ENQ */

   Set at the INSERT/INSERT_CTX and POST_ENQ probe sites (overwrite each time); clear on PICK alongside the existing dbg3530_insert_ts = 0. Dump the stalled task's saved fields at stall detection (see probe 3). Bounded cost: 3 × u64 per sched_ext task.
2. Enlarge the ring. DBG3530_RING_SIZE = 128 → 4096 extends per-CPU history from ~28 ms to ~900 ms at current event rate. Insufficient for the 30-s stall, but useful for localizing bursts near the watchdog sample. Memory cost: 2 MB system-wide (4096 × 32 B × 16 CPUs).
3. Add dump hook at scx stall detection. Inside check_rq_for_timeouts (ext.c, next to the existing scx_exit(sch, SCX_EXIT_ERROR_STALL, …)), before scx starts bypass-draining, add:

   trace_printk("DBG3530_STALL_ONSET pid=%d stall_ms=%u insert_curr=%u insert_flags=0x%llx post_enq_reason=0x%llx\n",
       p->pid, dur_ms, p->scx.dbg3530_insert_curr_pid,
       p->scx.dbg3530_insert_flags, p->scx.dbg3530_post_enq_reason);
   dbg3530_dump_all_rings("stall-detected", p->pid, dur_ms);

   This fires at 30 s + stall time, captures the stalled task's onset-time context (from probe 1) and the recent ring snapshot (from probe 2) in one record.

Primary + residence triggers remain armed unchanged. Residence threshold stays at 25 s.

Open questions

1. What does dsq_id = 0x8000000000000003 represent in this tree? It appeared on the final-leg INSERT of pid 790240 whereas every other local-DSQ INSERT observed this run used 0x8000000000000002 (canonical SCX_DSQ_LOCAL per upstream enum). Answering this is H10's decisive observable.
2. Does the scx runnable-task-stall watchdog's invariant-check call at ext.c:2746 ever actually satisfy is_idle_task(curr) ∧ local_nr > 0? This round the residence fallback latched dbg3530_done 30 s into the stall and the watchdog ran 10 s later (40-s mark) with done=1; the invariant check early-returned on atomic_read(&dbg3530_done). To test this independently: next round, disable residence, run to watchdog eject, and observe whether the watchdog-backstop invariant check fires. Alternative: make the stall-detection dump (next-probes 3) unconditional (skip cmpxchg) so it captures regardless of what latched first.
3. Why does ksoftirqd/3 (pid 44, the CPU 3 kthread) consume the CPU while pid 790240 sits on its local DSQ? Current scx_cages tier policy may classify the kthread into the janitor tier (HK-only dispatch) and then fail its CPU-pinning constraint, producing the kthread starvation behavior the user's project notes already track. If that interacts with the stall (ksoftirqd's enqueue path also uses SCX_DSQ_LOCAL per §2), the two phenomena could share a root cause in local_dsq_post_enq's SCX_RQ_IN_BALANCE handling. Needs separation: run with a minimal scheduler (e.g. scx_simple) and observe whether the stall still reproduces.

[root-drift-flux]

Wiring more debug probes and ill be back with more soon.