Report job progress / prevent job from being detected as stale if it's making continuous progress.

[Penners]

I like the timeout mechanism for detecting stale jobs. However I have long running jobs thus requiring a high timeout. I would like to keep timeout low in order to quickly determine stale jobs. I was thinking that for long running jobs they should be able to report their progress and reset the timestamp used for determining if a job is stale.

e.g I could have a very long running job 1hr+ but with a timeout / stale detection of it not making any progress within a 5 minute window.

This all sort of hinges on the assumption of a job being in a stuck in an active state but not actually executing. e.g ungraceful shutdown or bad error handling.

Was thinking maybe there could be a new abstract timestamp on the job that could be used solely for the purpose of determining if a job is stale. With a new job method that'd re-set this timestamp each time it was called. I was thinking something like progressed_at where initially it'd be set via attempted_at / claimed_at Timestamp then continuously set to the current date each time the job reports progress via a convenience method on the job instance class. e.g reportProgress()

Additionally the reportProgress method could update arbitrary metadata on the job itself that may be useful in determining progress or useful if the job is re-attempted.

Backend implementations could then determine if jobs are stale via the following logic. Date.now() > job.progressed_at + job.timeout

Wonder what you think of this proposal? I can sort of see how it could be implemented without adding a new field, by re-using the attemped_at timestamp. But I think this might be less than ideal as you'd loose visibility of when the job execution started.

[GiovaniGuizzo]

Interesting proposal. I will talk to @merencia and see if it's feasible.

I don't know what your job is doing, but if it has iterations, maybe you could break it into smaller jobs?
It's just a thought and maybe not feasible in your context.

Anyway, we will talk about it. Having the feature of "progress" is indeed interesting.