Add OCS/GCS backend support by ernst-bablick · Pull Request #342 · mschubert/clustermq

ernst-bablick · 2026-02-27T11:55:50Z

As discussed in #341, this PR adds native backend support for OCS and GCS.

Summary of changes:

Added OCS and GCS as separate scheduler backends.
Implemented dedicated submission templates for both backends.
Switched the primary job identifier from job name to job ID (job names are not guaranteed to be unique).
Added a finalize handler.
Adjusted cleanup handling (the current SGE implementation appears to reset the flag before finalize() is executed).
Added documentation, including references to OCS/GCS manuals and inline template documentation.
Added CLion's .idea/ to .gitignore.

Testing:

Verified functionality with OCS/GCS 9.1.0beta1.
Added an automated test for clustermq + GCS integration.
Added an R Mandelbrot example to GCS that demonstrates usage.
Opened a Jira issue regarding qsub -terse handling in 9.0.x (expected to be fixed in 9.0.12).
Added howto to GCS distribution with references to you and this project

Many thanks for the helpful feedback and suggestions.

Best regards,
Ernst

mschubert

Thanks for the PR! I added some comments, please have a look and explain/amend as appropriate.

mschubert · 2026-03-08T09:55:30Z

R/zzz.r

+        qname = c("SLURM", "LSF", "SGE", "GCS", "OCS", "LOCAL")
+        exec = Sys.which(c("sbatch", "bsub", "qsub", "qsub", "qsub"))
+        select = c(which(nchar(exec) > 0), 6)[1]


This will not work: We're checking available shell commands to guess the scheduler. If qsub is available, this is SGE (by assumption). We can not distinguish between PBS, Torque, GCS, OCS; so it makes no sense to check these here.

What is your recommendation here?

I think it's best to leave the code as it was before. We could also consider parsing the output of e.g. qsub --help to disambiguate between the different schedulers, but we'd need to make sure this is stable across versions.

mschubert · 2026-03-08T09:56:13Z

vignettes/userguide.Rmd

+* [GCS](#gcs) - *works without setup*
+* [OCS](#ocs) - *works without setup*


This is incorrect, needs clustermq.scheduler to be set (see PBS/Torque)

mschubert · 2026-03-08T09:57:16Z

README.md

+* [GCS](https://mschubert.github.io/clustermq/articles/userguide.html#gcs) - *works without setup*
+* [OCS](https://mschubert.github.io/clustermq/articles/userguide.html#ocs) - *works without setup*


This is incorrect, needs clustermq.scheduler to be set (see PBS/Torque)

Ok. Now I understand. From the scheduler end there is no change required. A default OCS/GCS installtion is sufficient.

vignettes/userguide.Rmd

mschubert · 2026-03-08T10:04:37Z

vignettes/userguide.Rmd

 the missing options.

+
+### GCS


The templates and docs of GCS/OCS are a lot more verbose than for the other schedulers. We should try to be consistent here

Does this mean that I should remove helpfull comments?

Maybe we can make them a bit more concise. Can you make it more in the style of the SGE docs?

mschubert · 2026-03-08T10:13:00Z

R/qsys_sge.r

+OCS = R6::R6Class("OCS",
+    inherit = QSys,
+
+    public = list(


Why is the SGE initializer not reused here? Job names are guaranteed to be unique within clustermq; but if IDs are better, we should use them in SGE as well

I have no access to SGE and do not know if job names were unique back then with the old Sun Microsystems release.

I feel fairly strongly that we should duplicate this code only if necessary. The ideal implementation would be shared between SGE, OCS and GCS

mschubert · 2026-03-08T10:14:05Z

R/qsys_sge.r

+                              log_worker=FALSE, log_file=NULL, verbose=TRUE) {
+            super$initialize(addr=addr, master=master, template=template)
+
+            # fill the template with options and required fields


Please only add comments/whitespace where they add value. If the function is called fill_template, a comment with fill the template is superfluous.

mschubert · 2026-03-08T10:14:30Z

R/qsys_sge.r

+                private$template_error(class(self)[1], status, filled)
+
+            # try to read the job ID from stdout. On error stop
+            private$job_id <- regmatches(private$qsub_stdout, regexpr("^[0-9]+", private$qsub_stdout))


Please be consistent with assignments; we use = by convention in this project

mschubert · 2026-03-08T10:15:23Z

R/qsys_sge.r

+            # first call finalize to send qdel ...
+            private$finalize()


qdel should only be called if there are still running jobs, not otherwise. I believe the cleanup implementation in SGE is correct.

Ok. This did not work for me when i had the same code for OCS/GCS. qdel was not triggered although the worker tasks where processed and there were still pending jobs...

If it doesn't work for the already implemented schedulers that would be a bug rather than an enhancement for OCS/GCS

mschubert · 2026-03-11T12:59:56Z

Added an automated test for clustermq + GCS integration

I didn't see this in the changes, did I miss it?

ernst-bablick · 2026-03-23T12:57:53Z

I made the suggested changes except for reusing the SGE initializer. The initially copied sections are not required for OCS/GCS. They have been removed now.

I still think the SGE cleanup/finalize handling is wrong. In case of successfull execution of the workload there might still be pending array job tasks in the SGE cluster. If they are not deleted they will get unnecessaily scheduled and consume resources. If they where all scheduled before the workload was sucessfully executed then the additional qdel call does no harm because the job is not running anymore.

The tests that I have mentioned in my initial comment are GCS tests that we run in our test environment with a real GCS cluster.

Added OCS/GCS scheduler support

0c16809

mschubert requested changes Mar 8, 2026

View reviewed changes

changes according to comments in pull request #343

220a922

		* [GCS](#gcs) - works without setup
		* [OCS](#ocs) - works without setup

		* [GCS](https://mschubert.github.io/clustermq/articles/userguide.html#gcs) - works without setup
		* [OCS](https://mschubert.github.io/clustermq/articles/userguide.html#ocs) - works without setup

Conversation

ernst-bablick commented Feb 27, 2026

Uh oh!

mschubert left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mschubert Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mschubert Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mschubert Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mschubert commented Mar 11, 2026

Uh oh!

ernst-bablick commented Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mschubert Mar 9, 2026 •

edited

Loading

mschubert Mar 9, 2026 •

edited

Loading

mschubert Mar 8, 2026 •

edited

Loading