Hpc ugent

README.md

@@ -1,12 +1,12 @@
 # pysqa
-Simple queue adapter for python 
+Simple queue adapter for python
 
 [![Codacy Badge](https://api.codacy.com/project/badge/Grade/9db80cb2477f46be870d1446540b4bf3)](https://www.codacy.com/app/pyiron-runner/pysqa?utm_source=github.com&utm_medium=referral&utm_content=pyiron/pysqa&utm_campaign=Badge_Grade_Dashboard)
 [![Build Status](https://travis-ci.org/pyiron/pysqa.svg?branch=master)](https://travis-ci.org/pyiron/pysqa)
 [![Build status](https://ci.appveyor.com/api/projects/status/9lpjai8rvt8324aj/branch/master?svg=true)](https://ci.appveyor.com/project/pyiron-runner/pysqa/branch/master)
 [![Coverage Status](https://coveralls.io/repos/github/pyiron/pysqa/badge.svg?branch=master)](https://coveralls.io/github/pyiron/pysqa?branch=master)
 
-The goal of pysqa is to make submitting to an HPC cluster as easy as starting another subprocess. This is based on the assumption that even though modern queuing systems allow for an wide range of different configuration, most users submit the majority of their jobs with very similar parameters. Therefore pysqa allows the users to store their submission scripts as jinja2 templates for quick access. After the submission pysqa allows the users to track the progress of their jobs, delete them or enable reservations using the built-in functionality of the queuing system. The currently supported queuing systems are: LFS, MOAB, SGE (tested), SLURM (tested), TORQUE. 
+The goal of pysqa is to make submitting to an HPC cluster as easy as starting another subprocess. This is based on the assumption that even though modern queuing systems allow for an wide range of different configuration, most users submit the majority of their jobs with very similar parameters. Therefore pysqa allows the users to store their submission scripts as jinja2 templates for quick access. After the submission pysqa allows the users to track the progress of their jobs, delete them or enable reservations using the built-in functionality of the queuing system. The currently supported queuing systems are: LFS, MOAB, SGE (tested), SLURM (tested), TORQUE.
 
 # Installation
 pysqa can either be installed via pip using:
@@ -18,23 +18,23 @@ Or via anaconda from the conda-forge channel
     conda install -c conda-forge pysqa
 
 
-# Usage 
+# Usage
 pysqa requires the user to configure the type of queuing system as well as the available templates. Example configuration are available at:
 https://github.com/pyiron/pysqa/tree/master/tests/config
 By default pysqa is searching for the queue configuration in `~/.queues/queue.yaml` and the corresponding jinja2 templates in the same folder.
 
 Import pysqa:
 
-    from pysqa import QueueAdapter 
-    sqa = QueueAdapter(directory=‘~/.queues’)  # directory which contains the queue.yaml file 
+    from pysqa import QueueAdapter
+    sqa = QueueAdapter(directory=‘~/.queues’)  # directory which contains the queue.yaml file
 
-List available queues as list of queue names: 
+List available queues as list of queue names:
 
-    sqa.queue_list 
+    sqa.queue_list
 
-List available queues in an pandas dataframe: 
+List available queues in an pandas dataframe:
 
-    sqa.queue_view 
+    sqa.queue_view
 
 Submit a job to the queue - if no queue is specified it is submitted to the default queue defined in the queue configuration:
 
@@ -50,9 +50,9 @@ Get status of a specifc job from the queuing system:
 
 Delete a job from the queuing sytem:
 
-    sqa.delete_job(process_id=1234) 
+    sqa.delete_job(process_id=1234)
 
-Sample configurations for the specific queuing systems are availabe in the tests: 
+Sample configurations for the specific queuing systems are availabe in the tests:
 
 * lsf - https://github.com/pyiron/pysqa/tree/master/tests/config/lsf
 * moab - https://github.com/pyiron/pysqa/tree/master/tests/config/moab
@@ -61,7 +61,7 @@ Sample configurations for the specific queuing systems are availabe in the tests
 * torque - https://github.com/pyiron/pysqa/tree/master/tests/config/torque
 
 # License
-pysqa is released under the BSD license https://github.com/pyiron/pysqa/blob/master/LICENSE . It is a spin-off of the pyiron project https://github.com/pyiron/pyiron therefore if you use pysqa for your publication, please cite: 
+pysqa is released under the BSD license https://github.com/pyiron/pysqa/blob/master/LICENSE . It is a spin-off of the pyiron project https://github.com/pyiron/pyiron therefore if you use pysqa for your publication, please cite:
 
     @article{pyiron-paper,
       title = {pyiron: An integrated development environment for computational materials science},

pysqa/queueadapter.py

@@ -119,7 +119,7 @@ def submit_job(self, queue=None, job_name=None, working_directory=None, cores=No
         """
         if isinstance(command, list):
             command = ''.join(command)
-        if working_directory is None: 
+        if working_directory is None:
             working_directory = '.'
         queue_script = self._job_submission_template(queue=queue, job_name=job_name,
                                                      working_directory=working_directory, cores=cores,
@@ -224,7 +224,7 @@ def get_status_of_jobs(self, process_id_lst):
             else:
                 results_lst.append('finished')
         return results_lst
-        
+
     def check_queue_parameters(self, queue, cores=1, run_time_max=None, memory_max=None, active_queue=None):
         """
 
@@ -387,9 +387,8 @@ def _value_in_range(cls, value, value_min=None, value_max=None):
             value_, value_min_, value_max_ = [cls._memory_spec_string_to_value(v)
                                               if v is not None and isinstance(v, str) else v
                                               for v in (value, value_min, value_max)]
-            # ATTENTION: '60000' is interpreted as '60000M' since default magnitude is 'M'
-            # ATTENTION: int('60000') is interpreted as '60000B' since _memory_spec_string_to_value return the size in
-            # ATTENTION: bytes, as target_magnitude = 'b'
+            # Numerical strings are interpreted as MB (default magnitude), while integers are interpreted as bytes
+            # String with magnitude prefixes are interpreted by _memory_spec_string_to_value
             # We want to compare the the actual (k,m,g)byte value if there is any
             if value_min_ is not None and value_ < value_min_:
                 return value_min
@@ -407,15 +406,15 @@ def _value_in_range(cls, value, value_min=None, value_max=None):
     def _is_memory_string(value):
         """
         Tests a string if it specifies a certain amount of memory e.g.: '20G', '60b'. Also pure integer strings are
-        also valid.
+        also valid. Also allow prefix + byte symbol format e.g.: '20GB'.
 
         Args:
             value (str): the string to test
 
         Returns:
             (bool): A boolean value if the string matches a memory specification
         """
-        memory_spec_pattern = r'[0-9]+[bBkKmMgGtT]?'
+        memory_spec_pattern = r'[0-9]+[bBkKmMgGtT]{0,2}'
         return re.findall(memory_spec_pattern, value)[0] == value
 
     @classmethod
@@ -447,7 +446,7 @@ def _memory_spec_string_to_value(cls, value, default_magnitude='m', target_magni
 
             magnitude = re.findall(magnitude_pattern, value)
             if len(magnitude) > 0:
-                magnitude = magnitude[0].lower()
+                magnitude = magnitude[0].lower()[0] # only take prefix
             else:
                 magnitude = default_magnitude.lower()
             # Convert it to default magnitude = megabytes

pysqa/wrapper/lsf.py

@@ -31,8 +31,8 @@ def get_queue_status_command(self):
 
     @staticmethod
     def get_job_id_from_output(queue_submit_output):
-	      raise NotImplementedError()
-      
+          raise NotImplementedError()
+
     @staticmethod
     def convert_queue_status(queue_status_output):
         raise NotImplementedError()

pysqa/wrapper/moab.py

@@ -31,8 +31,8 @@ def get_queue_status_command(self):
 
     @staticmethod
     def get_job_id_from_output(queue_submit_output):
-	      raise NotImplementedError()
-      
+          raise NotImplementedError()
+
     @staticmethod
     def convert_queue_status(queue_status_output):
         raise NotImplementedError()

pysqa/wrapper/sge.py

@@ -35,7 +35,7 @@ def get_queue_status_command(self):
     @staticmethod
     def get_job_id_from_output(queue_submit_output):
         return int(queue_submit_output)
-      
+
     @staticmethod
     def convert_queue_status(queue_status_output):
         def leaf_to_dict(leaf):