Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
d64d804
include license file
jan-janssen Feb 17, 2019
3bc2316
Add coveralls as dependency
jan-janssen Feb 17, 2019
5b341b8
Add License
jan-janssen Feb 17, 2019
c845bb9
Update setup.cfg
jan-janssen Feb 17, 2019
3e4c451
Update setup.py
jan-janssen Feb 17, 2019
583238e
Remove versioneer files from tests
jan-janssen Feb 18, 2019
48a3247
Add more tests and queue autocompletion
jan-janssen Feb 18, 2019
64aaff0
Fix coverage command
jan-janssen Feb 18, 2019
d6a82be
Import from init directly
jan-janssen Feb 18, 2019
df2ef95
Add DocStrings
jan-janssen Feb 18, 2019
83f25e1
Add more tests
jan-janssen Feb 18, 2019
b26deed
Start searching in the home directory.
jan-janssen Feb 19, 2019
6e8971c
Update README.md
jan-janssen Feb 19, 2019
3615146
Add long description
jan-janssen Feb 19, 2019
5dca2ad
if output is None return None
jan-janssen Feb 19, 2019
2f81a50
Update paper
jan-janssen Feb 23, 2019
dd39a85
update pyiron paper
jan-janssen Mar 27, 2019
39bcd6e
Create FUNDING.yml
jan-janssen Jun 5, 2019
c121363
Update FUNDING.yml
jan-janssen Jun 5, 2019
0228eec
Update FUNDING.yml
jan-janssen Jun 5, 2019
d59c4a8
Delete FUNDING.yml
jan-janssen Jun 5, 2019
69a813d
Update queueadapter.py
jan-janssen Jun 6, 2019
e73867c
Update slurm.py
jan-janssen Jul 11, 2019
7207626
Update test_queueadapter.py
jan-janssen Jul 14, 2019
359cb3a
Update slurm.py
jan-janssen Jul 14, 2019
94bd5ab
Update README.md
jan-janssen Jul 15, 2019
7234670
Update .travis.yml
jan-janssen Jul 15, 2019
057010a
Update README.md
jan-janssen Jul 15, 2019
0ce507b
Add convert_queue_status from Dominik
jan-janssen Jul 15, 2019
9088f3b
Add pandas import
jan-janssen Jul 15, 2019
25fc8b9
Add get_job_id_from_output
jan-janssen Jul 15, 2019
d3ff81b
Add get_job_id_from_output
jan-janssen Jul 15, 2019
250e8b7
Update sge.py
jan-janssen Jul 15, 2019
1368690
Update lsf.py
jan-janssen Jul 15, 2019
4a35b3f
Update moab.py
jan-janssen Jul 15, 2019
c1fafd5
Update torque.py
jan-janssen Jul 15, 2019
262aa64
Update queueadapter.py
jan-janssen Jul 15, 2019
d50e397
Update sge.py
jan-janssen Jul 15, 2019
b006a25
Update README.md
jan-janssen Jul 15, 2019
c489a15
Update setup.py
jan-janssen Jul 15, 2019
30c6b0a
Update README.md
jan-janssen Jul 15, 2019
38227da
Fix submit_job
jan-janssen Jul 15, 2019
18a8197
Update version
jan-janssen Jul 17, 2019
88ef924
Update slurm.py
jan-janssen Jul 17, 2019
f9dc871
Update queueadapter.py
jan-janssen Jul 19, 2019
00ba55c
Update README.md
jan-janssen Jul 21, 2019
c197c2e
Memory string comparison for pysqa.queueadapter._value_in_range
Aug 6, 2019
416fbbe
Update setup.py
jan-janssen Aug 6, 2019
779c600
Update appveyor.yml
jan-janssen Aug 6, 2019
8a71903
Update appveyor.yml
jan-janssen Aug 6, 2019
b05a1c4
Update queueadapter.py
jan-janssen Aug 6, 2019
3023f75
Merge pull request #4 from pyiron/check_memory
jan-janssen Aug 6, 2019
2c80824
azure pipelines
jan-janssen Aug 10, 2019
766ea45
use spaces
jan-janssen Aug 10, 2019
d5187c8
fix windows indentation
jan-janssen Aug 10, 2019
cf5d32f
Merge pull request #5 from pyiron/azure
jan-janssen Aug 11, 2019
5d00360
Create .coveragerc
jan-janssen Aug 14, 2019
e68e395
Update .coveragerc
jan-janssen Aug 14, 2019
a05e5e7
Update azure-pipelines.yml
jan-janssen Aug 14, 2019
1c2b50d
Update .travis.yml
jan-janssen Aug 14, 2019
53d2875
Merge pull request #6 from pyiron/coverage
jan-janssen Aug 14, 2019
b0d2124
coverage combine
jan-janssen Aug 21, 2019
2828388
Merge pull request #12 from pyiron/coverage
jan-janssen Aug 21, 2019
ac5bd7e
black code style
jan-janssen Aug 21, 2019
f824995
Merge pull request #13 from pyiron/black
jan-janssen Aug 21, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# .coveragerc to control coverage.py
[run]
source = pysqa
omit = pysqa/_version.py
concurrency = multiprocessing
7 changes: 4 additions & 3 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,24 +28,25 @@ matrix:
- conda info -a
- conda config --set always_yes yes --set changeps1 no
- conda update -q conda
- conda install -y -c conda-forge python=${PYTHONVER} coverage defusedxml pandas yaml jinja2
- conda install -y -c conda-forge python=${PYTHONVER} coverage coveralls defusedxml pandas yaml jinja2
- pip install --pre .
script:
- coverage run -m unittest discover tests
after_success:
- coverage combine
- coveralls
- coverage xml
- pip install codacy-coverage
- python-codacy-coverage -r coverage.xml
deploy:
- provider: pypi
user: jan-janssen
user: pyiron
password: ${PYPI_UPLOAD_TOKEN}
allow_failure: true
on:
branch: master
- provider: pypi
user: jan-janssen
user: pyiron
password: ${PYPI_UPLOAD_TOKEN}
allow_failure: true
on:
Expand Down
80 changes: 75 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,77 @@
# pysqa
Simple queue adapter for Python
Simple queue adapter for python

[![Codacy Badge](https://api.codacy.com/project/badge/Grade/de30f320fd9f44ac9cbcce28eb2c54d1)](https://app.codacy.com/app/jan-janssen/pysqa?utm_source=github.com&utm_medium=referral&utm_content=pysqa/pysqa&utm_campaign=Badge_Grade_Dashboard)
[![Build Status](https://travis-ci.org/pysqa/pysqa.svg?branch=master)](https://travis-ci.org/pysqa/pysqa)
[![Build status](https://ci.appveyor.com/api/projects/status/90cb1w7a57fql52q/branch/master?svg=true)](https://ci.appveyor.com/project/jan-janssen/pysqa/branch/master)
[![Coverage Status](https://coveralls.io/repos/github/pysqa/pysqa/badge.svg?branch=master)](https://coveralls.io/github/pysqa/pysqa?branch=master)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/9db80cb2477f46be870d1446540b4bf3)](https://www.codacy.com/app/pyiron-runner/pysqa?utm_source=github.com&utm_medium=referral&utm_content=pyiron/pysqa&utm_campaign=Badge_Grade_Dashboard)
[![Build Status](https://travis-ci.org/pyiron/pysqa.svg?branch=master)](https://travis-ci.org/pyiron/pysqa)
[![Build status](https://ci.appveyor.com/api/projects/status/9lpjai8rvt8324aj/branch/master?svg=true)](https://ci.appveyor.com/project/pyiron-runner/pysqa/branch/master)
[![Coverage Status](https://coveralls.io/repos/github/pyiron/pysqa/badge.svg?branch=master)](https://coveralls.io/github/pyiron/pysqa?branch=master)

The goal of pysqa is to make submitting to an HPC cluster as easy as starting another subprocess. This is based on the assumption that even though modern queuing systems allow for an wide range of different configuration, most users submit the majority of their jobs with very similar parameters. Therefore pysqa allows the users to store their submission scripts as jinja2 templates for quick access. After the submission pysqa allows the users to track the progress of their jobs, delete them or enable reservations using the built-in functionality of the queuing system. The currently supported queuing systems are: LFS, MOAB, SGE (tested), SLURM (tested), TORQUE.

# Installation
pysqa can either be installed via pip using:

pip install pysqa

Or via anaconda from the conda-forge channel

conda install -c conda-forge pysqa


# Usage
pysqa requires the user to configure the type of queuing system as well as the available templates. Example configuration are available at:
https://github.com/pyiron/pysqa/tree/master/tests/config
By default pysqa is searching for the queue configuration in `~/.queues/queue.yaml` and the corresponding jinja2 templates in the same folder.

Import pysqa:

from pysqa import QueueAdapter
sqa = QueueAdapter(directory=‘~/.queues’) # directory which contains the queue.yaml file

List available queues as list of queue names:

sqa.queue_list

List available queues in an pandas dataframe:

sqa.queue_view

Submit a job to the queue - if no queue is specified it is submitted to the default queue defined in the queue configuration:

sqa.submit_job(command=‘python test.py’)

Get status of all jobs currently handled by the queuing system:

sqa.get_queue_status()

Get status of a specifc job from the queuing system:

sqa.get_status_of_job(process_id=1234)

Delete a job from the queuing sytem:

sqa.delete_job(process_id=1234)

Sample configurations for the specific queuing systems are availabe in the tests:

* lsf - https://github.com/pyiron/pysqa/tree/master/tests/config/lsf
* moab - https://github.com/pyiron/pysqa/tree/master/tests/config/moab
* SGE - https://github.com/pyiron/pysqa/tree/master/tests/config/sge
* slurm - https://github.com/pyiron/pysqa/tree/master/tests/config/slurm
* torque - https://github.com/pyiron/pysqa/tree/master/tests/config/torque

# License
pysqa is released under the BSD license https://github.com/pyiron/pysqa/blob/master/LICENSE . It is a spin-off of the pyiron project https://github.com/pyiron/pyiron therefore if you use pysqa for your publication, please cite:

@article{pyiron-paper,
title = {pyiron: An integrated development environment for computational materials science},
journal = {Computational Materials Science},
volume = {163},
pages = {24 - 36},
year = {2019},
issn = {0927-0256},
doi = {https://doi.org/10.1016/j.commatsci.2018.07.043},
url = {http://www.sciencedirect.com/science/article/pii/S0927025618304786},
author = {Jan Janssen and Sudarsan Surendralal and Yury Lysogorskiy and Mira Todorova and Tilmann Hickel and Ralf Drautz and Jörg Neugebauer},
keywords = {Modelling workflow, Integrated development environment, Complex simulation protocols},
}
2 changes: 1 addition & 1 deletion appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ install:
- '"%CONDA%\Scripts\activate.bat"'
- "conda config --set always_yes yes --set changeps1 no"
- "conda update -q conda"
- "conda install -c conda-forge coverage defusedxml pandas yaml jinja2"
- 'conda install -c conda-forge coverage defusedxml "pandas>=0.23" yaml jinja2'
- "conda info -a"
- "pip install --pre ."

Expand Down
100 changes: 100 additions & 0 deletions azure-pipelines.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
variables:
linux: 'ubuntu-16.04'
mac: 'macos-10.13'
windows: 'vs2017-win2016'

pr:
branches:
include:
- '*'

jobs:
- job: 'Linux'
pool:
vmImage: $(linux)
strategy:
matrix:
Python27:
python.version: '2.7'
Python37:
python.version: '3.7'
maxParallel: 4

steps:
- bash: echo "##vso[task.prependpath]/usr/share/miniconda/bin"
displayName: Add conda to PATH

- bash: conda create -n pysqa -q --yes -c conda-forge python=$(python.version) coverage defusedxml pandas yaml jinja2
displayName: conda install

- bash: |
source activate pysqa
pip install --pre .
displayName: pysqa install

- bash: |
source activate pysqa
coverage run -m unittest discover tests
displayName: run tests

- job: 'OSX'
pool:
vmImage: $(mac)
strategy:
matrix:
Python27:
python.version: '2.7'
Python37:
python.version: '3.7'
maxParallel: 1

steps:
- bash: echo "##vso[task.prependpath]$CONDA/bin"
displayName: Add conda to PATH

- bash: sudo chown -R $USER $CONDA
displayName: Take ownership of conda installation

- bash: conda create -n pysqa -q --yes -c conda-forge python=$(python.version) coverage defusedxml pandas yaml jinja2
displayName: conda install

- bash: |
source activate pysqa
pip install --pre .
displayName: pysqa install

- bash: |
source activate pysqa
coverage run -m unittest discover tests
displayName: run tests

- job: 'Windows'
pool:
vmImage: $(windows)
strategy:
matrix:
Python27:
python.version: '2.7'
Python37:
python.version: '3.7'
maxParallel: 1

steps:
- powershell: Write-Host "##vso[task.prependpath]$env:CONDA\Scripts"
displayName: Add conda to PATH

- bash: echo "##vso[task.prependpath]$CONDA/bin"
displayName: Add conda to PATH

- script: conda create -n pysqa -q --yes -c conda-forge python=$(python.version) coverage defusedxml pandas yaml jinja2
displayName: conda install

- script: |
call activate pysqa
pip install --pre .
displayName: pysqa install

- script: |
call activate pysqa
coverage run -m unittest discover tests
displayName: run tests
5 changes: 3 additions & 2 deletions pysqa/__init__.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
__version__ = '0.0.1'
__version__ = "0.0.1"
__all__ = []

from pysqa.queueadapter import QueueAdapter

from ._version import get_versions
__version__ = get_versions()['version']

__version__ = get_versions()["version"]
del get_versions
Loading