Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,11 @@ With this mode, PipEngine will submit pipeline jobs to the scheduler.
-o, --output-dir=<s> Output directory (override standard output directory names)
-b, --pbs-opts=<s+> PBS options
-q, --pbs-queue=<s> PBS queue
-w, --pbs-set-group=<s> PBS -W group_list (default: )
-i, --inspect-pipeline=<s> Show steps
--log=<s> Log script activities, by default stdin. Options are fluentd (default: stdin)
-e, --log-adapter=<s> (stdin|syslog|fluentd) In case of fluentd use http://destination.hostname:port/yourtag
--tag=<s+> Overwrite tags present in samples.yml and pipeline.yml files (e.g. tag1=value1 tag2=value2)

-h, --help Show this message
```

Expand Down Expand Up @@ -651,6 +652,18 @@ within the job script.

If a specific queue needs to be selected for sending the jobs to PBS, the ```--pbs-queue``` (short version **-q**) parameter can be used. This will pass to the ```qsub``` command the ```-q <queue name>``` taken from the command line.

Working in a team can be usefull to share the stdout and stderr of PBS, usually those files (also joined in one single file) have a restricted access just to the owner. To set the group of those files pipengine will automaticcally set the group to the usergroup (primary) by default, otherwise specifying a different group it will be setted accordibly.

```shell
--pbs-set-group desired_group_name
```

will become, in the shell script:

```shell
#PBS -W group_list=desired_group_name
```

Copyright
=========

Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.9.3
0.9.4
1 change: 1 addition & 0 deletions bin/pipengine
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ when "run"
opt :output_dir, "Output directory (override standard output directory names)", :short => "o", :type => :string
opt :pbs_opts, "PBS options", :type => :strings, :short => "b"
opt :pbs_queue, "PBS queue", :type => :string, :short => "q"
opt :pbs_set_group, "PBS -W group_list", :type => :string, :short => "w", :default => ""
opt :inspect_pipeline, "Show steps", :short => "i", :type => :string
opt :log, "Log script activities, by default stdin. Options are fluentd", :type => :string, :default => "stdin"
opt :log_adapter, "(stdin|syslog|fluentd) In case of fluentd use http://destination.hostname:port/yourtag", :type => :string
Expand Down
2 changes: 1 addition & 1 deletion bio-pipengine.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

Gem::Specification.new do |s|
s.name = "bio-pipengine"
s.version = "0.9.3"
s.version = "0.9.4"

s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
s.require_paths = ["lib"]
Expand Down
2 changes: 2 additions & 0 deletions lib/bio-pipengine.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,5 @@
require 'bio/pipengine/step'
require 'bio/pipengine/job'
require 'bio/pipengine'

require 'etc'
28 changes: 25 additions & 3 deletions lib/bio/pipengine/job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,15 @@ def to_script(options)
file.puts "#!/usr/bin/env bash"
file.puts "#PBS -N #{self.name}"
file.puts "#PBS -d #{self.output}"

file.puts "#PBS -q #{options[:pbs_queue]}" if options[:pbs_queue]

if options[:pbs_set_group].empty?
file.puts "#PBS -W group_list=#{Etc.getgrgid(Etc.getpwnam(Etc.getlogin).gid).name}"
else
file.puts "#PBS -W group_list=#{options[:pbs_set_group]}"
end

if options[:pbs_opts]
file.puts "#PBS -l #{options[:pbs_opts].join(",")}"
else
Expand Down Expand Up @@ -167,14 +175,28 @@ def sub_placeholders(cmd,sample,step=nil)
tmp_cmd = cmd.gsub(/<sample>/,sample.name)
if tmp_cmd =~/<sample_path>/
sample_path_glob = (tmp_cmd.scan(/<sample_path>(\S+)/).map {|e| e.first})
if sample_path_glob.empty?
tmp_cmd.gsub!(/<sample_path>/,sample.path.join("\s"))
else
unless sample_path_glob.empty?
sample_path_glob.each do |append|
tmp_cmd.gsub!(/<sample_path>#{Regexp.quote(append)}/,(sample.path.map {|s| s+append}).join("\s"))
end
end
tmp_cmd.gsub!(/<sample_path>/,sample.path.join("\s"))
end

if tmp_cmd =~/<sample_group>/
sample_group_glob = (tmp_cmd.scan(/<sample_group>(\S+)/).map {|e| e.first})
unless sample_group_glob.empty?
sample_group_glob.each do |append|
tmp_cmd.gsub!(/<sample_group>#{Regexp.quote(append)}/,(sample.group+append))
end
end
tmp_cmd.gsub!(/<sample_group>/,sample.group)
end

if tmp_cmd =~/<sample_group\/>/
tmp_cmd = tmp_cmd.gsub(/<sample_group\/>/,self.output+"/")
end

# for resourcers and cpus
tmp_cmd = sub_resources_and_cpu(tmp_cmd,step)

Expand Down