BigSubmission: Difference between revisions
Jump to navigation
Jump to search
(Created page with "== Submitting more than 2500 jobs to the cluster == Use the script '''big-submission''' on the UIs. Here's the help: <pre> > big-submission -h This script will execute qs...") |
|||
| Line 9: | Line 9: | ||
at least 1500 jobs present every 900 seconds for the user on the cluster. | at least 1500 jobs present every 900 seconds for the user on the cluster. | ||
USAGE: | |||
big-submission myqsubcommands.txt | big-submission myqsubcommands.txt | ||
with myqsubcommands.txt looking like: | with myqsubcommands.txt looking like: | ||
qsub -q localgrid /path/to/myscript1.sh | qsub -q localgrid /path/to/myscript1.sh | ||
qsub -q localgrid /path/to/myscript2.sh | |||
# qsub -q localgrid /path/to/myscript2.sh | |||
qsub -q localgrid /path/to/myscript2.sh | qsub -q localgrid /path/to/myscript2.sh | ||
big-submission -h : will print this help. | big-submission -h : will print this help. | ||
DETAILS: | |||
* Commented lines starting with '#' will be omitted. | |||
* Each time a job is sent, it is deleted from myqsubcommands.txt, | |||
ie the file always contains jobs not sent. This way, if you kill | |||
big-submission, you know what's left to send. | |||
* The script is meant to run once per user. If you want to add | |||
some commands to send, [stop big-submission], add them to | |||
myqsubcommands.txt, [restart big-submission]. | |||
Editing the command list live is possible although not recommended, | |||
and should be done fast, with for instance `cat file2 >> myqsubcommands.txt` | |||
</pre> | </pre> | ||
Revision as of 16:02, 25 August 2016
Submitting more than 2500 jobs to the cluster
Use the script big-submission on the UIs. Here's the help:
> big-submission -h
This script will execute qsub lines taken from a file, making sure there's
at least 1500 jobs present every 900 seconds for the user on the cluster.
USAGE:
big-submission myqsubcommands.txt
with myqsubcommands.txt looking like:
qsub -q localgrid /path/to/myscript1.sh
qsub -q localgrid /path/to/myscript2.sh
# qsub -q localgrid /path/to/myscript2.sh
qsub -q localgrid /path/to/myscript2.sh
big-submission -h : will print this help.
DETAILS:
* Commented lines starting with '#' will be omitted.
* Each time a job is sent, it is deleted from myqsubcommands.txt,
ie the file always contains jobs not sent. This way, if you kill
big-submission, you know what's left to send.
* The script is meant to run once per user. If you want to add
some commands to send, [stop big-submission], add them to
myqsubcommands.txt, [restart big-submission].
Editing the command list live is possible although not recommended,
and should be done fast, with for instance `cat file2 >> myqsubcommands.txt`