From T2B Wiki
Submitting more than 2500 jobs to the cluster
Use the script big-submission on the UIs. Here's the help:
> big-submission -h This script will execute qsub lines taken from a file, making sure there's at least 1500 jobs present every 900 seconds for the user on the cluster. USAGE: big-submission myqsubcommands.txt with myqsubcommands.txt looking like: qsub -q localgrid /path/to/myscript1.sh qsub -q localgrid /path/to/myscript2.sh # qsub -q localgrid /path/to/myscript2.sh qsub -q localgrid /path/to/myscript2.sh big-submission -h : will print this help. DETAILS: * The script works in the background, so you can deconnect from your session while it continues sending jobs. * Commented lines starting with '#' as well as empty lines will be omitted. * Each time a job is sent, it is deleted from myqsubcommands.txt, ie the file always contains jobs not sent. This way, if you kill big-submission, you know what's left to send. * The lines that failed will be tried once more at the end. * The script is meant to run once per user. If you want to add some commands to send, [stop big-submission], add them to myqsubcommands.txt, [restart big-submission]. Editing the command list live is possible although not recommended, and should be done fast, with for instance `cat file2 >> myqsubcommands.txt` * 3 files are always created for each execution of the script: - myqsubcommands.txt.DATE : a backup containing all qsub lines to submit. - big-submission.DATE.err : a list of all qsub lines that failed. - big-submission.DATE.log : a log file of all actions executed. * To read the logs, one suggests using `less -r logfile`, this way colors will be printed nicely.