Dependent arrays

If you have 100-1000 files that you need to process in multiple steps, dependent job arrays can make the task simple. Going back to example 3 from the Job Arrays section, let’s say we had a group of files that we wanted to process after the job array completed. We can process all of these files in a second job array and create a dependency on the last task of the first job array. This way, the second array will wait to run until the first job array completely finished.

First, we submit the first array:

[abc123@wind ~]$ sbatch job_array.sh
Submitted batch job 1222012

Next, we look at the job number of the array (1222012) and the last task number. In job_array.sh, the ‐‐array=1-5 option tells us that the last task will be labeled 5. Armed with this information, we can submit our second job array:

[abc123@wind ~]$ sbatch --dependency=afterok:1222012_5 job_array2.sh

This can be done with any number of arrays. The important thing to remember is that the last task of the dependent job must be specified in the dependency (The number after the underscore indicates the task number). Above, simply specifying 1222012 as the dependent job would have allowed the second array to start immediately, failing to accomplish our goal.