This guide provides a short tutorial on how to submit computing jobs to the workload scheduler Slurm.

Creating Batch Scripts

To submit jobs, you need to create a batch script containing both your computing job parameters and the executable code. You can create and edit these scripts in two ways:

• Graphically: Connect to the cluster using FastX, where you can utilize graphical editing tools.
• Command-Line: Connect to the cluster on the terminal and use a command line editor such as vim or nano.

The following section shows the batch script for a simple MPI job:

1
2
3
4
5
6
7
8
9
10
11

#!/usr/bin/zsh 

### Job Parameters 
#SBATCH --ntasks=8              # Ask for 8 MPI tasks
#SBATCH --time=00:15:00         # Run time of 15 minutes
#SBATCH --job-name=example_job  # Sets the job name
#SBATCH --output=stdout.txt     # redirects stdout and stderr to stdout.txt
#SBATCH --account=<project-id>  # Replace with your project-id or delete the line

### Program Code
srun hostname

The batch script is comprised of three parts:

• The first line is always a shebang that defines the interpreter. The cluster only supports the zsh shell.
• The next lines specify the job parameters (here lines 4 - 8). Each line starts with #SBATCH followed by the option that you want to set.
• As soon as a non-comment line is encountered, job parameters are no longer read in and the execution of the job is started (here line 10). In this job, each MPI rank prints their hostname.

Further Batch Script Examples

In our Example Collection provide a comprehensive list of example batch scripts for various pruposes such as:

• OpenMP / multi-threading jobs
• MPI / multi-processing jobs (single and multi-node)
• Hybrid MPI + OpenMP
• GPU jobs (single GPU, multi-GPU, multi-node-multi-GPU)
• BeeOND

Submitting the Computing Job

After specifying your batch script, you need to login via SSH to submit your job. Navigate to the directory where you created the batch script and use the following command to submit it to the job to the Slurm queue:

> sbatch batch_script.sh
Submitted batch job 12345678

After submitting your job script, you will obtain a job id from Slurm, which can be used to manage your job. To check the current state of your job use:

> squeue --me
             JOBID PARTITION        NAME      USER ST       TIME  NODES NODELIST(REASON)
          12345678     c23ms example_job  AB123456  R       0:02      1 n23m0023

Most importantly, ST shows the current state of your job. PD stands for pending and R for running. When your job is no longer visible in the queue list, then it has completed.

Note: The directory from which sbatch is called, will be set as working directory.

Best Practices

• Running multiple similar jobs which only differ in terms of the input often is required in the context of parameter study / sensitivity analysis. Array jobs are recommended for this scenario.
• Long computations should be divided into shorter steps, creating a chain job.
• In order to avoid pitfalls and finding errors after waiting hours for a job to start, the following procedure for setting up batch jobs has been judged as useful:
  • Avoid loading modules in your .zshrc or .bashrc.
  • Test your application whether it starts as expected on a login node.
    • Put all commands necessary for the program to start into a script file.
    • Don't forget the module commands and the shebang (#!) at the first line of the script: #!/usr/bin/zsh
    • Use relative paths and ensure that you know the directory in which the job starts.
    • Make the script executable (chmod 775 myscript.sh), run it (./myscript.sh) and make sure that everything works as expected.
    • Ensure that you have configured your program's memory / stack limits when neccessary.
  • Then: Put all job requests in the script file using the #SBATCH directive. Pay attention to the limiting factors such as: memory, number of slots and job run time.
  • Submit your script with a run time of 5 minutes so that it has greater chance of failing early. If everything starts as expected.
  • Submit your final script with the correct time limit.

​See also

• Submit Jobs
• Write a job script
• Jobscript examples
• Introduction to batch scheduling on hpc-wiki
• Slurm tutorial on hpc-wiki
• Official Slurm documentation