Submitting a job

For those familiar with GridEngine, Slurm documentation provide a Rosetta Stone for schedulers, to ease the transition.

Slurm commands

Slurm allows requesting resources and submitting jobs in a variety of ways. The main Slurm commands to submit jobs are:

  • srun
    • Request resources and runs a command on the allocated compute node(s)

    • Blocking: will not return until the job ends

  • sbatch
    • Request resources and runs a script on the allocated compute node(s)

    • Asynchronous: will return as soon as the job is submitted

Job script

To run a job on the system you need to create a job script (or submission script, or batch script). A job script is a regular shell script (bash) with some directives specifying the number of CPUs, memory, etc., that will be interpreted by the batch system upon submission.

  • very simple

#!/bin/bash
#
#SBATCH --job-name=test

hostname -s
sleep 60s

Writing a submission script can be tricky, see more in Batch scripts.

First job

submit your job script with:

$ sbatch myfirstjob.sh
Submitted batch job 623

Slurm will return with a $JOBID if the job is accepted, else an error message. Without any options about output, it will be defaulted to slurm-$JOBID.out (here, slurm-623.out), in the submission directory.

Once submitted, the job enters the queue in the PENDING (PD) state. When resources become available and the job has sufficient priority, an allocation is created for it and it moves to the RUNNING (R) state. If the job completes correctly, it goes to the COMPLETED state, otherwise, its state is set to FAILED.

Tip

You can submit jobs from any login node to any partition. Login nodes are only segregated for build (CPU µarch) and scratch access.

Monitor your jobs

You can monitor your job using either his name (#SBATCH --job-name) or his $JOBID with Slurm’s squeue 1 command:

$ squeue -j 623
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  623        E5     test ltaulell  R       0:04      1 c82gluster2

By default, squeue show every pending and running jobs. You can filter in your own jobs, using -u $USER or --me option:

$ squeue --me
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  623        E5     test ltaulell  R       0:04      1 c82gluster2

If needed, you can modify the output of squeue 1. Here’s an example (add CPUs to default output):

$ squeue -o "%.7i %.9P %.8j %.8u %.2t %.10M %.6D %.4C %N"
  JOBID PARTITION     NAME     USER ST       TIME  NODES CPUS NODELIST
  38956      Lake     test ltaulell  R       0:41      1    1 c6420node172

You can obtain more detailed informations about a job using Slurm’s scontrol 2 command. This can be very usefull for troubleshooting.

$ scontrol show jobid $JOB_ID

$ scontrol show jobid 38956
JobId=38956 JobName=test
UserId=ltaulell(*****) GroupId=psmn(*****) MCS_label=N/A
Priority=8628 Nice=0 Account=staff QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:08 TimeLimit=8-00:00:00 TimeMin=N/A
SubmitTime=2022-07-08T12:00:20 EligibleTime=2022-07-08T12:00:20
AccrueTime=2022-07-08T12:00:20
StartTime=2022-07-08T12:00:22 EndTime=2022-07-16T12:00:22 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2022-07-08T12:00:22
Partition=Lake AllocNode:Sid=x5570comp2:446203
ReqNodeList=(null) ExcNodeList=(null)
NodeList=c6420node172
BatchHost=c6420node172
NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=1,mem=385582M,node=1,billing=1
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=385582M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/home/ltaulell/tests/env.sh
WorkDir=/home/ltaulell/tests
StdErr=/home/ltaulell/tests/slurm-38956.out
StdIn=/dev/null
StdOut=/home/ltaulell/tests/slurm-38956.out
Power=
NtasksPerTRES:0

Tip

Wait times in queue

As a quick rule of thumb, it’s important to keep in mind that the more resources your job requests (CPUs, memory, nodes, and time), the longer it may have to wait in queue before it could start.

In other words: accurately requesting resources to match your job’s needs will minimize your wait times.

1(1,2)

You can get the complete list of parameters by referring to the squeue manual page (man squeue).

2

You can get the complete list of parameters by referring to the scontrol manual page (man scontrol).