Compiling and executing a parallel hybrid MPI&OpenMP C program
Let’s compile and execute the program ProgPAR_Hybrid_OpenMP_MPI.c.
Definition of the compiling environment:
Supposing that we want to use Cascade partition, we have to connect to one of the login nodes for this partition, for example s92node01 (see Login nodes).
To use the software modules for this partition, we have to execute:
module use /applis/PSMN/debian11/Cascade/modules/all
Now we can see available modules:
module avail
Load a module for GNU compilers (GCC):
module load OpenMPI/4.1.1-GCC-10.3.0
export OMP_NUM_THREADS=8
We can check the compiler version:
mpicc --version
GNU Fortran (GCC) 10.3.0
Copyright (C) 2020 Free Software Foundation, Inc.
Compilation
mpicc -fopenmp ProgPAR_Hybrid_MPI_OpenMP.c -o ProgPAR_Hybrid_MPI_OpenMP.c.exe
The binary file (executable) ProgPAR_Hybrid_MPI_OpenMP.c.exe has been generated.
Note
If you prefer to use the Intel compiler, then you have to load the intel/2021a module (instead of OpenMPI/4.1.1-GCC-10.3.0) and launch the compilation with the mpiicc command (instead of mpicc)
Interactive execution (on login node):
mpirun -np 2 -bind-to socket -npersocket 1 ./ProgPAR_Hybrid_MPI_OpenMP.c.exe
The result is displayed on the screen :
Bonjour, je suis le thread 13 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 23 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 31 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 4 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 15 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 66 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 36 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 40 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
.........................................
Batch execution (by submitting to Cascade partition):
We use a submission script.sh (for hybrid MPI and OpenMP program) to submit a job which will run the program on compute nodes.
This submission script configures the environment and then run the binary (with its options, if any) on the compute node.
sbatch script.sh
Submitted batch job 710
The job is waiting that asked ressources become availables and then it will be in Running state:
squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
710 Cascade ProgPAR_ mylogin R 0:00 1 s92node17
As prescribed in submission script, the standard output is redirected to the file ProgPAR_Hybrid.710.s92node17.out and the standard error is redirected to the file ProgPAR_Hybrid.710.s92node17.err.
-rw-r--r-- 1 mylogin cbp 0 25 janv. 09:21 ProgPAR_Hybrid.710.s92node17.err
-rw-r--r-- 1 mylogin cbp 419 25 janv. 09:21 ProgPAR_Hybrid.710.s92node17.out
When the job is finished, we can see the output file as following:
cat ProgPAR_Hybrid.710.s92node17.out
Bonjour, je suis le thread 13 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 23 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 31 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 4 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 15 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 66 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 36 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
Bonjour, je suis le thread 40 parmi 96 threads du process 1 parmi 2 processes sur le noeud s92node01.psmn.ens-lyon.fr
.........................................