==============================================================
Compiling and executing a parallel hybrid MPI&OpenMP C program 
==============================================================

Let's compile and execute the program :doc:`ProgPAR_Hybrid_MPI_OpenMP_c`.

**Definition of the compiling environment**:

Supposing that we want to use Cascade partition, we have to connect to one of the login nodes for this partition, for example *s92node01* (see :doc:`../../clusters_usage/login_nodes`).

To use the software modules for this partition, we have to execute:

.. code-block:: bash

    module use /applis/PSMN/debian11/Cascade/modules/all

Now we can see available modules:

.. code-block:: bash

    module avail

Load a module for GNU compilers (GCC):

.. code-block:: bash

    module load OpenMPI/4.1.1-GCC-10.3.0
    export OMP_NUM_THREADS=8

We can check the compiler version:

.. code-block:: bash

    mpicc --version
    GNU Fortran (GCC) 10.3.0
    Copyright (C) 2020 Free Software Foundation, Inc. 

**Compilation**

.. code-block:: bash

    mpicc -fopenmp ProgPAR_Hybrid_MPI_OpenMP.c -o ProgPAR_Hybrid_MPI_OpenMP.c.exe 

The binary file (executable) *ProgPAR_Hybrid_MPI_OpenMP.c.exe* has been generated.

.. NOTE:: If you prefer to use the Intel compiler, then you have to load the *intel/2021a* module (instead of *OpenMPI/4.1.1-GCC-10.3.0*) and launch the compilation with the *mpiicc* command (instead of *mpicc*)


**Interactive execution (on login node):**

.. code-block:: bash

    mpirun -np 2 -bind-to socket -npersocket 1 ./ProgPAR_Hybrid_MPI_OpenMP.c.exe


The result is displayed on the screen :

.. code-block:: bash

     Bonjour, je suis le thread 13 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 23 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 31 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 4 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 15 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 66 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 36 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 40 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     .........................................

**Batch execution (by submitting to Cascade partition):**

We use a submission :doc:`scriptOpenMP_MPI` to submit a job which will run the program on compute nodes.

This submission script configures the environment and then run the binary (with its options, if any) on the compute node.

.. code-block:: bash

    sbatch script.sh
    Submitted batch job 710

The job is waiting that asked ressources become availables and then it will be in *Running* state: 

.. code-block:: bash

    squeue
        JOBID PARTITION     NAME     USER     ST       TIME     NODES  NODELIST(REASON)
        710   Cascade      ProgPAR_  mylogin  R        0:00      1     s92node17


As prescribed in submission script, the **standard output** is redirected to the file *ProgPAR_Hybrid.710.s92node17.out* and the **standard error** is redirected to the file *ProgPAR_Hybrid.710.s92node17.err*.


.. code-block:: bash

        -rw-r--r-- 1 mylogin cbp     0 25 janv. 09:21 ProgPAR_Hybrid.710.s92node17.err
        -rw-r--r-- 1 mylogin cbp   419 25 janv. 09:21 ProgPAR_Hybrid.710.s92node17.out

When the job is finished, we can see the output file as following: 

.. code-block:: bash

    cat ProgPAR_Hybrid.710.s92node17.out

     Bonjour, je suis le thread 13 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 23 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 31 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 4 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 15 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 66 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 36 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     Bonjour, je suis le thread 40 parmi 96 threads du process  1 parmi 2 processes sur le noeud  s92node01.psmn.ens-lyon.fr
     .........................................