Python and modules
==================

The default python installed in our Debian 11 system is ``python3.9``. As everyone use a different combination of python modules and versions, there is only a *small amount* (``python3 -m pip list``) of python modules directly installed on the system.

.. code-block:: bash

    python3 -m pip list | awk '{print "  * "$0}'


You will have to install, in your ``$HOME`` or team storage (see :doc:`../filesystems/storages`), your own environment(s).

venv : Virtual Environment
--------------------------

.. code-block:: bash

    mylogin@x5570comp2:~/$ mkdir -p ~/tests/py39 && cd ~/tests/py39
    mylogin@x5570comp2:~/tests/py39$ python3.9 -m venv ~/tests/py39

This create a minimal python3.9 virtual environment (or venv). You can now install every additionnal python modules you'll need.

.. note::

    **Only** if you want to use the already pre-installed python modules, add ``--system-site-packages`` when creating your venv. You should'nt.

* Activate (enter) a virtual environment:

.. code-block:: bash

    mylogin@x5570comp2:~/tests/py39$ source ~/tests/py39/bin/activate
    (py39) mylogin@x5570comp2:~/tests/py39$

* Exit (deactivate) a virtual environment:

.. code-block:: bash

    (py39) mylogin@x5570comp2:~/tests/py39$ deactivate 
    mylogin@x5570comp2:~/tests/py39$

* Install a python modules (inside a venv)

For example, ``scipy``:

.. code-block:: bash

    mylogin@x5570comp2:~$ cd tests/py39/
    mylogin@x5570comp2:~/tests/py39$ source ~/tests/py39/bin/activate
    (py39) mylogin@x5570comp2:~/tests/py39$ python3.9 -m pip install scipy
    
    [... some logs ...]
    
    Installing collected packages: numpy, scipy
    Successfully installed numpy-1.22.1 scipy-1.7.3
    (py39) mylogin@x5570comp2:~/tests/py39$

* Using ATLAS (*Automatically Tuned Linear Algebra Software*) library

For some python modules (numpy, for instance), you may need to add this line to your ``~/.bashrc`` (outside of virtual environment, and source it after editing):

.. code-block:: bash

    export LD_LIBRARY_PATH="/usr/lib/atlas-base:$LD_LIBRARY_PATH"

then you can install (or upgrade, with ``-U``) numpy, inside your venv:

.. code-block:: bash

    mylogin@x5570comp2:~$ cd tests/py39/
    mylogin@x5570comp2:~/tests/py39$ source ~/tests/py39/bin/activate
    (py39) mylogin@x5570comp2:~/tests/py39$ python3.9 -m pip install numpy
    Collecting numpy
      Using cached numpy-1.22.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
    Installing collected packages: numpy
    Successfully installed numpy-1.22.1
    (py39) mylogin@x5570comp2:~/tests/py39$

You can now use ``numpy`` in your virtual environment:

.. code-block:: bash

    (py39) mylogin@x5570comp2:~/tests/py39$ python3
    Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
    [GCC 10.2.1 20210110] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import numpy
    >>>

.. note:: With ATLAS library, ``numpy`` will be multithread-compatible, hence you may need to add ``export OMP_NUM_THREADS=1`` to your shell environment (``~/.bashrc``), and/or in your job scripts, in order to avoid parallelization overload.

Example with Jupyter-notebook
-----------------------------

.. NOTE::
    This should be used on a visualization server, see :doc:`../connection/visualization_server`.


Create and install a virtual environment dedicated to jupyter-notebook:

.. code-block:: bash

    mkdir -p ~/tests/py39-jnb && cd ~/tests/py39-jnb
    python3.9 -m venv ~/tests/py39-jnb
    source bin/activate
    python3 -m pip install jupyter
    jupyter-notebook

Run it:

.. code-block:: bash

    cd ~/tests/py39-jnb
    source bin/activate
    jupyter-notebook



Install python in your home
---------------------------

You can build your own python binary in ``$HOME``, or team storage (see :doc:`../filesystems/storages`).

Download, from `python.org <https://www.python.org/downloads/source/>`_, the version you need. Follow the installation instructions, using ``--prefix`` and ``altinstall``.

.. warning:: The python3 from the system will always be available, so take care of ``$PATH`` and ``$PYTHONPATH`` to prioritize your own.

.. note:: Do **NOT** use ``--enable-optimizations`` if you intend to use this build on all PSMN clusters/partitions.

.. code-block:: bash

    cd Python-3.9.4/
    ./configure --prefix=$HOME/py394
    # [...]
    make -j6
    # [...]
    make altinstall

Modify your ``$PATH`` and ``$PYTHONPATH`` environment variables as asked (in ``~/.bashrc``, ``export PATH="$HOME/py394/bin:$PATH"``, and do not forget to source it, or disconnect/reconnect).

To use ATLAS (*Automatically Tuned Linear Algebra Software*) library, add this to your environment (``~/.bashrc``):

.. code-block:: bash

    export LD_LIBRARY_PATH="/usr/lib/atlas-base:$LD_LIBRARY_PATH"

then you can install python with ATLAS support.

You can now use this python binary, preferably in a virtual environment (see above).