Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

This is a short guide to describe how to use jupyter nootebook on Izar through ssh port forwarding.

Installing IPython and Jupyter

The key to successfully installing the tools is using python virtual environments. Please note that you will need to install for each architecture (the $SYS_TYPE variable used below).

  1. Load the compiler and python using modules

    $ module load gcc
    $ module load python
    $ module load intel
    $ module load python
  2. Create a virtual environment

    Shown here for GCC but the same apply to Intel compiler

    $ virtualenv -p python3 --system-site-packages opt/$SYS_TYPE/venv-gcc
    Running virtualenv with interpreter /ssoft/spack/arvine/v1/opt/spack/linux-rhel7-skylake_avx512/gcc-8.4.0/python-3.7.7-drpdlwdbo3lmtkcbckq227ypnzno4ek3/bin/python3
    Already using interpreter /ssoft/spack/arvine/v1/opt/spack/linux-rhel7-skylake_avx512/gcc-8.4.0/python-3.7.7-drpdlwdbo3lmtkcbckq227ypnzno4ek3/bin/python3
    Using base prefix '/ssoft/spack/arvine/v1/opt/spack/linux-rhel7-skylake_avx512/gcc-8.4.0/python-3.7.7-drpdlwdbo3lmtkcbckq227ypnzno4ek3'
    New python executable in /home/user/opt/venv-gcc/bin/python3
    Also creating executable in /home/user/opt/venv-gcc/bin/python
    Installing setuptools, pip, wheel...
  3. Activate virtual environment

    $ source opt/$SYS_TYPE/venv-gcc/bin/activate
    (venv-gcc) [user@izar ~]$
  4. Installing Jupyter and ipyparallel

    (venv-gcc) [user@izar ~]$ pip install jupyter ipyparallel
    Collecting jupyter
  5. Set passwordless access to izar by using ssh key and have the following in your ~/ssh/.config. It is assumed that you are inside the EPFL network.

    Host izar
    User [username]
  6. Run jupyter notebook and ipcluster on izar.
    The script below is a template that allow to start ipcluster on a compute node of izar. Note how here we started tensorflow to provide an example that match real scenario as much as possible. Please use the modules you need for your case.

    #!/bin/bash -l
    #SBATCH --job-name=ipcluster
    #SBATCH --nodes=1
    #SBATCH --exclusive
    #SBATCH --time=01:00:00
    #SBATCH -p debug
    #SBATCH --output jupyter-log-%J.out
    module load gcc/8.4.0-cuda   cuda/10.2.89   cudnn/   mvapich2/2.3.4-cuda py-tensorflow
    source opt/$SYS_TYPE/venv-gcc/bin/activate
    echo "creating profile: ${profile}"
    ipython profile create ${profile}
    echo "Launching controller"
    ipcontroller --ip="*" --profile=${profile} &
    sleep 10
    echo "Launching engines"
    srun ipengine --profile=${profile} --location=$(hostname) 2> /dev/null 1>&2 &
    ipnport=$(shuf -i8000-9999 -n1)
    echo "${hostname}:${ipnport}" > jupyter-notebook-port-and-host
    jupyter-notebook --no-browser --port=${ipnport} --ip=$(hostname -i)

    Once the job is running analyze the output jupyter-log-[SLURM_ID].out. Then look for a line like the following:

        Or copy and paste one of these URLs:
  7. On your local machine do the following

    ssh -L 8504: izar -f -N
  8. Now you should be able to access to izar compute node through the web browser by pasting the following address

  9. Create a jupyter nootebook and add the following

    import ipyparallel as ipp
    c = ipp.Client(profile='job_[SLURM_JOB_ID]')
    view = c[:]

    Replace [SLURM_JOB_ID] with the job number you obtain by running the command: squeue -u $USER

  • No labels