====== Running Jupyter-notebook on GPU nodes ======
**Objective:** we will learn how to:
* Run Jupyter notebook with custom conda environment
* Use custom conda environment within a Notebook.
First we request some resources using SGE:
* 1 GPU
* 20 GB memory
* 2h walltime
$ qlogin -q volta.q -l h_vmem=20G -l h_rt=2:00:00
A new session well be opened in a GPU node.
We load anaconda
$ module load anaconda
We create a new environment with a specific version of python.
$ conda create -y --prefix $WORK/meso1 python=3.7
We check for our conda environment
$ conda env list
meso1
meso2
...
We use ''meso1'' for example
$ conda activate meso1
(meso1)$
By default Jupyter notebook use ''$HOME'' directory to store runtime data.\\
Since ''$HOME'' directory is in compute nodes. We need to tell Jupyter to use another directory.
Modify Jupyter data directory (we use ''$WORK'')
(meso1)$ export JUPYTER_DATA_DIR=$WORK/.jupyter
Set this conda environment (''meso1'') to Jupyter notebook, we need to install ''ipykernel''
(meso1)$ conda install -c anaconda ipykernel
and add ''meso1'' to Jupyter kernel
$ (meso1)$ python -m ipykernel install --user --name=meso1
And finally start Jupyter notebook
jupyter-notebook --no-browser --ip=0.0.0.0
Where command options are:
* ''--no-browser'' do not start the browser automatically
* ''--ip=0.0.0.0'' the server will listen on all interfaces
* The system will pick a random port number to use.
We should have an output like this:
[deleted lines]
...
[I 15:24:14.851 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 15:24:14.851 NotebookApp] http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
[I 15:24:14.851 NotebookApp] or http://127.0.0.1:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
[I 15:24:14.851 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:24:14.860 NotebookApp]
To access the notebook, open this file in a browser:
file:///Work/Users/kmazouzi/.jupyter/runtime/nbserver-42070-open.html
Or copy and paste one of these URLs:
http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
or http://127.0.0.1:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
(meso1)$
There are three important pieces of information, above, to remember:
* The node used by SGE, here **node3-70**
* The port picked by Jupyter-nootbook, here **8888**
* The token of the session, here **token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23**
We can't connect directly to the link above because **node3-70** is located in private network. We have to use ssh port tunneling.
====== SSH tunnel ======
We need to establish a tunnel from ''mesoshared'' to this node (node3-70).
From a new terminal on ''mesoshared'':
$ ssh -g -N -L 9999:localhost:8888 kmazouzi@node3-70
This command will map remote port ''8888'' from ''node3-70'' to local port ''9999'' in ''mesoshared'' (''localhost'').\\
We can use another value than ''9999'' for local port
Finally we will modify the URL above to connect to our Jupyter session.
OLD URL proposed by Jupyter
http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23\\
**Replace ''node3-70:8888'' by ''mesoshared.univ-fcomte.fr:9999''**
The new URL to use, from your browser
http://mesoshared.univ-fcomte.fr:9999/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23
Open a browser on local machine and browse the URL above.\\
Notice that we can open our conda environment (''meso1'') from the Notebook
{{ :labs:jupyter2.png?800 |}}