====== Running Jupyter notebook with V100 and SGE ======
We can run Jupyter notebook on the Volta nodes using SGE and ssh tunneling.
This is done in 3 steps:
- Submit SGE Job(login session)
- Start Jupyter notebook in the opened shell
- Establish ''ssh'' tunnel and connect to the notebook
====== Submit SGE Job ======
Request one GPU and 40Gb of memory for 4 hours :
$ qlogin -q volta.q -l h_vmem=40G -l h_rt=4:00:00
====== Start Jupyter Notebook ======
In the opened shell session, we need to start ''jupyter-notbook''. We will use ''conda''
to isolate a new environment, for example.
But first, as usual we need to modify HOME directory to WORK (because HOME is Read Only)
$ export HOME=$WORK
We load ''anaconda'' and ''conda'':
$ module load anaconda
We create or connect to our isolated ''conda'' environment.
Suppose that we have already an environment (use ''conda env list'' to view your environment list):
$ conda activate /Work/Users/kmazouzi/conda/meso
Finally we start ''jupyter-notebook'' inside:
$ (/Work/Users/kmazouzi/conda/meso) kmazouzi@node2-69:~$ jupyter-notebook --no-browser --ip=0.0.0.0
commands options are :
* ''--no-browser'' do not start the browser automatically
* ''--ip=0.0.0.0'' the server will listen on all interfaces.
* The system will pick a random port number to use.
====== Connect to the notebook ======
If all goes well we see this message with important information
[I 11:04:54.299 NotebookApp] JupyterLab extension loaded from /opt/anaconda3/lib/python3.7/site-packages/jupyterlab
[I 11:04:54.299 NotebookApp] JupyterLab application directory is /opt/anaconda3/share/jupyter/lab
[I 11:04:54.300 NotebookApp] Serving notebooks from local directory: /Work/Users/kmazouzi
[I 11:04:54.300 NotebookApp] The Jupyter Notebook is running at:
[I 11:04:54.301 NotebookApp] http://(node2-69 or 127.0.0.1):8888/?token=07f2f688b651f329387c7b1459389edc7d1f37da60c90d65
[I 11:04:54.301 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 11:04:54.307 NotebookApp]
To access the notebook, open this file in a browser:
file:///Work/Users/kmazouzi/.local/share/jupyter/runtime/nbserver-2683-open.html
Or copy and paste one of these URLs:
http://(node2-69 or 127.0.0.1):8888/?token=07f2f688b651f329387c7b1459389edc7d1f37da60c90d65
There are three important pieces of information, above, to remember:
* The node used by SGE, here **node2-69**
* The port picked by Jupyter-nootbook, here **8888**
* The token of the session, here ''token=07f2f688b651f329387c7b1459389edc7d1f37da60c90d65''
We can't connect directly to the link above because node2-69 is located in private network.
We have to use ssh port tunneling.
===== SSH tunnel =====
We need to establish a tunnel from ''mesoshared'' to this volta node.
Here is the command to use from ''mesoshared'' terminal:
$ ssh -g -N -L 9999:localhost:8888 kmazouzi@node2-69
This command will map remote port ''8888'' from ''node2-69'' to ''local'' port ''9999'' in ''mesoshared'' (localhost).
Finally we will modify the URL above to connect to our Jupyter session.
OLD URL proposed by Jupyter:
* http://node2-69:8888/?token=07f2f688b651f329387c7b1459389edc7d1f37da60c90d65
The new URL to use, from your browser:
* http://mesoshared.univ-fcomte.fr:9999/?token=07f2f688b651f329387c7b1459389edc7d1f37da60c90d65