====== Running Jupyter-notebook on GPU nodes ====== **Objective:** we will learn how to: * Run Jupyter notebook with custom conda environment * Use custom conda environment within a Notebook. First we request some resources using SGE: * 1 GPU * 20 GB memory * 2h walltime $ qlogin -q volta.q -l h_vmem=20G -l h_rt=2:00:00 A new session well be opened in a GPU node. We load anaconda $ module load anaconda We create a new environment with a specific version of python. $ conda create -y --prefix $WORK/meso1 python=3.7 We check for our conda environment $ conda env list meso1 meso2 ... We use ''meso1'' for example $ conda activate meso1 (meso1)$ By default Jupyter notebook use ''$HOME'' directory to store runtime data.\\ Since ''$HOME'' directory is in compute nodes. We need to tell Jupyter to use another directory. Modify Jupyter data directory (we use ''$WORK'') (meso1)$ export JUPYTER_DATA_DIR=$WORK/.jupyter Set this conda environment (''meso1'') to Jupyter notebook, we need to install ''ipykernel'' (meso1)$ conda install -c anaconda ipykernel and add ''meso1'' to Jupyter kernel $ (meso1)$ python -m ipykernel install --user --name=meso1 And finally start Jupyter notebook jupyter-notebook --no-browser --ip=0.0.0.0 Where command options are: * ''--no-browser'' do not start the browser automatically * ''--ip=0.0.0.0'' the server will listen on all interfaces * The system will pick a random port number to use. We should have an output like this: [deleted lines] ... [I 15:24:14.851 NotebookApp] Jupyter Notebook 6.1.4 is running at: [I 15:24:14.851 NotebookApp] http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23 [I 15:24:14.851 NotebookApp] or http://127.0.0.1:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23 [I 15:24:14.851 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 15:24:14.860 NotebookApp] To access the notebook, open this file in a browser: file:///Work/Users/kmazouzi/.jupyter/runtime/nbserver-42070-open.html Or copy and paste one of these URLs: http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23 or http://127.0.0.1:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23 (meso1)$ There are three important pieces of information, above, to remember: * The node used by SGE, here **node3-70** * The port picked by Jupyter-nootbook, here **8888** * The token of the session, here **token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23** We can't connect directly to the link above because **node3-70** is located in private network. We have to use ssh port tunneling. ====== SSH tunnel ====== We need to establish a tunnel from ''mesoshared'' to this node (node3-70). From a new terminal on ''mesoshared'': $ ssh -g -N -L 9999:localhost:8888 kmazouzi@node3-70 This command will map remote port ''8888'' from ''node3-70'' to local port ''9999'' in ''mesoshared'' (''localhost'').\\ We can use another value than ''9999'' for local port Finally we will modify the URL above to connect to our Jupyter session. OLD URL proposed by Jupyter http://node3-70:8888/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23\\ **Replace ''node3-70:8888'' by ''mesoshared.univ-fcomte.fr:9999''** The new URL to use, from your browser http://mesoshared.univ-fcomte.fr:9999/?token=a70aba5da2392737bea00f1225d8b62c7a45219ac3fb4f23 Open a browser on local machine and browse the URL above.\\ Notice that we can open our conda environment (''meso1'') from the Notebook {{ :labs:jupyter2.png?800 |}}