====== Dell servers with VOLTA V100 GPUs ======
====== How to connect ======
We use [[:SGE]] to access to the node.
Only authorized users are allowed to use this machine
===== Interactive mode =====
**Objective :** open a shell or run directly commands on the server.
* **login node** : mesologin1.univ-fcomte.fr or mesoshared.univ-fcomte.fr
* **queue** : volta.q
* **request GPU** : ''-l gpu=N'' where '' 0< N < =2'' ; default value ''gpu=1''
* Default SGE setting for ''h_vmem'' value is ''4Go'' per core. Use ''-l h_vmem'' to request more memory. We suggest using more than 20G.For example: ''-l h_vmem=20G'' for 20 Go memory. **Warning** Job will be killed if memory exceed
* ''HOME'' folder is mounted in read only mode.
* Use ''WORK'' folder (cd ''$WORK'') as your working space.
==== Examples ====
==== Shell session ====
* Request shell session with 1 GPU and 20Go memory for 2 hours:
[user@mesologin1 ~]$ qlogin -q volta.q -l h_vmem=20G -l h_rt=2:00:00
**Once connected, change directory to WORK**
[user@node2-69 ~]$ cd WORK
Check for allocated #GPU:
[user@node2-69 ~]$ nvidia-smi
Thu Feb 27 15:29:50 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... On | 00000000:3B:00.0 Off | 0 |
| N/A 26C P0 25W / 250W | 0MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
* Request shell session with 2 GPU and 64Go memory:
[user@mesologin1~]$ qlogin -q volta.q -l gpu=2 -l h_vmem=64G
All software are installed in the default path, you can invoke directly ''nvcc'' for example:
[user@node2-69 ~]$ nvcc program.cu -o program
/**
==== Interactive command ====
Run interactive command in server:
[user@mesologin1 ~]$ qrsh -q volta.q nvidia-smi
Run interactive program in server:
[user@mesologin1 ~]$ qrsh -q volta.q -lh_vmem=30G python tensorflow.py
*/
===== Batch Mode =====
**
Objective :** run programs in batch using SGE.\\
**Example Script to adapt for your needs** \\
#!/bin/bash -l
#$ -q volta.q
#$ -l gpu=1 ## adapter selon besoin
#$ -l h_vmem=30G ## job will cancel if memory exceed
#$ -N Test_GPU
#################"
## Please adapt this script for your need
#################
## 1-vanila python
python GPU_Program.py
## 2-anaconda
export PATH="/opt/anaconda3/bin:$PATH"
python GPU_Program.py
## 2-bis anaconda with env
conda activate $WORK/conda/meso && python GPU_Program.py
##3-singularity
singularity exec --nv tensorflow-gpu.simg python mytensorflow.py
Submit SGE job:
[user@mesologin1 ~]$ qsub gpu.sge
====== Installing and Running software ======
This host has its own software installed. It doesn't share any software with the rest of cluster (''module'' system). Only Deep-leaning Software based on python are installed.
===== Using default installed software =====
Please use anaconda for installing packages.
==== Vanilla Python ====
Several programs are installed by default ''python2.*'' and ''python3.*''
Use ''pip list'' or ''pip3 list'' command to view installed packages:
$ pip list
gpustat 0.5.0
grpcio 1.18.0
h5py 2.9.0
Keras 2.2.4
Keras-Applications 1.0.6
Keras-Preprocessing 1.0.5
Markdown 3.0.1
numpy 1.16.0
nvidia-ml-py 7.352.0
pbr 5.1.1
protobuf 3.6.1
scipy 1.2.0
setuptools 20.7.0
six 1.10.0
tensorboard 1.12.2
tensorflow-gpu 1.12.0
Run python program:
[user@node2-69~]$ python my_python_program.py
You can install new package **without** root access. Packages needs to be installed in ''$WORK'' because ''$HOME'' is read only.
The Idea is to set WORK as HOME directory (''export HOME=$WORK'') and using ''--user'' option of ''pip or pip3''
For example let's install ''panda'':
$ export HOME=$WORK
$ pip install --user panda
//Voilà ! ; That's all!//
/**
$ pip install --install-option="--prefix=$WORK/myPython" panda
We need to append ''PATH'' and ''PYTHONPATH'' (in ''.bashrc'' file for example) to take effect:
$ export PATH=$WORK/myPython:$PATH
$ export PYTHONPATH=$WORK/myPython:$PYTHONPATH
*/
==== Anaconda [Recommended] ====
For better performances Anaconda python V3 is installed in ''/opt/''. Several optimized scientific packages are available.
==== Using anaconda with conda ====
Conda is powerfull package system management and environment.
To use conda with anaconda:
/*
export PATH="/opt/anaconda3/bin:$PATH"
source /opt/anaconda3/etc/profile.d/conda.sh
*/
**EDIT (10/03/2020)**: we added ''module'' command only for local software in these nodes.
To load anaconda simply use:
$ module load anaconda
Check for available packages:
$ conda list
mkl 2019.1 144
mkl-service 1.1.2 py37he904b0f_5
mkl_fft 1.0.6 py37hd81dba3_0
mkl_random 1.0.2 py37hd81dba3_0
more-itertools 4.3.0 py37_0
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.1 hdf1c602_3
mpmath 1.1.0 py37_0
msgpack-python 0.5.6 py37h6bb024c_1
multipledispatch 0.6.0 py37_0
navigator-updater 0.2.1 py37_0
nbconvert 5.4.0 py37_1
nbformat 4.4.0 py37_0
ncurses 6.1 he6710b0_1
networkx 2.2 py37_1
nltk 3.4 py37_1
nose 1.3.7 py37_2
notebook 5.7.4 py37_0
You may need to install additional package without **having** root access. We instead recommend building a ''virtual environment'' locally and install packages there.\\
But before we need to tell conda to use WORK to store pkgs :
''export HOME=$WORK''
- Create a conda environment $ conda create -y --prefix $WORK/conda-env replace ''conda-env'' by a valid name.
- Activate the environment you just created $ source activate $WORK/conda-env
- Download and install the conda package in the environment $ conda install package replace package by the package you wan to install, for example ''panda''
- Run your analysis from **within** the environment
- Deactivating environment, once you are done with your analysis- you can deactivate the conda environment by running the command $ conda deactivate
**Example: installing Tensorflow GPU on local environment**
We will use an environment locally (in our WORK) and we install all needed package on it.
$ conda create -y --prefix $WORK/conda/meso
$ conda activate $WORK/conda/meso
$ (tensorflowGPU) user@node2-69:~$ conda install tensorflow-gpu
$ (tensorflowGPU) user@node2-69: python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf # this may take too long
>>> tf.test.gpu_device_name()
...
2020-02-27 15:18:44.636689: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56308c48ef00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-02-27 15:18:44.636746: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
'/device:GPU:0'
* To list your conda environment use conda env list
* To remove an environment conda remove --name myenv --all
/**
Use conda info --envs to view all your environment
==== Using juypter notebook ====
For demonstration studying purpose, on can spawn jupyter notebook.
With ''anaconda'' or ''python*'':
$ jupyter-notebook --no-browser --ip=0.0.0.0
[I 13:27:12.331 NotebookApp] JupyterLab extension loaded from /opt/anaconda3/lib/python3.7/site-packages/jupyterlab
[I 13:27:12.331 NotebookApp] JupyterLab application directory is /opt/anaconda3/share/jupyter/lab
[I 13:27:12.333 NotebookApp] Serving notebooks from local directory: /Home/Users/kmazouzi
[I 13:27:12.333 NotebookApp] The Jupyter Notebook is running at:
[I 13:27:12.333 NotebookApp] http://localhost:8888/?token=1dc192cd50aa63f1320fe3c6efaa341f507c087179140ba3
[I 13:27:12.333 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:27:12.338 NotebookApp]
To access the notebook, open this file in a browser:
file:///Home/Users/kmazouzi/.local/share/jupyter/runtime/nbserver-31962-open.html
Or copy and paste one of these URLs:
http://localhost:8888/?token=1dc192cd50aa63f1320fe3c6efaa341f507c087179140ba3
Replace ''localhost'' by ''mesogpu1.univ-fcomte.fr'' in the URL above. And then connect with your preferred navigator. For example: ''URL=http://mesogpu1.univ-fcomte.fr:8888/?token=1dc192cd50aa63f1320fe3c6efaa341f507c087179140ba3''
*/
===== Using Singularity v3 =====
This section is for exprerts and advanced usage
We can use Singularity's ''pull'' sub-command to import a container image directly from [[https://hub.docker.com|Docker Hub]] without having root or superuser privileges (or Docker) on your host system.
For example, to use Singularity to import the image of the latest Tensorflow (GPU version) into the present working directory.
$ singularity pull tensorflow-gpu.simg docker://tensorflow/tensorflow:latest-gpu
Notice that we prepend the original URL with docker:// prefix.
This will download and build singularity image named ''tensorflow-gpu.simg''
We can spawn an interactive shell within a container with the ''shell'' sub-command.
$ singularity shell --nv tensorflow-gpu.simg
Or execute directly tensorflow within container:
$ singularity exec --nv tensorflow-gpu.simg python mytensorflow.py
The parameter ''--nv'' is mandatory for GPU applications.
===== Monitoring =====
==== Command line ====
From login or ''mesoshared'' nodes you can access to all GPUs stats using ''voltastat'' command.
For example:
user@mesologin1:~# voltastat
node2-69 Wed Mar 4 23:47:01 2020 440.33.01
[0] Tesla V100-PCIE-16GB | 36'C, 80 % | 15553 / 16160 MB | plop(15541M)
[1] Tesla V100-PCIE-16GB | 28'C, 0 % | 0 / 16160 MB |
[2] Tesla V100-PCIE-16GB | 28'C, 0 % | 0 / 16160 MB |
node3-70 Wed Mar 4 23:46:01 2020 440.33.01
[0] Tesla V100-SXM2-32GB | 34'C, 0 % | 0 / 32510 MB |
[1] Tesla V100-SXM2-32GB | 32'C, 0 % | 0 / 32510 MB |
[2] Tesla V100-SXM2-32GB | 30'C, 0 % | 0 / 32510 MB |
[3] Tesla V100-SXM2-32GB | 34'C, 0 % | 0 / 32510 MB |
''voltastat'' has 10s granularity. i.e it request GPU stats each 10 second interval
==== Ganglia Graph====
* Direct Ganglia graph to [[https://mesoportail.univ-fcomte.fr/ganglia-cluster/graph_all_periods.php?title=node2-69+node&cs=&ce=&vl=&x=&n=&hreg[]=node2-69&mreg[]=gpu0_util&mreg[]=gpu1_util&mreg[]=gpu2_util>ype=line&glegend=show&aggregate=1| node2-69 ]]
* Direct Ganglia graph to [[https://mesoportail.univ-fcomte.fr/ganglia-cluster/graph_all_periods.php?title=node3-70+node&cs=&ce=&vl=&x=&n=&hreg[]=node3-70&mreg[]=gpu0_util&mreg[]=gpu1_util&mreg[]=gpu2_util&mreg[]=gpu3_util>ype=line&glegend=show&aggregate=1| node3-70 ]]
{{https://mesoportail.univ-fcomte.fr/ganglia-cluster/graph.php?r=day&z=xlarge&title=node2-69+node&vl=&x=&n=&hreg[]=node2-69&mreg[]=gpu0_util&mreg[]=gpu1_util&mreg[]=gpu2_util>ype=line&glegend=show&aggregate=1&.png?}}
----
{{https://mesoportail.univ-fcomte.fr/ganglia-cluster/graph.php?r=day&z=xlarge&title=node3-70+node&vl=&x=&n=&hreg[]=node3-70&mreg[]=gpu0_util&mreg[]=gpu1_util&mreg[]=gpu2_util&mreg[]=gpu3_util>ype=line&glegend=show&aggregate=1&.png?}}
====== FAQ ======
??? How to request GPU having 32Gb of memory?
!!! Use this SGE option: ''-l dgx=1''
??? How to check if my Job use really using GPU?
!!! On the submit host, use this command: ''voltastat'' this will pull the GPU each 10s
??? How to install caffe-gpu (Bonus)?
Once connected to the node, create or enter to your environment. For example :
$ conda activate $WORK/conda/meso and install caffe-gpu with conda:
$ conda install caffe-gpu
To test your installation import caffe
caffe.set_mode_gpu()
caffe.set_device(0)
??? How to use Jupyter Notebook?
!!! Please follow this tutorial [[deeplearning::dellvolta100::jupyter]]