Ceci est une ancienne révision du document !


Performance studies

The objective of this study is to analyze xgboost's performance on the cluster.

Prerequisites

  • Python Ananoda 3.7
  • Xgboost (Installed with conda)
  • Mesoshared (CPU Xeon Platinum 8160, 2.1 GHZ) With 92 cores

Covert the notebook to python script:

$ ipython nbconvert --to python 3.XGBoost
Xgboost is multithreaed. We control the number of threads by OMP_NUm_THREADS env variable

For example to for 8 cores:

$ module load lang/python/anaconda/3.7
$ OMP_NUM_TREADS=8 time ipython 3.XGBoost.py
# Threads Time(m)
111m32s
82m03s
161m18s
241m8s
321m5s
401m5s
481m5s
641m54s
722m13s
923m21s
Conclusion : Uper limit scalability is about 24 cores

We will use Volta GPU node.

The package py-xgboost-gpu is installed in this node with conda
$ qlogin -qvolta.q -l h_vmem=20G
We need to modify our program to use GPU. I just added this param tree_method= 'gpu_hist' in xgb_reg object.

<code bash>

150 xgb_reg = xgboost.XGBRegressor(learning_rate = 0.01, 151 max_depth = 10, 152 random_state=42, 153 n_estimators = 173, 154 tree_method= 'gpu_hist', 155 n_jobs=-1, 156 objective = 'count:poisson')

<code>

  • perf/xgboost.1562335882.txt.gz
  • Dernière modification : 2019/07/05 14:11
  • de admin