Ceci est une ancienne révision du document !


Performance studies

The objective of this study is to analyze xgboost's performance on the cluster.

Prerequisites

  • Python Ananoda 3.7
  • Xgboost (Installed with conda)
  • Mesoshared (CPU Xeon Platinum 8160, 2.1 GHZ) With 92 cores

Covert the notebook to python script:

$ ipython nbconvert --to python 3.XGBoost
Xgboost is multithreaed. We control the number of threads by OMP_NUm_THREADS env variable

For example to for 8 cores:

$ module load lang/python/anaconda/3.7
$ OMP_NUM_TREADS=8 time ipython 3.XGBoost.py
# Threads Time(m)
111m32s
82m03s
161m18s
241m8s
321m5s
401m5s
481m5s
641m54s
722m13s
923m21s
Conclusion : Uper limit scalability is about 24 cores

We will use Volta GPU node.

The package py-xgboost-gpu is installed in this node with conda
$ qlogin -qvolta.q -l h_vmem=20G
We need to modify our program to use GPU. I just added this param tree_method= 'gpu_hist' in xgb_reg object.
150 xgb_reg = xgboost.XGBRegressor(learning_rate = 0.01,
151                                        max_depth = 10,
152                                        random_state=42,
153                                        n_estimators = 173,
154                                        tree_method= 'gpu_hist',
155                                        n_jobs=-1,
156                                        objective = 'count:poisson')
 
  • perf/xgboost.1562335899.txt.gz
  • Dernière modification : 2019/07/05 14:11
  • de admin