The objective of this study is to analyze xgboost's performance on the cluster.
Prerequisites
Covert the notebook to python script:
$ ipython nbconvert --to python 3.XGBoost
OMP_NUm_THREADS env variable
For example to for 8 cores:
$ module load lang/python/anaconda/3.7 $ OMP_NUM_TREADS=8 time ipython 3.XGBoost.py
| # Threads | Time(m) |
|---|---|
| 1 | 11m32s |
| 8 | 2m03s |
| 16 | 1m18s |
| 24 | 1m8s |
| 32 | 1m5s |
| 40 | 1m5s |
| 48 | 1m5s |
| 64 | 1m54s |
| 72 | 2m13s |
| 92 | 3m21s |
We will use Volta GPU node.
py-xgboost-gpu is installed in this node with conda
$ qlogin -qvolta.q -l h_vmem=20G
tree_method= 'gpu_hist' in xgb_reg object.
150 xgb_reg = xgboost.XGBRegressor(learning_rate = 0.01, 151 max_depth = 10, 152 random_state=42, 153 n_estimators = 173, 154 tree_method= 'gpu_hist', 155 n_jobs=-1, 156 objective = 'count:poisson')
We load anaconda
$ export PATH=/opt/anaconda3/bin:$PATH
Finaly we run our program:
$ time ipython 3.XGBoostGPU.py
User time is about 38seconds