Différences

Ci-dessous, les différences entre deux révisions de la page.

Lien vers cette vue comparative

Prochaine révision
Révision précédente
volta [2020/01/13 13:06] – créée adminvolta [2020/01/13 14:07] (Version actuelle) – [Default execution] admin
Ligne 1: Ligne 1:
-====== Testing CC410 Node ======+====== Testing Dell C4140 Node ====== 
 +<code bash> 
 +# rpm -q centos-release  
 +centos-release-8.0-0.1905.0.9.el8.x86_64 
 + 
 +</code> 
 + 
 + 
 +<code bash> 
 +# uname -a 
 +Linux node2-70 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux 
 + 
 +</code> 
  
 <code bash> <code bash>
Ligne 9: Ligne 22:
  
 </code> </code>
 +
 +<code bash>
 +# nvidia-smi 
 +Mon Jan 13 14:07:30 2020       
 ++-----------------------------------------------------------------------------+
 +| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
 +|-------------------------------+----------------------+----------------------+
 +| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
 +| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
 +|===============================+======================+======================|
 +|   0  Tesla V100-SXM2...  On   | 00000000:1A:00.0 Off |                    0 |
 +| N/A   37C    P0    44W / 300W |      0MiB / 32510MiB |      0%      Default |
 ++-------------------------------+----------------------+----------------------+
 +|   1  Tesla V100-SXM2...  On   | 00000000:1C:00.0 Off |                    0 |
 +| N/A   34C    P0    42W / 300W |      0MiB / 32510MiB |      0%      Default |
 ++-------------------------------+----------------------+----------------------+
 +|   2  Tesla V100-SXM2...  On   | 00000000:1D:00.0 Off |                    0 |
 +| N/A   33C    P0    41W / 300W |      0MiB / 32510MiB |      0%      Default |
 ++-------------------------------+----------------------+----------------------+
 +|   3  Tesla V100-SXM2...  On   | 00000000:1E:00.0 Off |                    0 |
 +| N/A   36C    P0    45W / 300W |      0MiB / 32510MiB |      0%      Default |
 ++-------------------------------+----------------------+----------------------+
 +                                                                               
 ++-----------------------------------------------------------------------------+
 +| Processes:                                                       GPU Memory |
 +|  GPU       PID   Type   Process name                             Usage      |
 +|=============================================================================|
 +|  No running processes found                                                 |
 ++-----------------------------------------------------------------------------+
 +
 +</code>
 +
 +===== Using gpu-burn tools=====
 +Activate persistence mode:
 +<code bash>
 +# nvidia-smi -pm 1
 +# nvidia-smi -pm 1
 +Enabled persistence mode for GPU 00000000:1A:00.0.
 +Enabled persistence mode for GPU 00000000:1C:00.0.
 +Enabled persistence mode for GPU 00000000:1D:00.0.
 +Enabled persistence mode for GPU 00000000:1E:00.0.
 +
 +</code>
 +==== Default execution ====
 +By default **gpuburn**will choose all available GPU. 
 +
 +<code bash>
 +
 +$ ./gpu_burn 
 +Run length not specified in the command line.  Burning for 10 secs
 +GPU 0: Tesla V100-SXM2-32GB (UUID: GPU-ca51cbd1-4eb0-f265-9cb6-613f848d9ebd)
 +GPU 1: Tesla V100-SXM2-32GB (UUID: GPU-d02431f6-3ad3-e257-2858-8e830e06fa56)
 +GPU 2: Tesla V100-SXM2-32GB (UUID: GPU-2c34530c-3db6-3cd2-df6b-39f65d3448c2)
 +GPU 3: Tesla V100-SXM2-32GB (UUID: GPU-c8740a89-14af-b1f3-d17c-629cb2454737)
 +
 +Couldn't init a GPU test: Error: 
 +</code>
 +
 +The computation still running in 3 GPU. The GPU 3 seems to be in error according to the ''dmesg'' : 
 +<code bash>
 +[ 2432.287866] NVRM: Xid (PCI:0000:1e:00): 48, pid=2442, An uncorrectable double bit error (DBE) has been detected on GPU in the framebuffer at partition 5, subpartition 1.
 +[ 2432.288198] NVRM: Xid (PCI:0000:1e:00): 64, pid=2442, Dynamic Page Retirement: Blacklisting failed.  Page should have been blacklisted on boot(0x00000000007fffa9).
 +[ 2432.714101] NVRM: Xid (PCI:0000:1e:00): 64, pid=2442, Dynamic Page Retirement: Blacklisting failed.  Page should have been blacklisted on boot(0x00000000007fffb8).
 +[ 2434.693780] NVRM: Xid (PCI:0000:1e:00): 45, pid=2442, Ch 00000000
 +[ 2434.694046] NVRM: Xid (PCI:0000:1e:00): 45, pid=2442, Ch 00000001
 +[ 2434.694263] NVRM: Xid (PCI:0000:1e:00): 45, pid=2442, Ch 00000002
 +
 +</code>
 +
 +After this run, we could not run ''nvidia-smi'' anymore:
 +<code bash>
 +# nvidia-smi 
 +Unable to determine the device handle for GPU 0000:1E:00.0: Unknown Error
 +
 +</code>
 +
 +It looks like the GPU#3 is not working. Let's try to run directly only in this GPU:
 +<code bash >
 +# CUDA_VISIBLE_DEVICES=3 ./gpu_burn
 +</code> 
 +Error message:
 +
 +''NVRM: Xid (PCI:0000:1e:00): 45, pid=2769, Ch 00000001''
 +
 +
 +
 +
  • volta.1578920816.txt.gz
  • Dernière modification : 2020/01/13 13:06
  • de admin