Différences

Ci-dessous, les différences entre deux révisions de la page.

Lien vers cette vue comparative

Les deux révisions précédentes Révision précédente
Prochaine révision
Révision précédente
volta [2020/01/13 13:11] – adminvolta [2020/01/13 14:07] (Version actuelle) – [Default execution] admin
Ligne 53: Ligne 53:
  
 </code> </code>
 +
 +===== Using gpu-burn tools=====
 +Activate persistence mode:
 +<code bash>
 +# nvidia-smi -pm 1
 +# nvidia-smi -pm 1
 +Enabled persistence mode for GPU 00000000:1A:00.0.
 +Enabled persistence mode for GPU 00000000:1C:00.0.
 +Enabled persistence mode for GPU 00000000:1D:00.0.
 +Enabled persistence mode for GPU 00000000:1E:00.0.
 +
 +</code>
 +==== Default execution ====
 +By default **gpuburn**will choose all available GPU. 
 +
 +<code bash>
 +
 +$ ./gpu_burn 
 +Run length not specified in the command line.  Burning for 10 secs
 +GPU 0: Tesla V100-SXM2-32GB (UUID: GPU-ca51cbd1-4eb0-f265-9cb6-613f848d9ebd)
 +GPU 1: Tesla V100-SXM2-32GB (UUID: GPU-d02431f6-3ad3-e257-2858-8e830e06fa56)
 +GPU 2: Tesla V100-SXM2-32GB (UUID: GPU-2c34530c-3db6-3cd2-df6b-39f65d3448c2)
 +GPU 3: Tesla V100-SXM2-32GB (UUID: GPU-c8740a89-14af-b1f3-d17c-629cb2454737)
 +
 +Couldn't init a GPU test: Error: 
 +</code>
 +
 +The computation still running in 3 GPU. The GPU 3 seems to be in error according to the ''dmesg'' : 
 +<code bash>
 +[ 2432.287866] NVRM: Xid (PCI:0000:1e:00): 48, pid=2442, An uncorrectable double bit error (DBE) has been detected on GPU in the framebuffer at partition 5, subpartition 1.
 +[ 2432.288198] NVRM: Xid (PCI:0000:1e:00): 64, pid=2442, Dynamic Page Retirement: Blacklisting failed.  Page should have been blacklisted on boot(0x00000000007fffa9).
 +[ 2432.714101] NVRM: Xid (PCI:0000:1e:00): 64, pid=2442, Dynamic Page Retirement: Blacklisting failed.  Page should have been blacklisted on boot(0x00000000007fffb8).
 +[ 2434.693780] NVRM: Xid (PCI:0000:1e:00): 45, pid=2442, Ch 00000000
 +[ 2434.694046] NVRM: Xid (PCI:0000:1e:00): 45, pid=2442, Ch 00000001
 +[ 2434.694263] NVRM: Xid (PCI:0000:1e:00): 45, pid=2442, Ch 00000002
 +
 +</code>
 +
 +After this run, we could not run ''nvidia-smi'' anymore:
 +<code bash>
 +# nvidia-smi 
 +Unable to determine the device handle for GPU 0000:1E:00.0: Unknown Error
 +
 +</code>
 +
 +It looks like the GPU#3 is not working. Let's try to run directly only in this GPU:
 +<code bash >
 +# CUDA_VISIBLE_DEVICES=3 ./gpu_burn
 +</code> 
 +Error message:
 +
 +''NVRM: Xid (PCI:0000:1e:00): 45, pid=2769, Ch 00000001''
 +
 +
 +
 +
  • volta.1578921095.txt.gz
  • Dernière modification : 2020/01/13 13:11
  • de admin