OpenMPI fails with slurm

Jochen — Tue, 24 Aug 2010 11:00:50 -0000

OpenMPI fails when started from a TurboVNC Session, please see History

History (beginning @the end):

################## 24.08.2010 - 00:30 pm ##################

I changed my <xstartup.turbovnc> in adding the following line:
for VAR in `env | grep SLURM | while read LINE ; do echo $LINE | cut -d'=' -f1 ; done` ; do unset $VAR ; done

The reason for still not working seems to be the memlock-limit set to “64”, which causes OpenMPI to fail.
The issue here is that an unprivileged user is not allowed to change his limit…

Can you please explain:
- why this low limit is set?
- if this limit option is necessary?

################## 23.08.2010 - 06:30 pm ##################

Please try unset of Slurm values within tvnc session:
- "set | grep SLURM_"
- "unset" alle gefundenen

################## 18.08.2010 - 04:24 pm ##################

<http://www.open-mpi.org/faq/?category=slurm>.
"You need to ensure that SLURM sets up the locked memory limits properly."

################## Initial Report ##################

1) HelloWorld from TVNC-Session fails with OpenMPI:

[11:05:36][khr@viz04: helloworld]$ env | grep -i slurm
[11:05:43][khr@viz04: helloworld]$ mpirun -np 2 MPI_HelloWorld
--------------------------------------------------------------------------
The OpenFabrics (openib) BTL failed to initialize while trying to allocate some locked memory. This typically can indicate that the memlock limits are set too low. For most HPC installations, the memlock limits should be set to "unlimited". The failure occured
here:

Local host: viz04
OMPI source: btl_openib_component.c:1055
Function: ompi_free_list_init_ex_new()
Device: mlx4_0
Memlock limit: 65536

You may need to consult with your system administrator to get this problem fixed. This FAQ entry on the Open MPI web site may also be
helpful:

http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

Local host: viz04
Local device: mlx4_0
--------------------------------------------------------------------------
Process 1 on viz04 out of 2
Process 0 on viz04 out of 2
[viz04:06519] 1 more process has sent help message help-mpi-btl-openib.txt / init-fail-no-mem [viz04:06519] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [viz04:06519] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init

2) HelloWorld after direct SSH-Login on viz04 works fine:
[11:00:19][khr@viz04: helloworld]$ mpirun -np 2 MPI_HelloWorld Process 1 on viz04 out of 2 Process 0 on viz04 out of 2

Usage of single /multiple GPU's

Jochen — Tue, 24 Aug 2010 08:33:50 -0000

Just a question according the reservation of a specific GPU:

When allocating only one GPU on a node with the option "--specific-gpus": Where can I check which GPU realy is used?

Example:
1) Start a job with the second GPU: “viz-tvnc-mpg --specific-gpus -a viz02/1 -g 1920x1200”
vs-info output:
140 hpviz 00:00:04 viz-tvnc-mpg viz02:1 with viz02:5 with 1 GPUs

2) Now start a job with the first GPU: “viz-tvnc-mpg --specific-gpus -a viz02/0 -g 1920x1200”
vs-info output:
140 hpviz 00:01:28 viz-tvnc-mpg viz02:1 with viz02:5 with 1 GPUs
141 hpviz 00:00:04 viz-tvnc-mpg viz02:2 with viz02:6 with 1 GPUs

I would have expected here some output that points to the used GPU, or is there another way to get this information?
(Confusing: the first job get's the tvnc connection string ":1" even if it's on GPU 1, and the second job ":2" even it's on GPU 0)

Thanks in advance & best regards,
bigcompany

Recent changes to support-requests

OpenMPI fails with slurm

Usage of single /multiple GPU's