TURBOMOLE
TURBOMOLE is an ab initio computational chemistry program that implements various quantum chemistry methods.
Table of Contents
There are multiple versions of TURBOMOLE installed, in serial, SMP and MPI parallelised flavours. They are distinguished by a suffix to the version. For instance version 7.5.1, 7.5.1-smp and 7.5.1-mpi respectively.
TURBOMOLE can be loaded directly:
module load TURBOMOLE/7.5.1
Available TURBOMOLE versions can be listed with module spider TURBOMOLE. Specifying a version will list the needed modules: module spider TURBOMOLE/7.5.1
The smp and mpi variants load basically the same TURBOMOLE installation but set some environment variables as described in [2] and [3], respectively. These modules are also prepared to interact with our Slurm batch system (inherit some values for environment variables), you you are kindly requested to NOT set/owerwrite these environment variables but follow the example scripts below.
- http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html
- http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html#x16-330003.2.2
- http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html#x16-240003.2.1
Serial job
#!/usr/bin/zsh
### Job name
#SBATCH --job-name=TURBOMOLE_SERIAL
### File / path where the output will be written, the %J is the job id
#SBATCH --output=TURBOMOLE_SERIAL.%J
### Request the time you need for execution. The full format is D-HH:MM:SS
### You must at least specify minutes or days and hours and may add or
### leave out any other parameters
#SBATCH --time=80
#### Request memory you need for your job. You can specify this
### in either MB (1024M) or GB (4G).
#SBATCH --mem-per-cpu=3900M
### Change to the work directory, if needed
#cd "$HOME/workdirectory"
### load modules and execute
module load TURBOMOLE/7.5.1
### start non-interactive batch job
jobex [inputfile]
Shared memory parallelised (SMP) job
#!/usr/bin/zsh
### Job name
#SBATCH --job-name=TURBOMOLE_SMP
### File / path where the output will be written, the %J is the job id
#SBATCH --output=%J.TURBOMOLE_SMP
# ask for eight cores
#SBATCH --cpus-per-task=8
### load modules and execute
module load TURBOMOLE/7.5.1-smp
### Do NOT set/tweak environment variables PARNODES, HOSTS_FILE, TURBOMOLE_SYSNAME, PARA_ARCH !!!
### start non-interactive batch job, e.g. do the tests
cp -rL ${TURBODIR}/TURBOTEST ${SLURM_JOB_ID} && cd ${SLURM_JOB_ID}
TTEST
### start non-interactive batch job
#jobex [inputfile]
Distrubuted memory parallelised (MPI) job
#!/usr/bin/zsh
### Job name
#SBATCH --job-name=TURBOMOLE_MPI
### File / path where the output will be written, the %J is the job id
#SBATCH --output=%J.TURBOMOLE_MPI
# ask for eight MPI ranks
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=8
### load modules and execute
module load TURBOMOLE/7.5.1-mpi
### Do NOT set/tweak environment variables PARNODES, HOSTS_FILE, TURBOMOLE_SYSNAME, PARA_ARCH !!!
### start non-interactive batch job, e.g. do the tests
cp -rL ${TURBODIR}/TURBOTEST ${SLURM_JOB_ID} && cd ${SLURM_JOB_ID}
TTEST
### start non-interactive batch job
#jobex [inputfile]
Version 7.3.0 is known to run into some numerical accuracy issues when running
TTEST. Use a different version if in doubt. Example:THIS IS TEST /rwthfs/rz/cluster/work/pk224850/CALLS/20190905-0410-turbomole-lm120094/TESTEN/9733105/dscf/short/LiF.DIFFUSE. LIF- WITH DIFFUSE BFS, KEYWORD DIFFUSE SWITCHED ON, SYMMETRY C3V Fri Sep 27 10:05:01 2019 TEST CALCULATIONS STARTED NOW. PERFORMING COMMANDS: /usr/local_rwth/sw/turbomole/tmX4.4.0/TURBOMOLE/bin/em64t-unknown-linux-gnu_smp/dscf > output.test 2> output.err Total CPU time : 231 s Total wall time: 31.51 s TEST RESULTS: CRITERION NAME TYPE RESULT REFRESULT SUCCESS Total energy c -107.48907339 -107.48907187 failed Kinetic energy c 107.16907564217 107.16897845262 failed number of alpha electrons c 7.0003202974 7.0000849344 failed number of beta electrons c 6.0000000291 6.0000000289 ok number of overall gridpoints c 57257 57257 ok Number of iterations c 343 81 failed 1. max. norm for Fia-block c 2.592E-01 2.592E-01 ok last max. norm for Fia-block c 2.784E-03 1.328E-04 failed 1. max. resid. fock norm c 2.259E+01 2.259E+01 ok last max. resid. fock norm c 7.366E-03 1.328E-04 failed spin of irrep (Fia-block) g ok spin of irrep (fock norm) g failed TOTAL ENERGY NOT CORRECT. KINETIC ENERGY NOT CORRECT. NUMBER OF ALPHA ELECTRONS NOT CORRECT. NUMBER OF ITERATIONS NOT CORRECT. LAST MAX. NORM FOR FIA-BLOCK NOT CORRECT. LAST MAX. RESID. FOCK NORM NOT CORRECT. SPIN OF IRREP (FOCK NORM) NOT CORRECT. THERE IS A PROBLEM WITH /rwthfs/rz/cluster/work/pk224850/CALLS/20190905-0410-turbomole-lm120094/TESTEN/9733105/dscf/short/LiF.DIFFUSE.[workaround] version 7.0.1 need
MPI_IB_PKEYto be set to some crude hex value for both MPI and SMP runs; so we tweaked the pmpi.conf file from the HP-MPI/Parastation MPI component of TURBOMOLE. We tried both values and did not found any diffefence on Intel Omni Path Network, and decided for a random one:$ diff /rwthfs/rz/SW/turbomole/tmX4.11/TURBOMOLE/mpirun_scripts/em64t-unknown-linux-gnu_mpi/PMPI/etc/pmpi.conf* 147,149d146 < # pk224850 20190905-0410 < MPI_IB_PKEY = "0xffff" < #MPI_IB_PKEY = "0xff7f"[workaround] version 7.3.0 has different values for
MPI_IB_PKEY, however this version did not rely on the variable being set,TESTEN/9730908/ridft/short/Ag3_C2v_GGA_dd/TESTDIR.em64t-unknown-linux-gnu_smp srun: TOPOLOGY: warning -- no switch can reach all nodes through its descendants.Do not use route/topology ridft_mpi: Rank 0:4: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY ridft_mpi: Rank 0:2: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY ridft_mpi: Rank 0:4: MPI_Init: pkey table: ridft_mpi: Rank 0:1: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY ridft_mpi: Rank 0:2: MPI_Init: pkey table: ridft_mpi: Rank 0:1: MPI_Init: pkey table: ridft_mpi: Rank 0:4: MPI_Init: 0x8001 ridft_mpi: Rank 0:2: MPI_Init: 0x8001 ridft_mpi: Rank 0:4: MPI_Init: 0x7fff ridft_mpi: Rank 0:2: MPI_Init: 0x7fff ridft_mpi: Rank 0:2: MPI_Init: 0xffff ridft_mpi: Rank 0:1: MPI_Init: 0x8001 ridft_mpi: Rank 0:4: MPI_Init: 0xffff ridft_mpi: Rank 0:5: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY ridft_mpi: Rank 0:2: MPI_Init: ibv_get_pkey() failed ridft_mpi: Rank 0:0: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY ridft_mpi: Rank 0:1: MPI_Init: 0x7fff ridft_mpi: Rank 0:4: MPI_Init: ibv_get_pkey() failed ridft_mpi: Rank 0:1: MPI_Init: 0xffff ridft_mpi: Rank 0:5: MPI_Init: pkey table: ridft_mpi: Rank 0:2: MPI_Init: Can't initialize RDMA device ridft_mpi: Rank 0:0: MPI_Init: pkey table: ridft_mpi: Rank 0:4: MPI_Init: Can't initialize RDMA device ridft_mpi: Rank 0:1: MPI_Init: ibv_get_pkey() failed ridft_mpi: Rank 0:2: MPI_Init: Internal Error: Cannot initialize RDMA protocol ridft_mpi: Rank 0:1: MPI_Init: Can't initialize RDMA device ridft_mpi: Rank 0:5: MPI_Init: 0x8001 ridft_mpi: Rank 0:4: MPI_Init: Internal Error: Cannot initialize RDMA protocol ridft_mpi: Rank 0:1: MPI_Init: Internal Error: Cannot initialize RDMA protocol ridft_mpi: Rank 0:5: MPI_Init: 0x7fff ridft_mpi: Rank 0:0: MPI_Init: 0x8001 ridft_mpi: Rank 0:3: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY ridft_mpi: Rank 0:5: MPI_Init: 0xffff ridft_mpi: Rank 0:0: MPI_Init: 0x7fff ridft_mpi: Rank 0:3: MPI_Init: pkey table: ridft_mpi: Rank 0:0: MPI_Init: 0xffff ridft_mpi: Rank 0:5: MPI_Init: ibv_get_pkey() failed ridft_mpi: Rank 0:5: MPI_Init: Can't initialize RDMA device ridft_mpi: Rank 0:0: MPI_Init: ibv_get_pkey() failed ridft_mpi: Rank 0:0: MPI_Init: Can't initialize RDMA device ridft_mpi: Rank 0:5: MPI_Init: Internal Error: Cannot initialize RDMA protocol ridft_mpi: Rank 0:3: MPI_Init: 0x8001 ridft_mpi: Rank 0:3: MPI_Init: 0x7fff ridft_mpi: Rank 0:0: MPI_Init: Internal Error: Cannot initialize RDMA protocol ridft_mpi: Rank 0:3: MPI_Init: 0xffff ridft_mpi: Rank 0:3: MPI_Init: ibv_get_pkey() failed ridft_mpi: Rank 0:3: MPI_Init: Can't initialize RDMA device ridft_mpi: Rank 0:3: MPI_Init: Internal Error: Cannot initialize RDMA protocol MPI Application rank 0 exited before MPI_Init() with status 1 mpirun: Broken pip[workaround] version 7.3.0 need
MPI_ROOTto be set inpmpi.conffile from the HP-MPI/Parastation MPI component of TURBOMOLE. (This is a side effect of our Intel MPI being loaded by default in the standard environment.)$ diff /rwthfs/rz/SW/turbomole/tmX4.4.0/TURBOMOLE/mpirun_scripts/PMPI/etc/pmpi.conf* 173,177d172 < # pk224850 20190905-0410 < #MPI_IB_PKEY = "0xffff" < #MPI_IB_PKEY = "0x7fff" < #MPI_IB_PKEY = "0x8001" < MPI_ROOT = /rwthfs/rz/SW/turbomole/tmX4.4.0/TURBOMOLE/mpirun_scripts/PMPI- On old InfiniBand (IB) network (= some nodes in Integrative Hosting) the user experience on TURBOMOLE may differ from your wishes. We observe hangups (or at least grand slow-down) of 7.0.1smp on IB nodes; version 7.3.0 seem to be mainly correct. We observe many
TTESTcases with not correct termination on 7.3.0mpi on IB nodes; version 7.0.1smp seem to work correctly. As the old IB nodes are only a small part of the current cluster we will not perform further investigations on these issues.
