turbomole
TURBOMOLE is an ab initio computational chemistry program that implements various quantum chemistry methods.
1. How to Access the Software
There are multiple versions of TURBOMOLE installed, in serial, SMP and MPI parallelised flavours.
module load CHEMISTRY module avail turbomole turbomole-smp turbomole-mpi module load turbomole |
The software is available also with parallelisation [1]. The 'turbomole-smp' and 'turbomole-mpi' variants load basically the same TURBOMOLE installation but set some environment variables as described in [2] and [3], respectively. These modules are also preparied to interact with our SLURM batch system (inherit some values for environment variables), you you are kindly required NOT to set/owerwrite these environment variables (PARNODES, HOSTS_FILE, TURBOMOLE_SYSNAME, PARA_ARCH, ..) but follow the example scripts below.
- http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html
- http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html#x16-330003.2.2
- http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html#x16-240003.2.1
2. Example Batch Scripts
Serial job
#!/usr/local_rwth/bin/zsh ### Job name #SBATCH --job-name=TURBOMOLE_SERIAL ### File / path where the output will be written, the %J is the job id #SBATCH --output=TURBOMOLE_SERIAL.%J ### Request the time you need for execution. The full format is D-HH:MM:SS ### You must at least specify minutes or days and hours and may add or ### leave out any other parameters #SBATCH --time=80 #### Request memory you need for your job. You can specify this ### in either MB (1024M) or GB (4G). #SBATCH --mem-per-cpu=1850M ### Change to the work directory, if needed #cd /home/user/workdirectory ### load modules and execute module load CHEMISTRY module load turbomole ### start non-interactive batch job jobex [inputfile] |
Shared memory parallelised (SMP) job
#!/usr/local_rwth/bin/zsh ### Job name #SBATCH --job-name=TURBOMOLE_SMP ### File / path where the output will be written, the %J is the job id #SBATCH --output=%J.TURBOMOLE_SMP # ask for eight cores #SBATCH --cpus-per-task=8 ### load modules and execute module load CHEMISTRY module load turbomole-smp/7.3.0 ### Do NOT set/tweak environment variables PARNODES, HOSTS_FILE, TURBOMOLE_SYSNAME, PARA_ARCH !!! ### start non-interactive batch job, e.g. do the tests cp -rL ${TURBODIR}/TURBOTEST ${SLURM_JOB_ID} && cd ${SLURM_JOB_ID} TTEST ### start non-interactive batch job #jobex [inputfile] |
Distrubuted memory parallelised (MPI) job
#!/usr/local_rwth/bin/zsh ### Job name #SBATCH --job-name=TURBOMOLE_MPI ### File / path where the output will be written, the %J is the job id #SBATCH --output=%J.TURBOMOLE_MPI # ask for eight MPI ranks #SBATCH --ntasks=8 #SBATCH --ntasks-per-node=8 ### load modules and execute module load CHEMISTRY module load turbomole-mpi/7.3.0 ### Do NOT set/tweak environment variables PARNODES, HOSTS_FILE, TURBOMOLE_SYSNAME, PARA_ARCH !!! ### start non-interactive batch job, e.g. do the tests cp -rL ${TURBODIR}/TURBOTEST ${SLURM_JOB_ID} && cd ${SLURM_JOB_ID} TTEST ### start non-interactive batch job #jobex [inputfile] |
3. Known issues
Verison 7.3.0 is known to run into some numerical accuracy issues when running TTEST; use turbomole-mpi/7.0.1(default) if in doubt. Example:
THIS IS TEST /rwthfs/rz/cluster/work/pk224850/CALLS/20190905-0410-turbomole-lm120094/TESTEN/9733105/dscf/short/LiF.DIFFUSE.LIF- WITH DIFFUSE BFS, KEYWORD DIFFUSE SWITCHED ON, SYMMETRY C3VFri Sep 27 10:05:01 2019TEST CALCULATIONS STARTED NOW.PERFORMING COMMANDS:/usr/local_rwth/sw/turbomole/tmX4.4.0/TURBOMOLE/bin/em64t-unknown-linux-gnu_smp/dscf > output.test 2> output.errTotal CPU time : 231 sTotal wall time: 31.51 sTEST RESULTS:CRITERION NAME TYPE RESULT REFRESULT SUCCESSTotal energy c -107.48907339 -107.48907187 failedKinetic energy c 107.16907564217 107.16897845262 failednumber of alpha electrons c 7.0003202974 7.0000849344 failednumber of beta electrons c 6.0000000291 6.0000000289 oknumber of overall gridpoints c 57257 57257 okNumber of iterations c 343 81 failed1. max. norm for Fia-block c 2.592E-01 2.592E-01 oklast max. norm for Fia-block c 2.784E-03 1.328E-04 failed1. max. resid. fock norm c 2.259E+01 2.259E+01 oklast max. resid. fock norm c 7.366E-03 1.328E-04 failedspin of irrep (Fia-block) g okspin of irrep (fock norm) g failedTOTAL ENERGY NOT CORRECT.KINETIC ENERGY NOT CORRECT.NUMBER OF ALPHA ELECTRONS NOT CORRECT.NUMBER OF ITERATIONS NOT CORRECT.LAST MAX. NORM FOR FIA-BLOCK NOT CORRECT.LAST MAX. RESID. FOCK NORM NOT CORRECT.SPIN OF IRREP (FOCK NORM) NOT CORRECT.THERE IS A PROBLEM WITH /rwthfs/rz/cluster/work/pk224850/CALLS/20190905-0410-turbomole-lm120094/TESTEN/9733105/dscf/short/LiF.DIFFUSE.
[workaround] version 7.0.1 need MPI_IB_PKEY to be set to some crude hex value for both MPI and SMP runs; so we tweaked the pmpi.conf file from the HP-MPI/Parastation MPI component of TURBOMOLE. We tried both values and did not found any diffefence on Intel Omno Path Network, and decieded for a random one:
$ diff /rwthfs/rz/SW/turbomole/tmX4.11/TURBOMOLE/mpirun_scripts/em64t-unknown-linux-gnu_mpi/PMPI/etc/pmpi.conf*147,149d146< # pk224850 20190905-0410< MPI_IB_PKEY = "0xffff"< #MPI_IB_PKEY = "0xff7f"[workaround] version 7.3.0 has different values for MPI_IB_PKEY, however this version did not rely on this variable being set,
TESTEN/9730908/ridft/short/Ag3_C2v_GGA_dd/TESTDIR.em64t-unknown-linux-gnu_smpsrun: TOPOLOGY: warning -- no switch can reach all nodes through its descendants.Do not use route/topologyridft_mpi: Rank 0:4: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEYridft_mpi: Rank 0:2: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEYridft_mpi: Rank 0:4: MPI_Init: pkey table:ridft_mpi: Rank 0:1: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEYridft_mpi: Rank 0:2: MPI_Init: pkey table:ridft_mpi: Rank 0:1: MPI_Init: pkey table:ridft_mpi: Rank 0:4: MPI_Init: 0x8001ridft_mpi: Rank 0:2: MPI_Init: 0x8001ridft_mpi: Rank 0:4: MPI_Init: 0x7fffridft_mpi: Rank 0:2: MPI_Init: 0x7fffridft_mpi: Rank 0:2: MPI_Init: 0xffffridft_mpi: Rank 0:1: MPI_Init: 0x8001ridft_mpi: Rank 0:4: MPI_Init: 0xffffridft_mpi: Rank 0:5: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEYridft_mpi: Rank 0:2: MPI_Init: ibv_get_pkey() failedridft_mpi: Rank 0:0: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEYridft_mpi: Rank 0:1: MPI_Init: 0x7fffridft_mpi: Rank 0:4: MPI_Init: ibv_get_pkey() failedridft_mpi: Rank 0:1: MPI_Init: 0xffffridft_mpi: Rank 0:5: MPI_Init: pkey table:ridft_mpi: Rank 0:2: MPI_Init: Can't initialize RDMA deviceridft_mpi: Rank 0:0: MPI_Init: pkey table:ridft_mpi: Rank 0:4: MPI_Init: Can't initialize RDMA deviceridft_mpi: Rank 0:1: MPI_Init: ibv_get_pkey() failedridft_mpi: Rank 0:2: MPI_Init: Internal Error: Cannot initialize RDMA protocolridft_mpi: Rank 0:1: MPI_Init: Can't initialize RDMA deviceridft_mpi: Rank 0:5: MPI_Init: 0x8001ridft_mpi: Rank 0:4: MPI_Init: Internal Error: Cannot initialize RDMA protocolridft_mpi: Rank 0:1: MPI_Init: Internal Error: Cannot initialize RDMA protocolridft_mpi: Rank 0:5: MPI_Init: 0x7fffridft_mpi: Rank 0:0: MPI_Init: 0x8001ridft_mpi: Rank 0:3: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEYridft_mpi: Rank 0:5: MPI_Init: 0xffffridft_mpi: Rank 0:0: MPI_Init: 0x7fffridft_mpi: Rank 0:3: MPI_Init: pkey table:ridft_mpi: Rank 0:0: MPI_Init: 0xffffridft_mpi: Rank 0:5: MPI_Init: ibv_get_pkey() failedridft_mpi: Rank 0:5: MPI_Init: Can't initialize RDMA deviceridft_mpi: Rank 0:0: MPI_Init: ibv_get_pkey() failedridft_mpi: Rank 0:0: MPI_Init: Can't initialize RDMA deviceridft_mpi: Rank 0:5: MPI_Init: Internal Error: Cannot initialize RDMA protocolridft_mpi: Rank 0:3: MPI_Init: 0x8001ridft_mpi: Rank 0:3: MPI_Init: 0x7fffridft_mpi: Rank 0:0: MPI_Init: Internal Error: Cannot initialize RDMA protocolridft_mpi: Rank 0:3: MPI_Init: 0xffffridft_mpi: Rank 0:3: MPI_Init: ibv_get_pkey() failedridft_mpi: Rank 0:3: MPI_Init: Can't initialize RDMA deviceridft_mpi: Rank 0:3: MPI_Init: Internal Error: Cannot initialize RDMA protocolMPI Application rank 0 exited before MPI_Init() with status 1mpirun: Broken pip
[workaround] version 7.3.0 need MPI_ROOT to be set in pmpi.conf file from the HP-MPI/Parastation MPI component of TURBOMOLE. (This us a side effect of our Intel MPI beind loaded by default in the standard environment.)
$ diff /rwthfs/rz/SW/turbomole/tmX4.4.0/TURBOMOLE/mpirun_scripts/PMPI/etc/pmpi.conf*173,177d172< # pk224850 20190905-0410< #MPI_IB_PKEY = "0xffff"< #MPI_IB_PKEY = "0x7fff"< #MPI_IB_PKEY = "0x8001"< MPI_ROOT = /rwthfs/rz/SW/turbomole/tmX4.4.0/TURBOMOLE/mpirun_scripts/PMPI- On old InfiniBand (IB) network (= some nodes in Integrative Hosting) the user expirince on TURBOMOLE may differ from your wishes. We observe hangup (or at least grand slow-down) of turbomole-smp/7.0.1(default) on IB nodes; version 7.3.0 seem to be mainly correct. We observe many TTEST cases with not correct termination on turbomole-mpi/7.3.0 on IB nodes; version turbomole-smp/7.0.1(default) seem to work correctly. As the old IB nodes are only a small part of the current cluster we will not perform further investigations on these issues.
