Sie befinden sich im Service: RWTH Compute Cluster Linux (HPC)

turbomole

turbomole

Kurzinformation

TURBOMOLE is an ab initio computational chemistry program that implements various quantum chemistry methods.


 Detailinformation

1. How to Access the Software

There are multiple versions of TURBOMOLE installed, in serial, SMP and MPI parallelised flavours.

module load CHEMISTRY
module avail turbomole turbomole-smp turbomole-mpi
module load turbomole

The software is available also with parallelisation [1]. The 'turbomole-smp' and 'turbomole-mpi' variants load basically the same TURBOMOLE installation but set some environment variables as described in [2] and [3], respectively. These modules are also preparied to interact with our SLURM batch system (inherit some values for environment variables), you you are kindly required NOT to set/owerwrite these environment variables (PARNODES, HOSTS_FILE, TURBOMOLE_SYSNAME, PARA_ARCH, ..) but follow the example scripts below.

  1. http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html
  2. http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html#x16-330003.2.2
  3. http://www.cosmologic-services.de/downloads/TM72-documentation/DOKse9.html#x16-240003.2.1

2. Example Batch Scripts

Serial job

#!/usr/local_rwth/bin/zsh
 
### Job name
#SBATCH --job-name=TURBOMOLE_SERIAL
 
### File / path where the output will be written, the %J is the job id
#SBATCH --output=TURBOMOLE_SERIAL.%J
 
### Request the time you need for execution. The full format is D-HH:MM:SS
### You must at least specify minutes or days and hours and may add or
### leave out any other parameters
#SBATCH --time=80
 
#### Request memory you need for your job. You can specify this
### in either MB (1024M) or GB (4G).
#SBATCH --mem-per-cpu=1850M
 
### Change to the work directory, if needed
#cd /home/user/workdirectory
 
### load modules and execute
module load CHEMISTRY
module load turbomole
 
 
### start non-interactive batch job
jobex [inputfile]
 

Shared memory parallelised (SMP) job

#!/usr/local_rwth/bin/zsh
  
### Job name
#SBATCH --job-name=TURBOMOLE_SMP
  
### File / path where the output will be written, the %J is the job id
#SBATCH --output=%J.TURBOMOLE_SMP
 
# ask for eight cores
#SBATCH --cpus-per-task=8
 
 
### load modules and execute
module load CHEMISTRY
module load turbomole-smp/7.3.0
 
### Do NOT set/tweak environment variables PARNODES, HOSTS_FILE, TURBOMOLE_SYSNAME, PARA_ARCH !!!
### start non-interactive batch job, e.g. do the tests
cp -rL ${TURBODIR}/TURBOTEST ${SLURM_JOB_ID}  && cd ${SLURM_JOB_ID}
TTEST
 
### start non-interactive batch job
#jobex [inputfile]
 

Distrubuted memory parallelised (MPI) job

#!/usr/local_rwth/bin/zsh
  
### Job name
#SBATCH --job-name=TURBOMOLE_MPI
  
### File / path where the output will be written, the %J is the job id
#SBATCH --output=%J.TURBOMOLE_MPI
 
# ask for eight MPI ranks
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=8
 
### load modules and execute
module load CHEMISTRY
module load turbomole-mpi/7.3.0
 
### Do NOT set/tweak environment variables PARNODES, HOSTS_FILE, TURBOMOLE_SYSNAME, PARA_ARCH !!!
### start non-interactive batch job, e.g. do the tests
cp -rL ${TURBODIR}/TURBOTEST ${SLURM_JOB_ID}  && cd ${SLURM_JOB_ID}
TTEST
 
### start non-interactive batch job
#jobex [inputfile]
 

3. Known issues

  • Verison 7.3.0 is known to run into some numerical accuracy issues when running  TTEST; use turbomole-mpi/7.0.1(default) if in doubt. Example:

    THIS IS TEST /rwthfs/rz/cluster/work/pk224850/CALLS/20190905-0410-turbomole-lm120094/TESTEN/9733105/dscf/short/LiF.DIFFUSE.
    LIF- WITH DIFFUSE BFS, KEYWORD DIFFUSE SWITCHED ON, SYMMETRY C3V
    Fri Sep 27 10:05:01 2019
    TEST CALCULATIONS STARTED NOW.
    PERFORMING COMMANDS:
    /usr/local_rwth/sw/turbomole/tmX4.4.0/TURBOMOLE/bin/em64t-unknown-linux-gnu_smp/dscf  > output.test 2> output.err
    Total CPU time : 231 s
    Total wall time: 31.51 s
    TEST RESULTS:
    CRITERION NAME                 TYPE RESULT          REFRESULT           SUCCESS
    Total energy                   c    -107.48907339   -107.48907187       failed 
    Kinetic energy                 c    107.16907564217 107.16897845262     failed 
    number of alpha electrons      c    7.0003202974    7.0000849344        failed 
    number of beta electrons       c    6.0000000291    6.0000000289        ok     
    number of overall gridpoints   c    57257           57257               ok     
    Number of iterations           c    343             81                  failed 
    1. max. norm for Fia-block     c    2.592E-01       2.592E-01           ok     
    last max. norm for Fia-block   c    2.784E-03       1.328E-04           failed 
    1. max. resid. fock norm       c    2.259E+01       2.259E+01           ok     
    last max. resid. fock norm     c    7.366E-03       1.328E-04           failed 
    spin of irrep (Fia-block)      g                                        ok     
    spin of irrep (fock norm)      g                                        failed 
    TOTAL ENERGY NOT CORRECT.
    KINETIC ENERGY NOT CORRECT.
    NUMBER OF ALPHA ELECTRONS NOT CORRECT.
    NUMBER OF ITERATIONS NOT CORRECT.
    LAST MAX. NORM FOR FIA-BLOCK NOT CORRECT.
    LAST MAX. RESID. FOCK NORM NOT CORRECT.
    SPIN OF IRREP (FOCK NORM) NOT CORRECT.
    THERE IS A PROBLEM WITH /rwthfs/rz/cluster/work/pk224850/CALLS/20190905-0410-turbomole-lm120094/TESTEN/9733105/dscf/short/LiF.DIFFUSE.
 
  • [workaround] version 7.0.1 need MPI_IB_PKEY to be set to some crude hex value for both MPI and SMP runs; so we tweaked the pmpi.conf file from the HP-MPI/Parastation MPI component of TURBOMOLE. We tried both values and did not found any diffefence on Intel Omno Path Network, and decieded for a random one:

    $ diff /rwthfs/rz/SW/turbomole/tmX4.11/TURBOMOLE/mpirun_scripts/em64t-unknown-linux-gnu_mpi/PMPI/etc/pmpi.conf*
    147,149d146
    < # pk224850 20190905-0410
    < MPI_IB_PKEY = "0xffff"
    < #MPI_IB_PKEY = "0xff7f"
  • [workaround] version 7.3.0 has different values for MPI_IB_PKEY, however this version did not rely on this variable being set,

    TESTEN/9730908/ridft/short/Ag3_C2v_GGA_dd/TESTDIR.em64t-unknown-linux-gnu_smp
    srun: TOPOLOGY: warning -- no switch can reach all nodes through its descendants.Do not use route/topology
    ridft_mpi: Rank 0:4: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY
    ridft_mpi: Rank 0:2: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY
    ridft_mpi: Rank 0:4: MPI_Init: pkey table:
    ridft_mpi: Rank 0:1: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY
    ridft_mpi: Rank 0:2: MPI_Init: pkey table:
    ridft_mpi: Rank 0:1: MPI_Init: pkey table:
    ridft_mpi: Rank 0:4: MPI_Init: 0x8001
    ridft_mpi: Rank 0:2: MPI_Init: 0x8001
    ridft_mpi: Rank 0:4: MPI_Init: 0x7fff
    ridft_mpi: Rank 0:2: MPI_Init: 0x7fff
    ridft_mpi: Rank 0:2: MPI_Init: 0xffff
    ridft_mpi: Rank 0:1: MPI_Init: 0x8001
    ridft_mpi: Rank 0:4: MPI_Init: 0xffff
    ridft_mpi: Rank 0:5: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY
    ridft_mpi: Rank 0:2: MPI_Init: ibv_get_pkey() failed
    ridft_mpi: Rank 0:0: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY
    ridft_mpi: Rank 0:1: MPI_Init: 0x7fff
    ridft_mpi: Rank 0:4: MPI_Init: ibv_get_pkey() failed
    ridft_mpi: Rank 0:1: MPI_Init: 0xffff
    ridft_mpi: Rank 0:5: MPI_Init: pkey table:
    ridft_mpi: Rank 0:2: MPI_Init: Can't initialize RDMA device
    ridft_mpi: Rank 0:0: MPI_Init: pkey table:
    ridft_mpi: Rank 0:4: MPI_Init: Can't initialize RDMA device
    ridft_mpi: Rank 0:1: MPI_Init: ibv_get_pkey() failed
    ridft_mpi: Rank 0:2: MPI_Init: Internal Error: Cannot initialize RDMA protocol
    ridft_mpi: Rank 0:1: MPI_Init: Can't initialize RDMA device
    ridft_mpi: Rank 0:5: MPI_Init: 0x8001
    ridft_mpi: Rank 0:4: MPI_Init: Internal Error: Cannot initialize RDMA protocol
    ridft_mpi: Rank 0:1: MPI_Init: Internal Error: Cannot initialize RDMA protocol
    ridft_mpi: Rank 0:5: MPI_Init: 0x7fff
    ridft_mpi: Rank 0:0: MPI_Init: 0x8001
    ridft_mpi: Rank 0:3: MPI_Init: multiple pkey found in partition key table, please choose one via MPI_IB_PKEY
    ridft_mpi: Rank 0:5: MPI_Init: 0xffff
    ridft_mpi: Rank 0:0: MPI_Init: 0x7fff
    ridft_mpi: Rank 0:3: MPI_Init: pkey table:
    ridft_mpi: Rank 0:0: MPI_Init: 0xffff
    ridft_mpi: Rank 0:5: MPI_Init: ibv_get_pkey() failed
    ridft_mpi: Rank 0:5: MPI_Init: Can't initialize RDMA device
    ridft_mpi: Rank 0:0: MPI_Init: ibv_get_pkey() failed
    ridft_mpi: Rank 0:0: MPI_Init: Can't initialize RDMA device
    ridft_mpi: Rank 0:5: MPI_Init: Internal Error: Cannot initialize RDMA protocol
    ridft_mpi: Rank 0:3: MPI_Init: 0x8001
    ridft_mpi: Rank 0:3: MPI_Init: 0x7fff
    ridft_mpi: Rank 0:0: MPI_Init: Internal Error: Cannot initialize RDMA protocol
    ridft_mpi: Rank 0:3: MPI_Init: 0xffff
    ridft_mpi: Rank 0:3: MPI_Init: ibv_get_pkey() failed
    ridft_mpi: Rank 0:3: MPI_Init: Can't initialize RDMA device
    ridft_mpi: Rank 0:3: MPI_Init: Internal Error: Cannot initialize RDMA protocol
    MPI Application rank 0 exited before MPI_Init() with status 1
    mpirun: Broken pip
 
  • [workaround] version 7.3.0 need MPI_ROOT to be set in pmpi.conf file from the HP-MPI/Parastation MPI component of TURBOMOLE. (This us a side effect of our Intel MPI beind loaded by default in the standard environment.)

    $ diff /rwthfs/rz/SW/turbomole/tmX4.4.0/TURBOMOLE/mpirun_scripts/PMPI/etc/pmpi.conf*
    173,177d172
    < # pk224850 20190905-0410
    < #MPI_IB_PKEY = "0xffff"
    < #MPI_IB_PKEY = "0x7fff"
    < #MPI_IB_PKEY = "0x8001"
    < MPI_ROOT = /rwthfs/rz/SW/turbomole/tmX4.4.0/TURBOMOLE/mpirun_scripts/PMPI
  • On old InfiniBand (IB) network (= some nodes in Integrative Hosting) the user expirince on TURBOMOLE may differ from your wishes. We observe hangup (or at least grand slow-down) of turbomole-smp/7.0.1(default) on IB nodes; version 7.3.0 seem to be mainly correct. We observe many TTEST cases with not correct termination on turbomole-mpi/7.3.0 on IB nodes; version turbomole-smp/7.0.1(default) seem to work correctly. As the old IB nodes are only a small part of the current cluster we will not perform further investigations on these issues.

zuletzt geändert am 29.01.2021

Wie hat Ihnen dieser Inhalt geholfen?

Creative Commons Lizenzvertrag
Dieses Werk ist lizenziert unter einer Creative Commons Namensnennung - Weitergabe unter gleichen Bedingungen 3.0 Deutschland Lizenz