OpenFOAM is a free, open source CFD software package. OpenFOAM has an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

Table of Contents

1. How to access the software
2. User libraries and extensions
3. Example batch scripts
4. Further information / Known issues

How to access the software

Depending on the version, you might have to load additional modules until you can load OpenFOAM:

module load GCC/10.3.0
module load OpenMPI/4.1.1
module load OpenFOAM/9

Available OpenFOAM versions can be listed with module spider OpenFOAM. Specifying a version will list the needed modules: module spider OpenFOAM/9

User libraries and extensions

There are many extensions and libraries for OpenFOAM like cfMesh and any user is able to write their own applications. These are to be installed into the user's $HOME directory, typically into $WM_PROJECT_USER_DIR. Typical installation of such an extension consists of:

• downloading the tarball, extract it into some directory, cd into it
• load the OpenFOAM module you like to use
• call Allwclean and Allwmake or somewhat alike.

Note that in order to use the extension you will need the same OpenFOAM module to be loaded; changing the version of OpenFOAM means you need to recompile the extension/library/application.

Example batch scripts

Serial Job:

#!/usr/bin/zsh

### Job name
#SBATCH --job-name=FOAM_SERIAL

### Request the time you need for execution. The full format is D-HH:MM:SS
### You must at least specify minutes or days and hours and may add or
### leave out any other parameters
#SBATCH --time=80

#### Request the memory you need for your job. You can specify this
### in either MB (1024M) or GB (4G).
#SBATCH --mem-per-cpu=3900M

### load modules
module load GCC/10.3.0
module load OpenMPI/4.1.1
module load OpenFOAM/9

### Create a project directory, Copy the tutorial examples
mkdir -p $FOAM_RUN
cp -Lr "$FOAM_TUTORIALS" "$FOAM_RUN"
cd "$FOAM_RUN/tutorials/incompressible/icoFoam/cavity/cavity"

# start non-interactive batch job
blockMesh
icoFoam

# (OFF) after the batch job has finished
#paraFoam

MPI Parallel Job:

Running applications in parallel

#!/usr/bin/zsh

### Job name
#SBATCH --job-name=FOAM_SERIAL

### Request the time you need for execution. The full format is D-HH:MM:SS
### You must at least specify minutes or days and hours and may add or
### leave out any other parameters
#SBATCH --time=80

#### Request the memory you need for your job. You can specify this
### in either MB (1024M) or GB (4G).
#SBATCH --mem-per-cpu=3900M

### Request 12 processes, all on a single node
#SBATCH --nodes=1
#SBATCH --ntasks=12

### Load the required module files
module load GCC/10.3.0
module load OpenMPI/4.1.1
module load OpenFOAM/9

### start the OpenFOAM binary in parallel, cf.
$MPIEXEC $FLAGS_MPI_BATCH dieselFoam -parallel

Further information / Known issues

• Known issue on parallel start: As described in Running applications in parallel you have to start you application in parallel by using $MPIEXEC and add the -parallel flag. However, if you get this error (known for multiphaseInterFoam):

  --> FOAM FATAL ERROR in Foam::findEtcFiles() : could not find mandatory file
      'controlDict'
  

  try to use foamExec wrapper as workaround (see forum entry):

  $MPIEXEC $FLAGS_MPI_BATCH foamExec multiphaseInterFoam -parallel
  

  In case this still does not work define a batch job which will definitely run on a single node.

• SIGFPE, coredump, exit 136 and friends

  Your calculation is aborted with messages like this:

  #0  Foam::error::printStack(Foam::Ostream&) at ??:?
  #1  Foam::sigFpe::sigHandler(int) at ??:?
  #2  ? in "/lib64/libc.so.6"
  #3  void Foam::fv::rotorDiskSource::calculate<Foam::geometricOneField>(Foam::geometricOneField const&, Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&, Foam::Field<Foam::Vector<double> >&, bool, bool) const at ??:?
  #4  Foam::fv::rotorDiskSource::addSup(Foam::fvMatrix<Foam::Vector<double> >&, int) at ??:?
  #5  ? at ??:?
  #6  ? at ??:?
  #7  __libc_start_main in "/lib64/libc.so.6"
  #8  ? at ??:?
  

  (in Slurm you get an exit code 136; you also may find a file named "core.login18-1.hpc.itc.rwth-aachen.de.271599.11" in your directory). What does that mean?

  • The application did run into a Floating point exception which could be a division by 0 or an over/underflow, see
    • signal man page
    • forum entry
  • The root of issue could be
    • an error in the application (less likely but possible in OpenFOAM-provided binaries, more likely in self-written/-installed extensions, or
    • some numerical issue (round-off error), or
    • physically meaningless data set, or a data set with error(s), or
    • something else (Learning never stops).
  • Possible ways to go:
    • check your data set

    • debug your application (if #3 line in stack dump points to your appilcation), see OpenFOAM wiki on HowTo debugging

    • try out a different version of OpenFOAM (newer is usually better, sometimes an old good one is good for you)

    • try out the same version compiled by different compiler, e.g. by GCC:

      prior to loading the OpenFOAM module in a different shell/batch job!

    • set/change FOAM_SIGFPE, FOAM_SETNAN envvars, see OpenFOAM API
      • WARNING: YOU MAY GET PHYSICALLY MEANINGLESS RESULTS.
         

• paraFoam ParaView issues

  OpenFOAM is huge. It also brings a boxed version of ParaView, which itself is not that easy to be build. Not all versions of OpenFOAM contain a working version of ParaView. However we offer a standalone installation of ParaView.

Jülich - Aachen OpenFOAM User Group

In 2018 an inofficial Jülich - Aachen OpenFOAM User Group was founded. If you are interested in this user group, please subscribe to the mailing list.