1. General Information

OpenFOAM is a free, open source CFD software package released by multiple competitive groups:

• OpenFOAM Foundation: https://openfoam.org/   see https://www.openfoam.com/history/
• OpenCFD Ltd / ESI:   http://www.openfoam.com/  see https://openfoam.org/ip-history/
• The foam-extend project:  http://foam-extend.org  is a fork of the OpenFOAM for Computational Fluid Dynamics (CFD)

In the HPC Cluster, OpenFOAM is usually build in multiple versions (using Intel and GCC compilers, Intel and Open MPI). The version is chosen based on the loaded compiler and MPI modules (by default Intel compiler + Open MPI). Check the available versions (typically incompatible to each other!) by

2. How to Access the Software

IMPORTANT NOTE: the 'openfoam' (as well as other *foam) module is of special kind and has those technical restrictions:

• this module may not be unloaded. You cannot switch from one version of openfoam to other in the same console/xterm. Once loaded, the openfoam module blocks also 'module switch' commands of previously loaded modules. Thus, if you like to use a version of openfoam compiled with different compiler/MPI, please load those (e.g. 'module switch intel gcc') at forst step and load the openfoam module afterwards.
• the openfoam module may not be loaded in the same 'module load' step with other modules. The behaviour of e.g. this command is not defined: 'module load cuda openfoam cudnn' - instead, call 'module load cuda; module load openfoam; module load cudnn' in separate calls.

3. User Libraries and Extensions

There are many extensions and libraries for OpenFOAM like cfMesh, as well as any user is able to write own applications. These are to be installed into the user's $HOME directory, typically into $WM_PROJECT_USER_DIR. Typical installation of such an extension consists of

• downloading the tarbal, extract it into some directory, 'cd' into it
• load the OpenFOAM module you like to use
• call 'Allwclean' and 'Allwmake' or somewhat alike. 

Read the manual!

Note that in order to use the extension you will need the same OpenFOAM module to be loaded; changing the version of OpenFOAM used lead to need to recompile the extension/library/application.

4. Example Batch Scripts

Serial Job

MPI Parallel Job

5. Further Information / Known issues

• Note: Different from most modules, the 'openfoam' module is not unloadable.
  You also cannot switch to other version of OpenFOAM in the same shell; use different shells / batch jobs for different versions of OpenFOAM.

• Known issue on parallel start: As described in http://www.openfoam.org/docs/user/running-applications-parallel.php  you have to start you application in parallel by using $MPIEXEC and add the '-parallel' flag. However, if you  get this error (known for multiphaseInterFoam):

  --> FOAM FATAL ERROR in Foam::findEtcFiles() : could not find mandatory file     'controlDict'

  ... try to use 'foamExec' wrapper as workaround (see https://www.cfd-online.com/Forums/openfoam-solving/69561-foam-fatal-error-foam-findetcfile-could-not-find-mandatory-file.html#post322759):

  $MPIEXEC $FLAGS_MPI_BATCH foamExec multiphaseInterFoam -parallel

  In case this still does not work define a batch job which will definitely run on a single node.

• Known issues on openfoam/3.0.1 (see http://bugs.openfoam.org/view.php?id=2067)
  • 'decomposePar' and 'snappyHexMesh' - fixed by rebuild with the very default optimisation flags.
  • 'fireFoam' fails with message Floating point exception(core dumped) on 'tutorials/combustion/fireFoam/les/flameSpreadWaterSuppressionPanel' - currently no workaround; however future versions should be fixed, already (see above bug report)
• Known issues on openfoam/4.*

  • by loading the module the following message is printed:

    /usr/local_rwth/sw/openfoam/4.1/intel_16.0.2.181-openmpi_1.10.4/OpenFOAM-4.1/etc/config.sh/aliases:73: bad option: -t

    This message is results from incompatibility of ZSH (our default shell) and BASH. There are no limitations in using OpenFOAM known connected to this message; if in doubt try to use the BASH shell.

• SIGFPE, coredump, exit 136 and friends

  Your calculation is aborted with messages like this:

  #0  Foam::error::printStack(Foam::Ostream&) at ??:? #1  Foam::sigFpe::sigHandler(int) at ??:? #2  ? in "/lib64/libc.so.6" #3  void Foam::fv::rotorDiskSource::calculate<Foam::geometricOneField>(Foam::geometricOneField const&, Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&, Foam::Field<Foam::Vector<double> >&, bool, bool) const at ??:? #4  Foam::fv::rotorDiskSource::addSup(Foam::fvMatrix<Foam::Vector<double> >&, int) at ??:? #5  ? at ??:? #6  ? at ??:? #7  __libc_start_main in "/lib64/libc.so.6" #8  ? at ??:?

  (in LSF you get an exit code 136; you alco may find a file named "core.login18-1.hpc.itc.rwth-aachen.de.271599.11" in your directory). What does that mean?

  • The application did run into an Floating point exception which could be a division by 0 or an over/underflow, see
    • http://man7.org/linux/man-pages/man7/signal.7.html
    • https://www.cfd-online.com/Forums/openfoam-programming-development/128397-getting-foam-sigfpe-sighandler-int-custom-code.html
  • The root of issue could be
    • an error in the application (less likely but possible in OpenFOAM-provided binaries, more likely in self-written/-installed extensions, or
    • some numerical issue (round-off error), or
    • physically meaningless data set, or a data set with error(s), or
    • something else (Learning never stops).
  • Possible ways to go:
    • check your data set

    • debug your application (if #3 line in stack dump points to your appilcation), see http://openfoamwiki.net/index.php/HowTo_debugging

    • try out a different version of Open FOAM (new'er is usually better, sometimes an old good one is good for you)

    • try out the same version compiled by different compiler, e.g. by GCC:

      $ module switch intel gcc

      prior loading the OpenFOAM module in a different shell/batch job!

    • set/change FOAM_SIGFPE, FOAM_SETNAN envvars, see https://www.openfoam.com/documentation/guides/latest/api/classFoam_1_1sigFpe.html
      • WARNING: YOU MAY GET PHYSICALLY MEANINGLESS RESULTS.
         

• paraFoam ParaView issues

  OpenFOAM is huge. It also brings a boxed version of ParaView, which itself is not that easy to be build. Not all versions of OpenFOAM contain a working version of ParaView. However we offer a standalone installation of ParaView.
  • Known issues in SLURM/CLAIX18 environment:
  • openfoam/5.0: paraFoam dies with " FATAL ERROR: ParaView reader module libraries do not exist". No workaround known; try to use alternative methods to visualize your data sets after loading ParaView in a separete session, see https://www.openfoam.com/documentation/user-guide/paraview.php
  • openfoam/6: no boxed ParaView; need to load one from modules.  After that 'paraFoam -builtin' dies with '.... paraFoam: line 63: 168444 Segmentation fault      (core dumped) paraview $_opt "$@"'

    • workaround:

      export ParaView_GL=system

  • openfoam/esi-v2006: no boxed ParaView; need to load version paraview/5.6.0 from modules.

• swak4foam

  The swak4Foam is a library many users of OpenFOAM also want to use. This is a very dusty software; in fact we have seem many issues with it. As the last supporter version of OpenFOAM is v2.4 (!!!) we give no warranty it works on our cluster. However we know that using the below commands you are able to compile  swak4foam for OpenFOAM version 6:

  $ module switch intel gcc $ module load TECHNICS $ module load openfoam $ hg clone http://hg.code.sf.net/p/openfoam-extend/swak4Foam $ cd swak4Foam $ time  ./Allwmake 2>&1 | tee 20190613.02.gcc.txt $ ls -la $FOAM_USER_APPBIN $FOAM_USER_LIBBIN

  Note that the GCC-compiled version of OpenFOAM has been used (due to change to GCC form Intel compiler *PRIOR*  loading OpenFOAM).

  The Intel compiler cannot compile swak4foam.

   

Jülich - Aachen OpenFOAM User Group

In 2018 an inofficial Jülich - Aachen OpenFOAM User Group was founded. If you are interested in this user group, please subscribe yourself to the wiki or the mailing list.