openfoam
1. General Information
OpenFOAM is a free, open source CFD software package released by multiple competitive groups:
- OpenFOAM Foundation: https://openfoam.org/ see https://www.openfoam.com/history/
- OpenCFD Ltd / ESI: http://www.openfoam.com/ see https://openfoam.org/ip-history/
- The foam-extend project: http://foam-extend.org is a fork of the OpenFOAM for Computational Fluid Dynamics (CFD)
In the HPC Cluster, OpenFOAM is usually build in multiple versions (using Intel and GCC compilers, Intel and Open MPI). The version is chosen based on the loaded compiler and MPI modules (by default Intel compiler + Open MPI). Check the available versions (typically incompatible to each other!) by
$ module load TECHNICS $ module avail openfoam |
2. How to Access the Software
$ module load TECHNICS $ module load openfoam |
IMPORTANT NOTE: the 'openfoam' (as well as other *foam) module is of special kind and has those technical restrictions:
- this module may not be unloaded. You cannot switch from one version of openfoam to other in the same console/xterm. Once loaded, the openfoam module blocks also 'module switch' commands of previously loaded modules. Thus, if you like to use a version of openfoam compiled with different compiler/MPI, please load those (e.g. 'module switch intel gcc') at forst step and load the openfoam module afterwards.
- the openfoam module may not be loaded in the same 'module load' step with other modules. The behaviour of e.g. this command is not defined: 'module load cuda openfoam cudnn' - instead, call 'module load cuda; module load openfoam; module load cudnn' in separate calls.
3. User Libraries and Extensions
There are many extensions and libraries for OpenFOAM like cfMesh, as well as any user is able to write own applications. These are to be installed into the user's $HOME directory, typically into $WM_PROJECT_USER_DIR. Typical installation of such an extension consists of
- downloading the tarbal, extract it into some directory, 'cd' into it
- load the OpenFOAM module you like to use
- call 'Allwclean' and 'Allwmake' or somewhat alike.
Read the manual!
Note that in order to use the extension you will need the same OpenFOAM module to be loaded; changing the version of OpenFOAM used lead to need to recompile the extension/library/application.
4. Example Batch Scripts
Serial Job
#!/usr/local_rwth/bin/zsh ### Job name #SBATCH --job-name=FOAM_SERIAL ### Request the time you need for execution. The full format is D-HH:MM:SS ### You must at least specify minutes or days and hours and may add or ### leave out any other parameters #SBATCH --time=80 #### Request the memory you need for your job. You can specify this ### in either MB (1024M) or GB (4G). #SBATCH --mem-per-cpu=1850M ### load modules module load TECHNICS module load openfoam ### Create a project directory, Copy the tutorial examples mkdir -p $FOAM_RUN cp -Lr $FOAM_TUTORIALS $FOAM_RUN cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity/cavity # start non-interactive batch job blockMesh icoFoam # (OFF) after the batch job has finished #paraFoam |
MPI Parallel Job
#!/usr/local_rwth/bin/zsh ### Job name #SBATCH --job-name=FOAM_SERIAL ### Request the time you need for execution. The full format is D-HH:MM:SS ### You must at least specify minutes or days and hours and may add or ### leave out any other parameters #SBATCH --time=80 #### Request the memory you need for your job. You can specify this ### in either MB (1024M) or GB (4G). #SBATCH --mem-per-cpu=1850M ### Request 12 processes, all on a single node #SBATCH --nodes=1 #SBATCH --ntasks=12 ### Load the required module files module load TECHNICS module load openfoam ### start the OpenFOAM binary in parallel, cf. $MPIEXEC $FLAGS_MPI_BATCH dieselFoam -parallel |
5. Further Information / Known issues
- Note: Different from most modules, the 'openfoam' module is not unloadable.
You also cannot switch to other version of OpenFOAM in the same shell; use different shells / batch jobs for different versions of OpenFOAM. Known issue on parallel start: As described in http://www.openfoam.org/docs/user/running-applications-parallel.php you have to start you application in parallel by using $MPIEXEC and add the '-parallel' flag. However, if you get this error (known for multiphaseInterFoam):
--> FOAM FATAL ERROR in Foam::findEtcFiles() : could not find mandatory file'controlDict'... try to use 'foamExec' wrapper as workaround (see https://www.cfd-online.com/Forums/openfoam-solving/69561-foam-fatal-error-foam-findetcfile-could-not-find-mandatory-file.html#post322759):
$MPIEXEC $FLAGS_MPI_BATCH foamExec multiphaseInterFoam -parallelIn case this still does not work define a batch job which will definitely run on a single node.
- Known issues on openfoam/3.0.1 (see http://bugs.openfoam.org/view.php?id=2067)
- 'decomposePar' and 'snappyHexMesh' - fixed by rebuild with the very default optimisation flags.
- 'fireFoam' fails with message Floating point exception(core dumped) on 'tutorials/combustion/fireFoam/les/flameSpreadWaterSuppressionPanel' - currently no workaround; however future versions should be fixed, already (see above bug report)
- Known issues on openfoam/4.*
by loading the module the following message is printed:
/usr/local_rwth/sw/openfoam/4.1/intel_16.0.2.181-openmpi_1.10.4/OpenFOAM-4.1/etc/config.sh/aliases:73: bad option: -tThis message is results from incompatibility of ZSH (our default shell) and BASH. There are no limitations in using OpenFOAM known connected to this message; if in doubt try to use the BASH shell.
SIGFPE, coredump, exit 136 and friends
Your calculation is aborted with messages like this:
#0 Foam::error::printStack(Foam::Ostream&) at ??:?#1 Foam::sigFpe::sigHandler(int) at ??:?#2 ? in "/lib64/libc.so.6"#3 void Foam::fv::rotorDiskSource::calculate<Foam::geometricOneField>(Foam::geometricOneField const&, Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&, Foam::Field<Foam::Vector<double> >&, bool, bool) const at ??:?#4 Foam::fv::rotorDiskSource::addSup(Foam::fvMatrix<Foam::Vector<double> >&, int) at ??:?#5 ? at ??:?#6 ? at ??:?#7 __libc_start_main in "/lib64/libc.so.6"#8 ? at ??:?(in LSF you get an exit code 136; you alco may find a file named "core.login18-1.hpc.itc.rwth-aachen.de.271599.11" in your directory). What does that mean?
- The application did run into an Floating point exception which could be a division by 0 or an over/underflow, see
- The root of issue could be
- an error in the application (less likely but possible in OpenFOAM-provided binaries, more likely in self-written/-installed extensions, or
- some numerical issue (round-off error), or
- physically meaningless data set, or a data set with error(s), or
- something else (Learning never stops).
- Possible ways to go:
- check your data set
debug your application (if #3 line in stack dump points to your appilcation), see http://openfoamwiki.net/index.php/HowTo_debugging
try out a different version of Open FOAM (new'er is usually better, sometimes an old good one is good for you)
try out the same version compiled by different compiler, e.g. by GCC:
$ module switch intel gccprior loading the OpenFOAM module in a different shell/batch job!
- set/change FOAM_SIGFPE, FOAM_SETNAN envvars, see https://www.openfoam.com/documentation/guides/latest/api/classFoam_1_1sigFpe.html
- WARNING: YOU MAY GET PHYSICALLY MEANINGLESS RESULTS.
- WARNING: YOU MAY GET PHYSICALLY MEANINGLESS RESULTS.
paraFoam ParaView issues
OpenFOAM is huge. It also brings a boxed version of ParaView, which itself is not that easy to be build. Not all versions of OpenFOAM contain a working version of ParaView. However we offer a standalone installation of ParaView.- Known issues in SLURM/CLAIX18 environment:
- openfoam/5.0: paraFoam dies with " FATAL ERROR: ParaView reader module libraries do not exist". No workaround known; try to use alternative methods to visualize your data sets after loading ParaView in a separete session, see https://www.openfoam.com/documentation/user-guide/paraview.php
- openfoam/6: no boxed ParaView; need to load one from modules. After that 'paraFoam -builtin' dies with '.... paraFoam: line 63: 168444 Segmentation fault (core dumped) paraview $_opt "$@"'
workaround:
export ParaView_GL=system
openfoam/esi-v2006: no boxed ParaView; need to load version paraview/5.6.0 from modules.
swak4foam
The swak4Foam is a library many users of OpenFOAM also want to use. This is a very dusty software; in fact we have seem many issues with it. As the last supporter version of OpenFOAM is v2.4 (!!!) we give no warranty it works on our cluster. However we know that using the below commands you are able to compile swak4foam for OpenFOAM version 6:
$ module switch intel gcc$ module load TECHNICS$ module load openfoam$ hg clone http://hg.code.sf.net/p/openfoam-extend/swak4Foam$ cd swak4Foam$ time ./Allwmake 2>&1 | tee 20190613.02.gcc.txt$ ls -la $FOAM_USER_APPBIN $FOAM_USER_LIBBINNote that the GCC-compiled version of OpenFOAM has been used (due to change to GCC form Intel compiler *PRIOR* loading OpenFOAM).
The Intel compiler cannot compile swak4foam.
Jülich - Aachen OpenFOAM User Group
In 2018 an inofficial Jülich - Aachen OpenFOAM User Group was founded. If you are interested in this user group, please subscribe yourself to the wiki or the mailing list.