Updated 2023-04-28

Comsol

License

Comsol on PACE uses the Georgia Tech license, for which an annual access fee is required per user.

Run Comsol on the Cluster - Batch Mode

Overview

  • There are a couple general parts to the Comsol Workflow:
    • Make model in gui / import model
    • Solve interactively (gui) or in batch mode
    • Analyze results
  • This guide will focus on how to solve models using batch mode. This is especially helpful if you want to solve multiple models at once
  • Important: models imported from windows may have binarys that dont work on the cluster. Make sure models are .mph not .mphbin, or just make them on the cluster using comsol interactively.

Setup Storage Directory for Comsol

By default, Comsol places configuration and temporary files in a hidden directory in your home directory named ~/.comsol. However, Comsol files can be quite large, and your home directory has a relatively small storage quota. Hence, we strongly recommend that you create a symlink named ~/.comsol that points to your project space, which has a much larger storage quota.

If you don't already have a ~/.comsol directory:

  1. Create a .comsol directory in your project space: mkdir ~/p-<pi-username>-<number>/.comsol
  2. Create a symlink from your project space to your home directory: ln -s ~/p-<pi-username>-<number>/.comsol ~/.comsol, e.g., ln -s ~/p-gburdell3-0/.comsol ~/.comsol (Phoenix) or ln -s ~/data/.comsol ~/.comsol (Hive)

If you do have an existing ~/.comsol directory:

  1. Move your .comsol directory to your project space: mv ~/.comsol ~/p-<pi-username>-<number>/ (Phoenix, replacing with the username of the faculty member who provides your storage and with the volume, usually 0) or mv ~/.comsol ~/data/ (Hive)
  2. Create a symlink from your project space to your home directory: ln -s ~/p-<pi-username>-<number>/.comsol ~/.comsol (Phoenix) or ln -s ~/data/.comsol ~/.comsol (Hive)

Tips and Warnings

Warning

We strongly recommend you create models on the cluster. If a model is created in windows, the binary of the model will not work on the cluster (linux).

  • If you are a researcher use the -research versions of Comsol, otherwise for things like class, use the non-research version.
  • Make sure you load matlab and then comsol in your SBATCH Script, using
    module load <matlab version> <comsol version>. Find available versions with module avail comsol.

Run Multithreaded Batch Job

  • Add the following line to your SBATCH script (after you have loaded the comsol module) to run a comsol on multiple cores:
  • comsol batch -np 8 -inputfile <input.mph> -outputfile <output_name.mph>

Important

The number after the -np flag (number of processors) must equal the number you requested in the SBATCH script

Run Multiple Models

  • One option is to use a job array.
  • Another option is to supply a script that lists multiple jobs to be run, which will be explained below.
  • When logged into the cluster, create a plain file called COMSOL_BATCH_COMMANDS.bat (you can name it whatever you want, just make sure its .bat). Open the file in a text editor such as vim (vim COMSOL_BATCH_COMMANDS.bat).
  • With the file open, basically you just have to list the run command from above for every model:
#Contents of the .bat File
comsol batch -np 8 -inputfile <model1.mph> -outputfile <output_name.mph>
comsol batch -np 8 -inputfile <model2.mph> -outputfile <output_name.mph>
comsol batch -np 8 -inputfile <model3.mph> -outputfile <output_name.mph>
  • Then, in the SBATCH script instead of writing out the run command (the one that starts with comsol batch), include the name of the .bat file without the .bat, for example just write COMSOL_BATCH_COMMANDS
  • Make sure the .bat file and the model files are in the same dir as your SBATCH script, if you are using $SLURM_SUBMIT_DIR in your script.
  • Since you are running multiple models in one job, you will have to increase the walltime of your job

Walkthrough: Run Comsol in Batch Mode

  • This walkthrough will use an example .mph model, cold_water_glass.mph. The model is an example provided by Comsol, and more detail on the model can be found on their website here
  • After logging in, You can transfer the files to your account on the cluster to follow along. The file transfer guide may be helpful.

Part 1: The SBATCH Script

#!/bin/bash
#SBATCH -J SBATCHcomsolTest
#SBATCH -A [Account]
#SBATCH -N 1 --ntasks-per-node=8
#SBATCH -l pmem=8gb     
#SBATCH -t 10
#SBATCH -q inferno
#SBATCH -j oe
#SBATCH -o Report-%j.out
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=gburdell3@gatech.edu
>>>>>>> Stashed changes

cd $SLURM_SUBMIT_DIR
module load comsol/5.3a-research
comsol batch -np 8 -inputfile cold_water_glass.mph -outputfile cold_water_glass_solved.mph
  • The #SBATCH directives are standard, requesting just 1 minute of walltime and 1 node with 8 cores.
  • $SLURM_SUBMIT_DIR is simply a variable that represents the directory you submit the SBATCH script from. Make sure the .mph comsol model you want to run (in this case, cold_water_glass.mph) is in the same directory you put the SBATCH script.
  • module load comsol/5.3a-research loads the 5.3a version of comsol. To see what comsol versions are available, run module avail comsol, and load the one you want
  • comsol batch runs comsol
  • For multiple cpus (parallel), make sure the number of processors you request in the directives (top) part of the script is equal to the number you specify in the -np part of the comsol batch line

Part 2: Submit Job and Check Status

  • Make sure you're in the dir that contains the SBATCH Script <<<<<<< Updated upstream
  • Submit as normal, with sbatch <SBATCH script name>. In this case sbatch comsol.SBATCH
  • Submit as normal, with salloc <SBATCH script name>. In this case salloc comsol.SBATCH
  • Check job status with squeue -t 22182721, replacing the number with the job id returned after running qsub
  • You can delete the job with scancel 22182721 , again replacing the number with the jobid returned after running qsub

Part 3: Collecting Results

  • In the directory where you submitted the SBATCH script, you should see all the generated output files, including the solved model .mph file. Use cat comsolTest.out to view information on the completed job, which should look like:
---------- Current Progress: 100 % - Assembling matrices
Memory: 1046/1118 7077/7081
1726      119.82    0.087925     3567 1732 3567     1     1      2  3.4e-11  5.4e-16
1727       119.9    0.087925     3569 1733 3569     1     1      2  3.3e-11  6.8e-16
---------- Current Progress: 100 % - Assembling sparsity pattern
Memory: 957/1118 7046/7081
1728      119.99    0.087925     3571 1734 3571     1     1      2  2.8e-11  6.3e-16
---------- Current Progress: 100 % - Assembling matrices
Memory: 989/1118 7081/7081
   -         120           - out
1729      120.08    0.087925     3573 1735 3573     1     1      2    6e-11  5.3e-16
Time-stepping completed.
---------- Current Progress: 100 % -
Memory: 957/1118 7047/7081
Solution time: 559 s. (9 minutes, 19 seconds)
Physical memory: 1.12 GB
Virtual memory: 7.08 GB
Ended at 14-Sep-2018 10:42:08.
----- Time-Dependent Solver 1 in Study 1/Solution 1 (sol1) -------------------->
Run time: 563 s.
Saving model: /gpfs/pace2/project/pf1/shollister7/comsol/cold_water_glass_solved.mph
Save time: 0 s.
Total time: 572 s.
---------- Current Progress: 100 % - Done
---------------------------------------
Begin SBATCH Epilogue Fri Sep 14 10:42:09 EDT 2018
Job ID:     22409572.shared-sched.pace.gatech.edu
User ID:    shollister7
Job name:   comsolTest
Resources:  -N 1 --ntasks-per-node=4, -N 1 --ntasks-per-node=8,--mem-per-cpu=8G, -t 00:16:00
Rsrc Used:  cput=01:04:15,energy_used=0,mem=1080752kb,vmem=7572812kb, -t 00:09:41
Queue:      inferno
  • After the result files are produced, you can move the files off the cluster.
  • To open the solved model in the Comsol postprocessor, see the interactive guide below.
  • Congratulations! You successfully ran Comsol in batch mode on the cluster.

Common mistake

  • If you got strange java errors on running Comsol on PACE, you can double check if ~/.comsol points to ~/data/.comsol and there is no recursive symbolic link exists.

  • "why the same input file produces the result on windows but not Linux?". The answer could be that you should initialize the results by right click "Study"--->"Get Initial Value".

Comsol Interactively

Overview

  • Running Comsol interactively is very similar to any other interactive program. First you must setup the Open OnDemand session, then open comsol from the terminal
  • It is highly recommended to create models on the cluster in an interactive session to avoid trouble with binarys that may have been created on windows.

Set up Interactive (Open OnDemand) Session

  • Open the Open OnDemand Guide and select the version of Open OnDemand you want to use. Note: You must be connected to the Georgia Tech VPN to access Open OnDemand
  • Open the link and navigate to the top of the page and select Slurm Interactive Apps > Interactive Desktop
  • Enter your username and choose whether you want to use inferno (paid) or embers
  • Adjust the rest of the settings to your needs and then click Launch

Start Comsol on Open OnDemand

  • Open Terminal in the Open OnDemand Virutal Desktop
  • All commands here on will be typed in terminal in Open OnDemand
  • To see available versions of Comsol run
module avail comsol
  • Load comsol module and start comsol
module load <version of comsol>  #example: module load comsol/5.3a-research

  • To start comsol, run comsol -np <num of processors you requested>
  • It might take a while for Comsol to load