\[\def\RR{\bf R} \def\bold#1{\bf #1} \def\bs#1{\boldsymbol #1} \def\uvec#1{\hat{\boldsymbol #1}}\]

To run mathematica in CCV one needs to generate a script file with an extension .wls. There are certain notations and packages that cannot be used in a script file such as

• Symbolize
• definitions such as $u_i$, $\sigma_{ij}$
• Superscript, subscript, hat
• Haneesh packages
• ToElementMesh to name a few. Therefore, the notebook file has to be modified to replace these notations with anything else that is acceptable as long as it is in either greek letters or characters. To do so, Haneesh developed a a mathematica notebook file which searches symbols in the notebook and replaces them with alternate names. For example, $N^{(a)}$ can be replaced by Nsupa and so on.

Replacing symbols and notation

The file to do this operation automatically can be found here. The notebook SymbolReplace.nb basically opens your files and searches for symbols such as $N^{(a)}$ and replaces it with Nsupa. Data structure of the file is as follows

• The symbols need to be defined in the array Symbs

nb = NotebookOpen["/Users/kaushikvijaykumar/Dropbox \ (Brown)/3D_elasticity/script_test/3Delasticity_Tets_Mar_23_notched_\
beam_damage_3PB.nb"]

opens the notebook file that needs to be modified

* The module CreateComps searches for all combination of indicial notations such as, $u_{i\cdot}$, $\sigma_{i\cdot j\cdot}$ and replaces it with $u[[i]]$ and $\sigma[[i,j]]$

Once this is done the modified ‘.nb’ should be benchmarked using a simple and computationally tractable problem to make sure that the modifications are consistent.

Exporting and importing mesh

As mentioned earlier the mesh cannot be generated in a script file on ccv, therefore generate the mesh using the notebook file on your desktop and export it to the hard disk using

Export["meshdata.txt", NumberForm[#, ExponentFunction -> (Null &)] & /@ List @@ Omegasuph,"String"];

where “meshdata.txt” is the exported mesh file and Omegasuph is the generated mesh. This can be read back into the mathematica file using

Generating script files from notebook files

To create a mathematica script file from the notebook file, all the cells should be converted to initialization cells and then the notebook can be saved as as .wls file.

Running mathematica accross nodes in ccv

A sample batch file for ccv, a test script file and the associated mesh file can be found here. The file description are as follows

• Batch file is batch.script
• Mathematica script file is 3PB_rescale_11k_elements_6k_neq_2.wls
• Input mesh is meshdata.txt

In the batch file the command

srun hostname > nodefile
export NODEFILE=nodefile

srun is a SLURM command, which associates the number of nodes and number cores and the export command exports the nodes to the file nodefile. For example, if you have requested for 2 nodes with 4 processors each, then nodeifle will contain Node1 and Node2 4 times each. Indicating that you have 4 cores of node1 and 4 cores of node2 at your disposal for your job. The rest of the commands in the batch can be read on the ccv website. To run the mathematica script file across nodes copy paste the following in the script file

(*get association of resources, name of local host and remove local host from available resources*)
hosts = Counts[ReadList[Environment["NODEFILE"], "String"]]
local = First[StringSplit[Environment["HOSTNAME"],"."]]
hosts[local]--;
(*launch subkernels and connect them to the controlling Wolfram Kernel*)
Needs["SubKernels`RemoteKernels`"];
Map[If[hosts[#] > 0, LaunchKernels[RemoteMachine[#, "ssh -x -f -l `3` `1` /gpfs/runtime/opt/mathematica/11.0/Executables/wolfram -wstp -linkmode Connect `4` -linkname '`2`' -subkernel -noinit", hosts[#]]]]&, Keys[hosts]];

The variable hosts counts the number nodes and the associated cores from the node file and stores it. hosts[local]-- makes the node1-core1 as the master kernel. I don’t fully understand the next set of commands but in a lay mans language it launches RemoteKernels from path given in the code. Finally, save the notebook file as .wls file and run it using the batch script on ccv.

Executing files on ccv

Open the batch file sbatch.script and

• Set total time that you expect within which your job will be completed using the command #SBATCH --time=24:00:00, which sets the total time of job to be 24 hrs.
• Provide a job name, which basically indicates the name of the job that you are running on ccv. The job name is different from your file. For example, the command #SBATCH -J mathematica-sample indicates that your job name is mathematica-sample.
• Provide the standard ouptut and error file names. For example, in the command #SBATCH -o sample.out the output file is sample.out and in this file all the print statements from the mathematica script file will be executed. The command #SBATCH -e sample.err is the standard error file which will indicate if there are any compilation errors.
• Specificy the number of nodes using #SBATCH --nodes=4 and the number of processors using #SBATCH --ntasks-per-node=16. The number of nodes will depend on the availablility on the cluster, which can be checked using the command nodes-v. The number processors will depend on the node you want to run it on.
• Load the mathematica module using module load mathematica/11.0
• The script file can be executed using wolfram -script 3PB_rescale_11k_elements_6k_neq_2.wls

Rescale

All you need to do is remove the ccv commands in the .wls given in the previous section and add

Needs["Rescale'"]

to the .wls file. The command to execute the script file is the same as earlier wolfram -script 3PB_rescale_11k_elements_6k_neq_2.wls.