# Running AFRT on Poseidon

## Compiling

The Poseidon HPC cluster allows for parallelization of AFRT via the use of OpenMPI.
In order for this to work, the code must be compiled with MPI enabled.

Set the environment variable: OCSSW_MPI

```
$ export OCSSW_MPI=1
```
Build the required third party library, openmpi
```
$ cd $OCSSWROOT/opt/src
$ BuildIt.py
```
Build AFRT
```
$ mkdir $OCSSWROOT/build_mpi
$ cd $OCSSWROOT/build_mpi
$ cmake .. -DBUILD_AHMAD_FRASER=1
$ cd $OCSSWROOT/build_mpi/src/afrt_nc4
$ make -j 20 install
```

## Configuring
Assuming the desire is to generate output compatible with the post processing code used to generate the Rayleigh and aerosol tables for **l2gen**, the following configuration is required:

Define pointers to the various output directories, e.g.:
> odir_phs=/mnt/beegfs/poseidon/hpc/afrt/meris/rayleigh/phs
odir_ocn=/mnt/beegfs/poseidon/hpc/afrt/meris/rayleigh/ocn
odir_rt1=/mnt/beegfs/poseidon/hpc/afrt/meris/rayleigh/rt1
odir_rt2=/mnt/beegfs/poseidon/hpc/afrt/meris/rayleigh/rt2
odir_nc4=/mnt/beegfs/poseidon/hpc/afrt/meris/rayleigh/nc4

The above referenced directories must exist before AFRT is run.

Define the wavelengths to run, e.g.:

> wavelengths=0.4125 0.4425 0.4900 0.5100 0.5600 0.6650 0.68125 0.70875 0.75375 0.76188 0.77875 0.8650 0.8850 0.9000

Some general purpose defaults:
> pol=on\
ocn=on\
foam=off\
rayleigh=on\
gases=on\
\#iref=1 for zero wind speed, iref=2 for non-zero\
\# I think Don modified the code to do the switch as necessary so leaving at 2\
iref=2\
delta_tau= 0.02\
surface_pressure= 1.0\
sea_nr= 1.334\
sea_ni= 0.0\
theta_cell= 2.0\
phi_cell= 2.0

Since **l2gen** currently handles glint with it's own implementation of a Gordon-eque Cox-Munk calculation, we tell AFRT to remove glint from it's calculations:

> glint=on # on means remove glint...yes, counterintuitive\
\# iprob=1 is Gordon's formulation, 2 is traditional Cox-Munk, 3 exists, but don't know what it is...ask Zia\
iprob=1\
rms_slopes= 00 10 15 20 25 30 35 40\
humidities= 30 50 70 75 80 85 90 95

### Rayleigh
For generating the runs needed for the Rayleigh tables:
> aerosols=off\
dphi= 12.0\
dtheta= 4.0\
theta_angles= 0.0 4.0 8.0 12.0 16.0 20.0 24.0 28.0 32.0 36.0 40.0 44.0 48.0 52.0 56.0 60.0 64.0 68.0 72.0 76.0 80.0 84.0 90.0\
\# Rayleigh = no aerosol, so setting optical_depths to zero\
optical_depths= 0.0

>wave_id=1,14 \# change this to reflect the number of wavelengths you wish to process, 1-based indexes into the wavelengths array\
wind_id=1,8\
\# for rayleigh, model makes no sense and aerosol optical depth is zero
model_id=1,1
tau_id=1,1

### Aerosol
For generating the runs needed for the aerosol tables:
> aerosols=off\
dphi= 12.0\
dtheta= 6.0\
theta_angles= 0.0 3.0 9.0 15.0 21.0 27.0 33.0 39.0 45.0 51.0 57.0 63.0 69.0 75.0 81.0 87.0 90.0

>wave_id=1,14\
model_id=1,80\
tau_id=1,9\
\# Only running aerosols with zero wind speed\
wind_id=1,1

## Running
A simple batch script is helpful:

```
#!/bin/bash
PAR=$1
WVL=$2
ulimit -l unlimited
export TMPDIR=/dev/shm
cd /mnt/beegfs/poseidon/hpc/afrt/meris
echo $OCSSWROOT/opt/bin/mpirun afrt_nc4 par=$PAR $WVL
$OCSSWROOT/opt/bin/mpirun afrt_nc4 par=$PAR $WVL
```

### Rayleigh
Rayleigh runs are pretty quick, and for multispectral runs no need to limit the wavelengths for a give run, so I generally just use salloc, like thise e.g.:

```
$ salloc --ntasks=32 --ntasks-per-node=16 ~/batch/afrt.sh rayleigh.par
```

### Aerosol
Aerosol runs take more time because there are runs to be done.  So another useful batch script:

```
#!/bin/bash
#SBATCH --job-name=afrt_wvl # Job name
#SBATCH --output=./logs/afrt_wvl_%j.log   # Standard output and error log
#SBATCH --cpus-per-task=1
#SBATCH --nodes=4
#SBATCH --ntasks=192
#$SBATCH --ntasks-per-node=48

PAR=$1
WVL=`cat wave_id.lst | head -n $SLURM_ARRAY_TASK_ID | tail -n 1`

pwd; hostname; date

#cd /mnt/beegfs/poseidon/hpc/afrt/ocis
ulimit -l unlimited
export TMPDIR=/dev/shm

echo "Running AFRT for wavelength(s)"

export OCSSWROOT=/mnt/beegfs/poseidon/hpc/ocssw-develop
export OCDATAROOT=/mnt/beegfs/poseidon/hpc/ocssw-develop/share
export OCVARROOT=/mnt/beegfs/poseidon/hpc/ocssw-develop/var
export OCSSW_BIN=/mnt/beegfs/poseidon/hpc/ocssw-develop/bin

echo "Processing wavelength #:$WVL"

echo $OCSSWROOT/opt/bin/mpirun afrt_nc4 par=$PAR wave_id="$WVL"
$OCSSWROOT/opt/bin/mpirun $OCSSW_BIN/afrt_nc4 par=$PAR wave_id="$WVL"

date
```

Called using sbatch, e.g.:

```
$ sbatch --array=1-5 batch/afrt_by_wavelength.sh /mnt/beegfs/poseidon/hpc/afrt/meris/aerosol.par
```

The script uses a file called wave_id.lst, which just provides the wavelength ids in small batches, e.g.:

```
$ cat ../wave_id.lst
1,3
4,6
7,9
10,12
13,14
```