Date: June 1-2, 2014

Donald Willcox

The parallel version of VH-1 uses netcdf, and configuring mpich and netcdf and compiling the parallel version requires installing the following libraries. On Fedora, I installed all of these via yum install <library> and they are likely available via apt-get if your Linux distribution uses Synaptic.

• mpich, mpich-devel
• netcdf, netcdf-devel
• netcdf-mpich, netcdf-mpich-devel
• netcdf-fortran, netcdf-fortran-devel, netcdf-fortran-mpich, netcdf-fortran-mpich-devel

If for some reason you are unsuccessful in installing netcdf via your package manager, you can install it from source. I had to do this for a remote machine without root access. I'm assuming mpich and the gcc/gfortran build system are already installed.

You can get the source for v.4.2 of the NetCDF Library from Unidata's links as follows:

• http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-4.2.tar.gz
• ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-fortran-4.2.tar.gz
• https://github.com/Unidata/netcdf-cxx4/archive/v4.2.1.tar.gz

It depends on hdf5, szip, and zlib libraries, which you can get from:

• http://www.hdfgroup.org/ftp/lib-external/szip/2.1/src/szip-2.1.tar.gz
• http://zlib.net/zlib-1.2.8.tar.gz
• http://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.12/src/hdf5-1.8.12.tar.gz

First, build/install zlib, szip, and hdf5 in that order. Then build/install netcdf-4.2, netcdf-cxx4, and netcdf-fortran-4.2 in that order and you should be good to go. You'll have to set environment variables along the way such as CFLAGS, CPPFLAGS and LDFLAGS to point to the include and lib directories containing the library files you install so that building the library that depends on the previous one can find the appropriate files.

I'll detail that process in the following since getting all this sorted took me a bit of finagling!

I extract the .tar.gz files and then…

For szip and zlib, I used:

./configure –prefix=/home/dwillcox/local (and then make, make check, make install)

Then I set the flags as follows:

export CFLAGS=-I/home/dwillcox/local/include

export CPPFLAGS=-I/home/dwillcox/local/include

export LDFLAGS=-L/home/dwillcox/local/lib

That lets hdf5 find those libraries. So in the hdf5 directory I used:

./configure –enable-fortran –prefix=/home/dwillcox/local/hdf5

and then ran make, make check, and then make install. I recommend running make check before make install because if something is configured improperly, it will fail tests. This happened to me several times before I finally got everything right so it's a nice sanity check.

Then I added the hdf5 location to the flags as follows:

export CFLAGS=-I/home/dwillcox/local/include -I/home/dwillcox/local/hdf5/include

export CPPFLAGS=-I/home/dwillcox/local/include -I/home/dwillcox/local/hdf5/include

export LDFLAGS=-L/home/dwillcox/local/lib -L/home/dwillcox/local/hdf5/lib

Now you can build the netcdf-4.2 library. I used:

./configure –prefix=/home/dwillcox/local/netcdf –disable-dap

The –disable-dap was important because the first time I built and ran make check, it failed a test due to this feature not working properly.

Then I did make, make check, and make install as usual.

Adding the netcdf locations to the flags is necessary for installing the cxx and fortran versions of netcdf:

export CFLAGS=-I/home/dwillcox/local/include -I/home/dwillcox/local/hdf5/include -I/home/dwillcox/local/netcdf/include

export CPPFLAGS=-I/home/dwillcox/local/include -I/home/dwillcox/local/hdf5/include -I/home/dwillcox/local/netcdf/include

export LDFLAGS=-L/home/dwillcox/local/lib -L/home/dwillcox/local/hdf5/lib -L/home/dwillcox/local/netcdf/lib

With this done, I went to the netcdf-cxx and netcdf-fortran directories and did the following for each of them:

./configure –prefix=/home/dwillcox/local/netcdf, then make, make check, and make install

Finally, I set the following environment variable so building ScientificPython would go smoothly in the “Configuring YT” section!

export NETCDF_PREFIX=/home/dwillcox/local/netcdf

The next step is to modify the Makefile in VH1/src/Parallel because it expects to be compiled with Intel's ifort and I'm using gfortran and also because I need to compile mergeslabs.f90, which takes the multiple netcdf files output per timestep and merges them into one. The first modification is to set these flags as follows:

F90 = mpif90

FFLAGS = -c -O3 -Wall -I/usr/include -I/usr/lib64/gfortran/modules

LDR= mpif90

LDRFLAGS=

LIBS= -L/usr/lib64/mpich/lib -lnetcdff -lnetcdf

Then following the line defining VHOBJS, I added a line as follows:

MERGEOBJS = mergeslabs.o

Now, right before the line declaring the clean: target, I declare a target mlabs-mpi which is the binary file I generate by compiling mergeslabs.f90:

mlabs-mpi: $(MERGEOBJS) $(LDR) $(LDRFLAGS) -o mlabs-mpi $(MERGEOBJS) $(LIBS); mv mlabs-mpi ../../output/.

And finally, declare the dependency for mergeslabs.o by adding the following to the end of the Makefile:

mergeslabs.o: mergeslabs.f90

Now, running the Sod shock tube problem requires setting up the geometry and mpi settings. In the Parallel version, the grid geometry specification and initialization is done in init.f90, while specifying how many grid zones to use and how to allocate this to MPI is done in zonemod.f90. So in zonemod.f90 I set imax=10, jmax=10, kmax=10 for a 10x10x10 3D cartesian grid (specified in init.f90). Then I set 2 MPI tasks for the y-dimension (pey=2) and 2 MPI tasks for the z-dimension (pez=2). This means I must run on 2*2=4 processes.

Now make will put the executable vh1-mpi in the VH1 directory and make mlabs-mpi will put the executable mlabs-mpi in the VH1/output directory. Run vh1-mpi using mpiexec -np 4 ./vh1-mpi and then go to VH1/output and run mlabs-mpi. Voila, I've got output for the default 3D Sod shock tube problem and now comes visualizing the data…