Post-processing Workflows Using

Data Grids to Support Hydrologic

Modeling
Bakinam T. Essawy and Jonathan L. Goodall
Department of Civil and Environmental Engineering
University of Virginia

3rd CUAHSI Conference on Hydroinformatics

July 17, 2015

VIC Output data set

Variable Infiltration Capacity (VIC)
Macro-scale Hydrologic Model

VIC data set

http://boto.ocean.washington.edu/s
tory/show/45

VIC Output data set on the

iRODS server

Example for a flux file. Fluxes_x_y:

x = latitude, y = longitude
flux files contain information about
moisture and energy fluxes for each
time step for the three layers of soil
(Top, Middle, and Deep).

The VIC Model

VIC = Variable Infiltration Capacity;
A regional-scale land surface
hydrology model
VIC developed at UWashington and
Princeton; applied worldwide
Spatial resolution: 1/8-degree grid
cell
Three layers of soil:
top layer (Layer 0, 0-10cm)
mid layer (Layer 1, 10-30cm)
lower layer (Layer 2, 30-100cm)

Source : Gao et al. (2009)

The County-level population data

extracted from Terra Populus Website

Integrated Rule-Oriented Data System

(iRODS)
The iRODS-enabled Data Federation Consortium
(DFC) is an NSF project that provides support for
both federation of resources and services.
This work is funded the by DFC project, and uses
a DFC data grid for storage and long term access
to the stored datasets over heterogeneous
resources.
The DFC data grid also supports sharing of
workflows that enable the reproducibility of the
model results

Workflow Structured Object (WSO)

Within the iRODS data grid, a Workflow
Structured Object (WSO) enables the execution
of a workflow, while capturing provenance
information and archiving results.
The workflow, the input files, and the output
files can be shared.
The workflow can be re-executed with new input
files and versions of the output file are
automatically saved.

Objectives
Demonstrating how different data transfer
approaches can be used for connecting cyberinfrastructure systems developed by different
groups.
Demonstrate how iRODS can provide federation
across data grids.

Objectives
Using the AWS (Amazon Web Services) for
computing, and how public repositories like SEAD
allow sharing and uniquely identifying data and
modeling resources used within analyses.
We are trying to reach an approach for model
reproducibility, where a scientist can easily share
his model, input and output in an easy way so
others can benefit from it.

Main components and data flow in the

post-processing system

WSO files used by WSO for creating the visualization

Workflow file

Shell Script

Python Scripts

Parameter File

Visualization

Two main directories for storing all files required by

the WSO on the iRODS server
The location of the mounted
WSO on the iRODS server
The mounted collection
The parameter file, the generated run file
and the output RunDir for each time the
run file is accessed

Component of one of the runDir

associated with the WSO like the
staged data in or out, the cvs files
output from python scripts, and the
stdout

The location were the shell script

and the python scripts located on
the iRODS server

The execution of the WSO installed on the

hydrology grid from the client machine
The user log in to
client machine where
the icommands are
installed on
ils to list all the collection
under the path:
/hydrology/home/bakinam
icd to change collection
were the WSO files are
located
Listing the mounted
collection

Running the WSO through

iget command to run the
generated .run file. Output
Message indicates that the
WSO has been executed
successfully

Conclusion
Reproducible data visualizations on large
hydrological data collections
Using strong and weakFederation of data across
communities (e.g., TerraPop interoperability
example)
Publishing data along with workflow-produced
metadata (e.g., SEAD interoperability example)
using unique Identifier.

Future Plans
Swap SEAD with Hydroshare to share my
datasets and create a resource type from my
WSO.

Questions
Bakinam T. Essawy
Department of Civil and Environmental Engineering
University of Virginia
bte2rn@virginia.edu