Professional Documents
Culture Documents
CAPTURE ENHANCE
STORE
Developed by GXS
Under the DC20B Enhancing Filtration Membrane Fouling
Data Collection for Water Research Project
June 2011 - March 2012
FoulantProfiler
User Manual
Introduction
Purpose
Confocal Laser Scanning Microscopes (CLSM) are used by materials scientists, biologists and
environmentalists for analysis of biological and organic samples. Filtration Membrane
Researchers use CLSM extensively for investigation of membrane fouling and evaluation of
the effectiveness of new anti-fouling surface treatments.
A Confocal Laser Scanning Microscope typically works together with the LAS-AF acquisition
software running on a workstation linked to the microscope. The acquisition software
provides control and configuration of the microscope equipment and the scanning process.
After scanning, the acquisition software outputs the data as a stack of images with different
channels of color and some experimental data (instrument data). It is common for these raw
data images to contain different types of noise which may result from a number of factors
such as flaws in the equipment, poor operational skills or poor sample preparation. Further
image processing is usually necessary to reduce or eliminate the noise contamination, but
this processing is tedious and requires specialist computer vision knowledge and skills for
success. Furthermore, the acquisition software does not provide any quantitative
measurements of foulant configuration.
FoulantProfiler, developed by Deakin University, is an ANDS Funded open-source image
processing/analysis software application that works with the acquisition software LAS-AF for
Confocal Laser Scanning Microscopes (CLSM) to capture single or multiple channels of
fluorescent images from filtration membrane contaminated by different organic foulants,
enhance the images and profile the foulants, and store those data in a central research data
repository.
Deakins development of the FoulantProfiler software application was based on the capture
of images from CLSM scans of filtration membranes, generated within Deakins biological
research centre, BioDeakin.
FoulantProfiler is designed to enable filtration membrane scientists and water researchers
or environmentalists with limited training and knowledge on computer vision or
programming to obtain better quality of images for improved visualization and to
quantitatively profile the foulants automatically. FoulantProfiler also aims to foster faster
and easier capture and storage of raw research data, processed data and metadata in the
centralised data storage.
In keeping with ANDS Funding guidelines and Open-Source Software principles, the
FoulantProfiler software application has been designed to easily allow customisation and
further development of the application to support different technical equipment and
acquisition software. As a result, it could be easily adaptable to a wide range of research
data capture environments, image analysis and quantification needs.
Page 1
FoulantProfiler
User Manual
Project Background
The Federal Government, via the Australian National Data Service (ANDS), has funded the
creation and development of the Australian Research Data Commons (ARDC) infrastructure,
supporting the discovery of, and access to, research data held in Australian universities,
publicly funded research agencies and government organisations for the use of research.
Under this program, funding is available for Data Capture projects which create
infrastructure within institutions to collect and manage data, and to improve the way
metadata about it is managed.
Deakin University was been awarded funding under the Data Capture initiative for the
ANDS Enhancing Filtration Membrane Data Collection for Water Research (DC20A)
project, for completion during the period June 2011 June 2012.
The primary objective of this project was to develop infrastructure within Deakins Confocal
Microscope Facility to support the ongoing data capture and management of filtration
membrane fouling data and metadata, supporting the discovery, reuse and sharing of this
research data by other researchers into the future.
This project was managed and implemented collaboratively with two other Deakin ANDS
Projects Seeding the Commons (SC19) and Crystal Orientation Data Collection for
Conversion to a General Data Type (DC20A). Collectively, these projects implemented the
ANDS funded, open-source system ReDBox as a solution for research metadata capture and
storage in Deakins Confocal Microscope Facility and Electron Microscope Facility.
This specific project developed a customised open-source system, FoulantProfiler, for
capture, profiling and storage of research data and metadata within the Confocal
Microscope Facility. The research data and metadata are linked via identification using the
Deakin Project ID.
In developing FoulantProfiler, this project re-used architecture and software components of
the CRYSTO software application developed by the Crystal Orientation Data Collection for
Conversion to a General Data Type (DC20A) project. This ensured an approach and
architecture for data capture that was consistent across both projects, ensured maximum
reuse and provided efficiencies in the development of the software applications.
This project also utilised infrastructure developed by the ANDS Seeding the Commons
project for metadata storage, harvest and discovery.
Page 2
FoulantProfiler
User Manual
from the original data files, as output by the acquisition software package. Further ObjectLevel metadata fields can be specified on commencement of a data capture session, if
required. The Object-Level metadata is stored in a text file alongside the data files in the
central data repository, allowing for simple searching and filtering using the metadata.
Page 3
FoulantProfiler
User Manual
System Architecture and Design
Target Audience:
System Architecture
The following diagram shows the basic systems architecture required to support the
software application.
FoulantProfiler System Architecture
Local PC Workstation connected to Microscope
FoulantProfiler
Data Capture
Data
Storage
Processed Data
RESEARCH
DATA
STORE
Object -Level
Metadata
Server
Design Principles
The FoulantProfiler software application is based on the principle that all the data
files captured across the life of a Research Project will comprise the Research Data
Collections for that Project. A Research Project may generate a single or multiple
Research Data Collections.
The FoulantProfiler software application is based on the principle that all research
data collections will be stored in a centralised research data store, organised by
Project IDs. It is assumed that a centralised, networked and accessible data store will
be available for the storage of the data files created by the software application.
(Deakins Enhancing Filtration Membrane Fouling Data Collection For Water
Research project piloted the use of Deakins ISILON shares, which provide a secure,
supported, backed-up facility for the storage of research data collections. The ISILON
shares are web-enabled, supporting direct or managed access from Research Data
Australia (RDA) via the Deakin Research Online portal (DRO) to data collections
which have been released for open access.)
Page 4
FoulantProfiler
User Manual
The FoulantProfiler software application is not a metadata store and does not
provide capture, storage or harvest of Collection-Level metadata for research data
collections. FoulantProfiler is designed to be used with a research metadata store
such as ReDBox for the capture and storage of Collection-Level metadata.
(At Deakin University, ReDBox is implemented for the definition and storage of
Collection-Level metadata for research data collections. The research data files
stored in the Research Data Repository are linked to the Collection-Level metadata
via the Project Identifier (which acts as the Collection Identifier) within the data
repository, and via the via the URL of the collection as stored in the metadata.)
The following diagram represents how the software application could integrate with
an organisations metadata store in order to link the research data with the
collection metadata.
FoulantProfiler Architecture
Integration with a Metadata Store system
Local PC Workstation connected to Microscope
FoulantProfiler
Original Data Files
Metadata System
Metadata
Definition
Data Capture
Analysis
Data
Storage
Confocal Laser
Scanning Microscope
Networked System
Processed Data
Collection
Metadata
RESEARCH
DATA
STORE
RESEARCH
METADATA
STORE
Object-Level
Metadata
Server
System Components
The FoulantProfiler system has been developed with 3 primary functions, as follows:
Data Capture
Data Storage
These components have been developed as separate open-source software modules. This
modular nature supports the development of alternative data capture, data storage or
image conversion components to suit different equipment or research environments.
(Note that the Data Capture and Data Storage components re-use the software components
developed for the CRYSTO software application.)
Page 5
FoulantProfiler
User Manual
The components are described in detail below:
Data Capture
The Data Capture process is primarily a monitoring process that runs in the background
during a microscope scanning and research data capture session. The process runs on the
local PC workstation connected with the microscope, in parallel with the existing microscope
acquisition software application, (e.g., LAS-AF). The Data Capture process detects when any
research data files are created by the acquisition software and written to the specified data
folder on the local PC workstation.
It is important to note that acquisition software applications are typically commerciallyavailable, proprietary packages that do not allow customisation or direct integration with
the program code or output files created. Output files are typically generated in proprietary
format that cannot be changed. As a result, the FoulantProfiler software application has
been designed to operate separately from the acquisition process and to maintain the
integrity of the original data files.
The Data Capture process provides a simple data-entry solution for the identification of the
Project (and therefore the Collection) to which the research data captured during a session
will belong. The researcher can change the Project Identifier during a session by Starting a
New Session to allow for the scenario where a researcher captures data for different
Projects across the course of a data capture session.
The Data Capture process also provides a simple solution for the entry of Object-Level
metadata, such as Project ID, material type, fluorescent dye types, etc., related to specific
data-files. Required metadata fields can be specified, requiring the researcher to enter the
metadata at the commencement of each data capture session.
The Data Capture process then monitors the data (both images and instrumental data) with
certain formats (e.g., .Tiff, .JPG, .xml or .xsl) created by the acquisition software.
Image Processing
The Image Processing module runs between Data Capture and Data Storage, taking the data
files generated by the acquisition software application and automatically processing the
images, including transforming the original color space from RGB to HSV, enhancing the
quality of the images obtained and profiling the foulants, before finally outputting the
results to a new data file.
After the acquisition software is closed, the software automatically reads the created data
and passes the data to the Image Processing software component. The image analysis
software component automatically transforms the images RGB color space to HSV space,
enhance the qualities of the images in HSV space by denoising and filtering, then segment
and distinguish different foulants and compute the foulant configurations in 3-dimensional
volume within the fouling layer.
Page 6
FoulantProfiler
User Manual
In keeping with ANDS Funding requirements and Open-Source Systems development
principles, users are encouraged to further develop the software application by customising
or developing their own image processing/analysis algorithms and sharing them with the
Open-Source community. Alternatively, the FoulantProfiler software application could also
be installed without the Image Processing module to provide capture and storage capability
only.
The current version of the Image Processing module includes the extraction of some ObjectLevel metadata fields from the original data files. These metadata fields typically describe
parameters set within the acquisition software application for control of the microscope
scanning process or imaging. The metadata fields extracted will be dependent upon the
specific acquisition software applications.
Data Storage
The Data Storage process is simply a file management process that automatically stores the
output-files from the Image Processing module in a specified central file store.
Data files are stored according to the specified Project, identified by the Project Identifier,
with a separate folder created for each data capture session, identified by the Date and Time
of the session. All the data files captured during the session are stored within this session
folder. (The Data Capture process includes a simple method for changing the Project
Identifier during a session in the scenario where a researcher may capture data for different
Projects across the course of a data capture session.)
The transfer will include the raw data produced by instrument, exported data from the
acquisition software and processed data. The Data Storage process also stores any ObjectLevel metadata entered by the researcher or extracted from the data-files as a text-file
alongside the research data files in the data store. This file will be stored in the session
folder with the name METADATA.ini. This approach supports the development of filtering
and searching mechanisms to support discovery of research data files for re-use via the
meta-data.
The captured research data will also remain on local drive of the PC workstation connected
to the microscope equipment. Moving or deleting data from the local drive will not affect
the data already stored on network storage.
Page 7
FoulantProfiler
User Manual
Installation and Configuration
Target Audience:
System Administrator
System Requirements
Page 8
FoulantProfiler
User Manual
A dedicated network storage service is required for data storage via Server Message
Block (SMB), also known as Common Internet File System (CIFS) Protocol.
Project folders and project definition files must be setup on the storage service prior
the research activity, as follows:
o By default, each research project for which data may be captured must have
a corresponding folder already created in this storage service. The project
folders should be named with the Project Identifier, thereby allowing the
data storage process to correctly identify the location for storage of the data
files for the Project.
o If required, you may specify a particular sub-folder as the default path for
research data transfer. All research data will go into this sub-folder within
the Project folder. This approach can support the provision of both openaccess and restricted data storage areas within each Project folder on the
research data store.
o The software application will look for a project definition in your appointed
sub-folder. The definition needs to be stored in a file named PROJECT.ini.
Within this file these definition items are expected.
Item
Description
[project] project.name
[project] project.owner
[metadata]
All research data acquired during a research session, as well as the technical
metadata, will be organised into one single folder for that session. These are
named Session Folders on the network storage. Each Session Folder will be
assigned a unique name representing the timestamp when the activity conducted.
The session folders will be created automatically.
When setup has been completed as described above, the network storage is ready
to receive research data.
A configuration file will be loaded from path config\default.ini by default. You may
store your configuration file anywhere on local drive, then pass on to this software
via command line parameters.
Page 9
FoulantProfiler
User Manual
Description
sys.encryption.salt
sys.network.enabled
sys.network.smb.service
sys.network.smb.username
sys.network.smb.password
Encrypted Password
sys.network.smb.domain
sys.network.smb.storage
sys.network.checking.interval
sys.utils.smbclient.program
Page 10
FoulantProfiler
User Manual
In keeping with Open-Source Systems development principles, users are encouraged to
further develop the software application by customising or developing their own image
processing processes and sharing them with the Open-Source community.
Page 11
FoulantProfiler
User Manual
Software Usage
Target Audience:
Helpsheets
A set of helpsheets have been created for the FoulantProfiler software application to
provide users with a step-by-step guide to the basic use of the system. These helpsheets are
available from:
http://code.google.com/p/ands-enhancing-filtration-membrane/w/list
Page 12
FoulantProfiler
User Manual
Export data
When your research activity is completed, all data created may be exported. This is an
automated procedure and no interaction from the user is required.
The data exporting is configured in your instrument and acquisition software settings.
Processing/Analysis data
If required, your data will be processed by image processing/analysis scripts. This is an
automated procedure and no interaction from the user is required. The processing of data is
configured in your instrument and acquisition software settings.
Your research data will also remain on local drive of the PC workstation connected to the
microscope equipment. Moving or deleting data from the local drive will not affect the data
already stored on network storage.
For the path of your research data, please consult your system administrator.
When data transfer is completed, the FoulantProfiler software application will idle and get
ready for the next research activity.
For a step-by-step guide to the export, processing and storage, refer to the Helpsheet:
Page 13
FoulantProfiler
User Manual
System Processes and Workflows
Target Audience:
Page 14
FoulantProfiler
User Manual
A session will also record a list of data files to be transferred to network storage, including
raw research data and any data files generated after the conversion process.
A Session exchange file is created during the activity. It is then uploaded to the Session
folder on network storage with a name METADATA.ini, together with all data files
acquired during the session.
System Processes
Several data streams are working together within the FoulantProfiler software application.
Project Update
This data stream will synchronise the project definitions from network server. The
FoulantProfiler software application will look for PROJECT.ini in each project
folder. Synchronisation happens every 24 hours. If no project definition stored on
local, the synchronisation will happen immediately when this software is started.
Page 15
FoulantProfiler
User Manual
System Workflows
The life cycle of each internal process component is as follows:
Page 16
FoulantProfiler
User Manual
Extending and Re-Development
Target Audience:
Software Developer
Technical Details
Compile Environment and Dependencies
The FoulantProfiler software application is developed with dependencies including
Page 17
FoulantProfiler
User Manual
Note:
If your instrument is not supported yet, you will need to work out these settings
Instrument Configuration
Instrument Export Script (Optional)
Data File Processing Program
Instrument Configuration
The FoulantProfiler software application is a universal tool which reads the instrument and
research data. It has no knowledge about your instrument. The first step is to have your
instrument defined to the software application. The instrument configuration is created for
this purpose.
An instrument configuration is a series of settings in the system configuration file, by default
located in config/default.ini. You will need to provide settings for:
Page 18
FoulantProfiler
User Manual
Item
Description
equipment.EID.id
equipment.EID.name
equipment.EID.launcher.enabled
equipment.EID.launcher.program
equipment.EID.export.script
Page 19
FoulantProfiler
User Manual
equipment.EID.storage.flatfolder
Note:
Replace EID with the identity of your instrument. The identity is a unique
string composed of lower-case alphanumeric only (I.e.: letters a-z and
numbers 0-9). This identity is case sensitive and must be unique.
If more than one instrument is managed by your operation system, for example, multiple
instrument associated software installed on the same computer, you also need to add your
instrument identity to the system configuration as following
Item
Description
equipment.list
Page 20
FoulantProfiler
User Manual
--launched-from-manipulating-software --research-exchange-file FILE
where:
FILE will be replaced with the absolute path to a session exchange file.
The session exchange file will contain metadata of the session, including the project profile,
equipment information and technical metadata acquired from researchers. The session
exchange file will also contain a list of data files acquired during the session. The processing
program may produce new data files based on the file list. The new data file may be
monitored and recorded in the session exchange file, or be added to the session exchange
file by the processing program explicitly.
To indicate if the processing is successful or not, the processing program can return a status
code and message by putting two settings in the metadata section of session exchange file.
Item
Description
error.code
error.message
Note:
Users are encouraged to contribute their equipment settings and processing programs to
this project.
You need to re-compile the source code and generate a new binary execution file
afterwards.
Page 21
FoulantProfiler
User Manual
Page 22
FoulantProfiler
User Manual
Acknowledgement
This project is supported by the Australian National Data Service (ANDS). ANDS is supported
by the Australian Government through the National Collaborative Research Infrastructure
Strategy Program and the Education Investment Fund (EIF) Super Science Initiative.
Page 23