Hyperion Data Processing Instructions

CSIRO Office of Space Science & Applications
Earth Observation Centre

HYPERION DATA PROCESSING WORKSHOP:
Hands-on processing instructions

Bisun Datt & David Jupp

CSIRO Earth Observation Centre

CSIRO Earth Observation Centre
CSIRO Atmospheric Research

Hyperion Data & Processing Discussions

Foreword

The Hyperion (Pearlman et al., 2001) sensor on the Earth Orbiter 1 (EO-1) satellite
(Ungar et al., 2001) of the NASA New Millennium Program (NMP) is the first
hyperspectral instrument in space that covers the visible, near infra-red (VNIR) and
short-wave infrared (SWIR) bands between 400 nm and 2500 nm. It operated for one
year as an experimental mission during which time a team of scientists (the NASA
Science Validation Team, or SVT) assessed the technology and its applications. The
mission has been extended to provide hyperspectral data operationally from space for
a longer period with the data being publicly available and distributed through the
USGS up to and beyond October 2002.

These notes have been written to complement and extend the base material presented
in Barry (2001) and Jupp et al. (2002). Those documents, published respectively by
TRW Inc, the makers of the Hyperion instrument, and CSIRO EOC have
comprehensive discussions on the data and its processing. The material presented in
this document is a step-by-step, hands-on, guide to the basic processing of Hyperion
Level 1B1 data. It complements a 2-CD set of images and ancillary data as well as
background documentation that is provided to course participants.

In presenting this workshop, the input, support and continuing help of NASA, the
SVT and especially its Lead Scientist Steve Ungar as well as staff from TRW, Jay
Pearlman, Carol Segal, Pamela Barry (now with Raytheon) and many others is
acknowledged and appreciated. Hyperion has proved a great success and a significant
boost for worldwide Hyperspectral research. The privilege of being part of the SVT
and working with TRW has provided CSIRO and Australia with many opportunities
and benefits.

IMPORTANT NOTICE:

These notes are currently Draft material and are being released to the workshop
participants on the understanding that material they contain will usefully contribute to
their understanding of the workshop. When the document is completed and approved
by the normal CSIRO processes it will be replaced by a final version. Until that time
no part of this document may be quoted, transmitted or copied without permission of
CSIRO.

Copyright 2004 CSIRO
No part of this document may be copied or used without written approval of CSIRO

2
Hyperion Data Processing Workshop
HYPERION DATA PROCESSING WORKSHOP

TABLE OF CONTENTS

1 INTRODUCTION ...................................................................................................... 5
1.1 Delivered Hyperion Level 1R and 1B1 Data.......................................................5
1.2 Documentation and Data......................................................................................6
1.3 Hyperion Data Cube Structure.............................................................................9
1.4 Note on Hyperion Header Files ...........................................................................9
2 INTRODUCTION TO ENVI TOOLS YOU WILL USE ........................................ 10
2.1 Opening Files.....................................................................................................10
2.2 VNIR: Grayscale................................................................................................10
2.3 SWIR: Grayscale ...............................................................................................10
2.4 Link the VNIR and SWIR grayscale images .....................................................11
2.5 VNIR: visible RGB............................................................................................11
2.6 VNIR: vegetation RGB......................................................................................11
2.7 SWIR: RGB.......................................................................................................11
2.8 Spectral Profiles.................................................................................................11
3 LEVEL 1B1 RADIANCE AND INFORMATION CONTENT .............................. 12
3.1 Fixing bad pixels................................................................................................12
3.1.1 Fixing Bad pixels using Bad Pixel List......................................................... 13
3.2 Re-calibration of Hyperion Level 1B1 data.......................................................13
3.2.1 Re-calibration using ENVI Band Math........................................................ 14
3.2.2 Recalibration using Gain/offset ascii file...................................................... 15
3.3 Band Selection and Reviewing Radiance Data..................................................15
3.3.1 Setting Bad Bands......................................................................................... 15
3.3.2 Creating the 179-band subset with re-calibration......................................... 17
3.3.3 Examine Radiance Spectra ........................................................................... 17
3.4 Fix Out of Range Data .......................................................................................18
4 IMAGE NOISE & DIMENSIONALITY................................................................. 19
4.1 Data Dimensionality and SNR...........................................................................19
4.2 Fixing outlier pixels ...........................................................................................20
4.3 Streaking ............................................................................................................21
4.3.1 Global Destreaking ....................................................................................... 22
4.3.2 Local destreaking.......................................................................................... 24
5 ATMOSPHERIC CORRECTION............................................................................ 28
5.1 Atmospheric Correction using ACORN............................................................28
5.1.1 Ancillary data required for ACORN............................................................. 28
5.1.2 Running ACORN Atmospheric Correction.................................................. 30
5.2 Atmospheric Correction using FLAASH...........................................................32
5.2.1 Input parameters required for FLAASH....................................................... 32
5.2.2 Running FLAASH Atmospheric Correction ................................................ 36
6 VALIDATION OF DERIVED SURFACE REFLECTANCES............................... 38
6.1 Stable Band Selection ........................................................................................38
6.2 Setting up Ground Spectra in a Spectral Library...............................................39
6.3 Resampling the Spectral Library to Hyperion Bands ........................................40
6.4 Comparing Ground and Image Spectra..............................................................40
7 Advanced processing ................................................................................................ 41
3
7.1 Spectral Smile ....................................................................................................41
7.1.1 De-smiling Hyperion data............................................................................. 42
7.1.2 Effect of De_smiling on MNF transform...................................................... 42
7.2 Broad And Narrow Band Comparisons .............................................................43
7.2.1 Synthesis of LANDSAT and ALI bands from Hyperion Bands................... 43
7.2.1.1 Building User Defined Filter Functions........................................................ 43
7.2.1.2 Creating Filters for Binning Hyperion data into LANDSAT ETM and
ALI bands.................................................................................................................. 44
7.2.1.3 Examining Filter Statistics........................................................................... 45
7.2.1.4 Binning Hyperion bands into Landsat7 and ALI using ENVI spectral
resampling................................................................................................................. 45
7.2.1.5 Binning Hyperion bands into Landsat7 and Ali using the Workshop
Program-Apply Filter to Images. .............................................................................. 46
8 GEOMETRIC CORRECTION OF HYPERION DATA......................................... 47
9 INFORMATION EXTRACTION DEMONSTRATIONS ...................................... 47
APPENDIX: NOTE ON ENVI BAND MATHS TO OBTAIN CALIBRATED
IMAGE DATA................................................................................................................. 48
References......................................................................................................................... 49

4
1 INTRODUCTION

1.1 Delivered Hyperion Level 1R and 1B1 Data

At present the level 1R Hyperion data are provided in HDF format by the USGS on
CDs or downloaded from a ftp site. The data files are the image file (with L1R
extension), image header file and a .met file. The .met file contains information on
image acquisition date, time, latitude/longitude, etc. A typical level 1R dataset will
have the following files:

EO1H0890842002120110KZ.hdr
EO1H0890842002120110KZ.L1R
EO1H0890842002120110KZ.MET

The Level 1B1 data were delivered on 8mm exabyte tape or on CD in HDF format.
The level 1B1 data contain many of the calibration log files along with the image
files. A typical tape will have the following files:

EO12002011_188A1889_r1_SGS_01.avg
EO12002011_188A1889_r1_SGS_01.avg.log
EO12002011_188A1889_r1_SGS_01.avg.txt
EO12002011_188A1889_r1_SGS_01.echo.log
EO12002011_188A1889_r1_SGS_01.hdr
EO12002011_188A1889_r1_SGS_01.smear.log
EO12002011_188D188C_r1_SGS_01.aln.log
EO12002011_188D188C_r1_SGS_01.cal.log
EO12002011_188D188C_r1_SGS_01.echo.log
EO12002011_188D188C_r1_SGS_01.fix.log
EO12002011_188D188C_r1_SGS_01.L1_B1
EO12002011_188D188C_r1_SGS_01.L1_B1.hdr
EO12002011_188D188C_r1_SGS_01.L1_B1.qa
EO12002011_188D188C_r1_SGS_01.sat
EO12002011_188D188C_r1_SGS_01.smear.log
EO12002011_188F188E_r1_SGS_01.avg
EO12002011_188F188E_r1_SGS_01.avg.log
EO12002011_188F188E_r1_SGS_01.echo.log
EO12002011_188F188E_r1_SGS_01.hdr
EO12002011_188F188E_r1_SGS_01.smear.log

The image file has the L1_B1 extension. Other files are pre and post-image dark
current files and calibration log files. For details on the metadata files and level 0 and
level 1 processing refer to the Hyperion Science Data Users Guide (Section 3.3-3.5).

To open the HDF image in ENVI 3.5 for the first time: Select File > Open Image File
from the ENVI main window. Select the Hyperion Level 1B1 image file. When the
HDF Dataset Selection Window appears >select the HDF Hyperion image and click
OK. In the HDF Data Set Storage Order window select BIL > click OK.

The files being used in this workshop to illustrate the processing steps are in internal
ENVI BIL (Band Interleaved by Line) and BSQ (Band Sequential) formats and are
5
signed integer data with values ranging between 32768 and 32767. These are natural
formats to use in the processing stages.

[Note. All Hyperion image data provided as sample data on the workshop CDs are
level 1B1 and have been fixed for SWIR geometry. These images are like the level 1R
data provided by the USGS. If older Hyperion images are being used (ie. Prior to
level 1B1) users should check the data before preprocessing them using the methods
outlined here].

1.2 Documentation and Data

In addition to this document you will receive hardcopy of:

Barry, P. (2001). EO-1/Hyperion Science Data Users Guide, Level 1B1. TRW Space,
Defense & Information systems. Redondo Beach, CA, CAGE Number 11892, 60p.

Jupp et al. (2002). Discussions around Hyperion Data: Background Notes for the
Hyperion Data Users Workshop. Notes provided for this workshop to supplement
Barry (2001).

You will have two CDs with the following contents. CD1 contains all files and data
for this workshop. CD2 contains sample ACORN and FLAASH atmospherically
corrected reflectance images for the three sites.

CD1
Directory Subdirecto
ry
Files File description
BP_info.txt Description of bandpass
matrices
ASD2Hyp.txt ASD to Hyperion (242 bands)
ASD2Hyp_179bands.txt ASD to Hyperion (179 bands)
ASD2Hyp_179bands.csv ASD to Hyperion (179 bands)
in csv format
ASD2ALI.txt ASD to ALI

Binning_
weight_
matrices
ASD2ETM.txt ASD to Landsat7
Bad_pixel_list.txt List of bad pixels and bands
Hyperion_cen_fwhm.txt Wavelength calibration file for
pixel 128
Hyperion_cen_fwhm_179.txt Wavelength calibration file for
pixel 128 (for 179 bands subset)
Hyperion_cen_fwhm_av.txt Average wavelength calibration
file for pixels 1 to 256
Hyperion_cen_fwhm_av_179.txt Average wavelength calibration
file for pixels 1 to 256 (for 179
band subset)

Calibration
files
HypGain_revC.txt Level 1B1 calibration file
Hyp_Wshop_40.zip Add-on programs for ENVI 4.0
Hyp_wshop_35.zip Add-on programs for ENVI 3.5

Ancillary
data
ENVI_IDL
module
Readme.txt Installation instructions for
ENVI-IDL .sav files
6
Proc_doc.pdf Description of programs in the
ENVI IDL .sav files
Notes_on_Splib.pdf Notes on using the spectral
library clustering program
(run_cluster.exe)
acorn_gain_offset.txt Gain File for ACORN
atmospheric correction
recal_179.txt Ascii gain file for 179 bands
(multipliers to get radiance
x100)
recal_242.txt Ascii gain file for 242 bands
(multipliers to get radiance
x100)

Gain-offset
files
recal_242to179.txt Ascii gain file to convert 242
Level 1B1 bands into 179 bands
(radiance x100)
Avwvl(179).txt Average central wavelengths
for 179 bands
BandWidthL0_dat.txt FWHM for Hyperion
Cwl(179).txt Central wavelengths for 256
columns and 179 bands

Hyperion_
smile
SpectralL0_revA_dat.txt Central wavelengths for
Hyperion (256 pixels by 242
bands)
ASD_TO_HYPERION
User
Defined
Filter
Functions
ASD_TO_HYPERION.HDR
Filter function (spectral library)
for binning ASD data to 155
Hyperion bands
Hyperion Data Processing
Instructions.pdf
This document
CHW_Introduction_to_Hyperion.p
df
Powerpoint presentation
EO1HyerionScienceDataUsersGuid
e_public_L1_B1.pdf
Hyperion data users guide by
P. Barry 2001.
Hyp_Wsn.pdf Supplementary workshop notes
Proc_doc.pdf Description of programs in the
ENVI IDL .sav files
Heading.xls Excel spreadsheet for
calculating satellite heading
Notes_on_Splib.pdf Notes on using the spectral
library clustering program
(run_cluster.exe)

Documents

EOC_EO1_report_Final.pdf Final report on the CSIRO
contribution to the NASA EO1
Science Validation Activity
Coleambally_soil1.txt
Coleambally_stubble1.txt ASD
Spectra
Coleambally_stubble2.txt
Ground ASD spectra - image
coordinates (pixel, line) in
brackets
Coleambally_L1_B1 Level 1B1 image data
Coleambally_L1_B1.hdr Image header file

Coleambally

Image Data
Location_Acquisition_data(Coleam
bally).txt
Image date/time, location

7
Mount_Fitton_L1_B1 Level 1B1 image data
Mount_Fitton_L1_B1.hdr Image header file Mt. Fitton

Image data
Location_Acquisition_data(Mt.
Fitton).txt
Tumbarumba_vegetation(35,574).tx
t
Tumbarumba_vegetation(5,595).txt

ASD
Spectra
Tumbarumba_vegetation(55,608).tx
t
Ground ASD spectra - image
coordinates (pixel, line) in
brackets
Tumbarumba_L1_B1 Level 1B1 image data
Tumbarumba_L1_B1.hdr Image header file

Tumbarumba

Image Data
Location_Acquisition_data(Tumbar
umba).txt

CD2
Directory Subdirectory Files File description
Coleambally_acorn_refl ACORN reflectance
image
Coleambally_acorn_refl
.hdr
Image header file
ACORN(Coleambally)
Coleambally_acorn_ctrl
.in
Acorn input parameters
file
Coleambally_flaash FLAASH reflectance
image
Coleambally_flaash.hdr Image Header file
Flaash(Coleambally)
Coleambally_template.t
xt
FLAASH input
parameters file
Mount_Fitton_acorn_re
fl
ACORN reflectance
image
Mount_Fitton_acorn_re
fl.hdr
Image header file
ACORN(Mount_Fitton)
Mount_Fitton_acorn_ct
rl.in
file
Mount_Fitton_flaash FLAASH reflectance
image
Mount_Fitton_flaash.hd
r
Image Header file
Flaash(Mount Fitton)
Mt Fitton_template.txt FLAASH input
parameters file
Tumbarumba_acorn_ref
l
ACORN reflectance
image
Tumbarumba_acorn_ref
l.hdr
Image header file
ACORN(Tumbarumba)
Tumbarumba_acorn_ctr
l.in
file
Tumbarumba_flaash FLAASH reflectance
image
Tumbarumba_flaash.hd
r
Image Header file

Hyp_Worksh
op_CD2.zip

Flaash(Tumbarumba)
Tumbarumba_template.
txt
FLAASH input
parameters file

8

1.3 Hyperion Data Cube Structure

Dimensions of a typical Hyperion Image are 256x1400x242.

Where: 256 = number of samples or pixels in Field Of View
1400 = number of lines (frames)
242 = number of spectral channels

Ground Pixel size = approx. 30m x 30m
Swath width = 7.7 km

(Note: The images used in this workshop are subsets of long 6925 line Hyperion
scenes collected during the SVT mission. The subsets have 1400 lines out of the
original 6925 to conform with the products being delivered by the USGS.)

The Level 1B1 data consists of 242 nominal bands covering the wavelengths from
356 nm to 2577 nm. The VNIR detector collects data in bands 1 to 70 and the SWIR
detector collects data from bands 71 to 242. There is an overlap between the VNIR
and SWIR regions.

Some of the 242 spectral channels are not calibrated in the Level 1B1 data; these are
channels that were not activated in the Hyperion arrays or some with low signal levels
and some other channels which are duplicated in the VNIR-SWIR overlap region. The
Level 1B1 data contains 200 calibrated channels of which 196 are unique.

The Level 1B1 data is provided as scaled radiance data in a signed integer format. The
calibrated radiance (W/m
2
/sr/m) values for spectral channels 1-70 have been
multiplied by a factor of 40 and channels 71-242 have been multiplied by a factor of
80 to maintain precision. This will be discussed in more detail later.

1.4 Note on Hyperion Header Files

There are two versions of header files being used with Hyperion Level 1B1 data.

1. The data processed by TRW uses the central wavelengths and FWHMs for
sample no. 128 in the header files. These are the current header files for the
images used in these workshop materials. The central wavelengths and
FWHMs for sample 128 are provided in the file Hyperion_cen_fwhm.dat
under the \Ancillay data\Calibration files\ directory on workshop CD1.
2. The more recent data processed by the EROS Data Centre (EDC) uses the
average central wavelengths and FWHMs for all 256 samples in their header
files. These average central wavelengths and FWHMs are provided in the file
Hyperion_cen_fwhm_av.dat under the \Ancillay data\Calibration files\
directory on workshop CD1.

If you are not sure which header files are used for any of your images you could
check the header information against the two files described above. Researchers using
9
Hyperion data may also wish to check if using either header produces any differences
in processing for the data you use - such as atmospheric corrections.

2 INTRODUCTION TO ENVI TOOLS YOU WILL USE

2.1 Opening Files

Start ENVI. (In Windows, double click the ENVI icon. In UNIX, enter envi at
the UNIX command line)
From the main ENVI window > select File > Open Image File
From the Enter Data Filenames Window > Navigate to the directory
C:\Hyperion_Data_Processing_Workshop\ > and select an image file from
one of the subdirectories e.g. \Coleambally\Image Data\Coleambally_L1_B1

The Available Bands window should appear with all the bands listed

2.2 VNIR: Grayscale
In the Available Bands List window > scroll to Band 40 (753 nm) > click Load
Band.

The Main Image Display, Scroll, and Zoom windows should appear. The
scroll window shows the full image; the main display shows in more detail the
section outlined by the red box in the scroll window; and the zoom window
shows a magnified view of the area inside the red box in the main display.

Examine the image by clicking at different areas in the Main Image window.
This will move the red box outlining the zoom window to the desired area of
the image. The zoom window updates each time. (The red box in the Main
Display can also be positioned by clicking inside the box and dragging it to the
desired position).

Change the zoom factor by clicking on the red - button within the zoom
window to zoom down and the + button to zoom up.

Holding down the left mouse button and moving the cursor around inside the
zoom window allows you to zoom to different areas of the main image
display.

Click (or drag the red box) inside the scroll window to display another area of
the image in the main display.

2.3 SWIR: Grayscale
In the Available Bands window > scroll to Band 93 (1074 nm) > click on the
Display # button > select new display > click Load Band.

Examine different areas of the image using the scroll, main display and zoom
windows as described above for the VNIR image.

10
2.4 Link the VNIR and SWIR grayscale images
Right click on the VNIR grayscale image and select Link Displays > Select
Yes for Display #s corresponding to the two images > Leave Dynamic
Display On > click OK.

Click on either display to toggle (flicker) the images. Note the effects of the
VNIR-SWIR pixel co-registration.

2.5 VNIR: visible RGB
To display a true color RGB image > Select the RGB color button in the
Available Bands window > Click on bands 29 (641 nm), 23(580 nm), 16(509
nm) >select new display > click load RGB to display the image.

2.6 VNIR: vegetation RGB
To display a color image in which vegetation appears red > Select the RGB
color button in the Available Bands window > Click on bands 50 (855 nm),
23(580 nm), 16(509 nm) >select new display > click load RGB to display the
image.

Also try RGB bands 45(803 nm), 33(680 nm), 20(550nm) and the reverse
view
RGB bands 33(680 nm), 20(550nm), 45(803 nm) the second is an example of
a False False Colour Composite

[NOTE: In some cases you will need to use the Enhance features to obtain a
good contrast to the image. Try both Linear 2% and Equalise.]

2.7 SWIR: RGB
To display a color image of the SWIR bands, bands 204:150:93 can be used as
a R:G:B combination.

Select the RGB color button in the Available Bands window > Click on bands
204(2194 nm):150(1649 nm):93(1074nm) >select new display > click load
RGB to display the image.

Also try RGB band combinations 187(2020nm), 140(1550nm), and
48(830nm)

2.8 Spectral Profiles
To extract the spectral profile for specific pixels in the image:

From the Main Display Window, select Tools > Profiles > Z Profile
(Spectrum)

The Spectral Profile Window comes up with the current spectrum displayed.
The spectrum corresponds to the pixel on which the zoom cross-hair is
positioned in the zoom window.

11
Clicking on different areas of the main image or zoom window updates the
displayed spectrum to match the selected pixel.

Clicking the right mouse button in the Spectral Profile window shows the
pixel coordinates of the displayed spectrum.

To display several spectra on the same display:

In the Spectral Profile window > select Options > Collect Spectra
Then click on different regions of the image to collect their spectra.

To stack the spectra:

In the Spectral Profile window > select Options > stack data

To change the stack offset:

In the Spectral Profile window > select Options > set stack offset
This will allow you to adjust the amount of overlap in the displayed spectra.

3 LEVEL 1B1 RADIANCE AND INFORMATION CONTENT

[This section uses signed integer BIL files as inputs and they initially have no bad
bands set.]

3.1 Fixing bad pixels

In pushbroom sensors such as Hyperion, poorly calibrated detectors in the VNIR or
SWIR arrays will leave vertical stripes or streaks in certain pixels of an image band.
The most extreme cases of these pixels contain little or no valid data and are
identified as bad pixels. In the level 1B1 processing these bad pixels have been filled
by averages of data from the two adjacent pixels. In more recent processing (level 1R)
these pixels have been left unmodified and a list of such pixels is provided with the
data, leaving users to determine ways of fixing these bad pixels. The bad pixel list
identifies about 17 pixels out of the total 242x256 pixels as bad. However, users have
identified additional bad pixels and even some pixels with severe striping can be
added to this list. The method used in the workshop fills bad pixels by replacing them
with averages of data from adjacent pixels. The bad pixels are listed in an ACSII file
which is used by the Bad Pixel List procedure in the workshop ENVI module. The
bad pixel ASCII file can contain individual bad pixels (listed as Band .Pixel pairs
on each line) or bad bands (listed as Band0 pairs). Bad bands here are those band
not utilised in Hyperion imagery and these are replaced with zeros. Due to the
geometry fix between the VNIR and SWIR bands the last pixel (256) is left blank for
all SWIR bands. This pixel can also be added to the bad pixel list and filled with data
from pixel 255. The sample bad pixel ascii file supplied with the workshop materials
contains most of the bad pixels and bands that need to be fixed. Users can modify this
file by adding/removing bands or pixels from the list to suit their datasets.

12
3.1.1 Fixing Bad pixels using Bad Pixel List
(Note: If the Workshop menu item does not appear on the ENVI main menu, the
Hyp_Wshop.sav module is not installed. To install this module, copy the file
Hyp_Wshop.sav from the \Ancillary data\ENVI_IDL module\ directory on the
Workshop CD1 and place it in the ENVI35 save_add directory. Restart ENVI to load
the module).

From the ENVI main toolbar > select Workshop > Apply Bad Pixel List
Read the module description in the Calibrating with an ascii file window and
click Go.

Select the input image and click ok. The input image should usually be the
original 242-band Hyperion level 1B1/1R image.

The OK to get on with fixing the Bad Pixels? window comes up. Check the
information listed in this window and click Yes.

Then select the Bad Pixel list and click Open. The ASCII file containing a list
of bad pixels is available under \Ancillary data\Calibration files.

Check the information in the File Data OK to Apply to the Image? window
and click Yes. Select the output file name and click Open to start the
processing. (By default the output file is named with a _bpfix extension
added to the input file name).

3.2 Re-calibration of Hyperion Level 1B1 data

After calibration and before being written to file as signed integers the Hyperion data
are multiplied by factors in the VNIR and SWIR to maintain precision. To take them
to a representative calibration where there is a uniform multiplier into radiance units
we also need to maintain the precision as integer data or else make the output files
real numbers as described in greater detail in Appendix A. Since the input to ACORN
and FLAASH assumes integer data we will maintain it in these steps and make the
conversion between Wm
-2
sr
-1
m
-1
and the data a factor of 100. The conversion into
two common units of radiance is:

Image Data/100 Wm
-2
sr
-1
m
-1
(used for input to ACORN)
Image Data/1000 W cm
-2
sr
-1
nm
-1
(used for input to FLAASH)

Re-calibration of the Level 1B1 data ensures a uniform calibration gain for each band,
providing data that is needed for the binning section later, and also provides better
interpretation of radiance spectra.

There are two methods for re-calibrating Hyperion Level 1B1 data to a uniform factor
of radiance times a 100 for all bands. The first method can be implemented in ENVI
and the second uses an ENVI add-on program provided with these workshop
materials. The second method is the preferred one and will be used during the
workshop.

13
3.2.1 Re-calibration using ENVI Band Math
This method uses the Band Math tool in ENVI to apply gains and offsets to the VNIR
and SWIR separately, and then the two files are put together. The resulting files are in
BSQ format and they may need to be converted to BIL later for certain types of
processing.

VNIR Bands (1-70)
From the ENVI main toolbar > select Basic Tools > Band Math
Under Enter an expression, type FIX(100.0*(float(b1)/40.0)+0.5) > click OK
In the Variables to Band Pairings window > click on Map variable to input
file
Select the filename > click on spectral subset > highlight bands 1 to 70 > click
OK, OK. Enter an output filename. Click OK to start the processing. This will
be the VNIR bands converted to absolute radiance and scaled by 100. The
output file type is BSQ.

SWIR Bands (71-242)
From the ENVI main toolbar > select Basic Tools > Band Math
under Enter an expression, type FIX(100.0*(float(b2)/80.0)+0.5) > click OK
In the Variables to Band Pairings window > click on Map variable to input
file
Select the filename > click on spectral subset > highlight bands 71 to 242 >
click OK, OK. Enter an output filename. Click OK to start the processing.
This will be the SWIR bands converted to absolute radiance and scaled by
100. Again the output file type is BSQ.

Combine the VNIR and SWIR Radiance Files
From the ENVI main toolbar > select file > save file as > ENVI Standard
In the New File Builder window > click on import file > select the VNIR
radiance file. Then import the SWIR radiance file. Click OK, OK. Enter an
output filename. The result will be the combined VNIR and SWIR radiance
data.

Attach the Wavelength Header
From the ENVI main toolbar > select file > Edit ENVI Header > select the
filename for the combined VNIR-SWIR radiance data set
In the Header Info window > click on Edit Attributes > Select Wavelengths
In the Edit Wavelength Values window > click on Import ASCII > select the
Hyperion_cen_fwhm.dat file (this file is in the Ancillary_Data subdirectory) >
enter 2 for the wavelength column > enter 3 for the FWHM column. Click
OK. Click OK on the previous windows as well.

While the above method can be implemented using existing ENVI tools, it is time
consuming and includes several steps, i.e. the VNIR and SWIR data are re-calibrated
separately, then they are recombined and the image header has to be added to the
combined file. The output from the band math procedure is a BSQ file, which needs
to be converted into a BIL format before atmospheric correction with ACORN or
FLAASH.

14
An alternative method in ENVI (using the tools under: Basic Tools > General
Purpose Utilities > Apply Gain and Offset) can also be used to apply gains and offsets
to each band. While this method produces a BIL output in a single step, it is still
tedious in that the gain values have to be entered manually for each band.

An ENVI-IDL procedure called Gain/offset using ascii file has been developed for a
more efficient recalibration of Hyperion data. This procedure is contained in the
Hyp_Wshop.sav file provided on the workshop CD1 (\Ancillary data\ENVI_IDL
module\ Hyp_Wshop.sav) and is also installed on the computers used for this
workshop. The use of the Gain/offset using ascii file procedure is described below.

3.2.2 Recalibration using Gain/offset ascii file
This method uses gain (and, if needed, offset) values for all bands from an ASCII file
and outputs a re-calibrated BIL file in a single step. It can also take an input file and
produce a re-calibrated output file containing a subset of the original bands.

The ascii file required for re-calibrating all 242 bands to radiance times 100 is called
recal_242.txt and is provided under \Ancillary data\Gain-offset files.

From the ENVI main toolbar > select Workshop > Gain/Off using ASCII file
Read the module description in the Calibrating with an ascii file window and
click Go.
Select the input image file.
Next select the gain/offset (.txt) file containing the gain and offset values for
all bands. Use the file recal_242.txt from the directory \Ancillary files\Gain-
offset files.
Select an output image filename and click open to start the calibration.

3.3 Band Selection and Reviewing Radiance Data
There are two ways to treat bad bands in an image. One method is to set them as bad
bands so that those bands will not be used in any further processing applied to the
image (as described in section 3.3.1). The other method is to exclude the bad bands
permanently by creating an image subset containing only the good bands (as
described in section 3.3.2).

The second method is quicker and more efficient, and is the preferred method to use.
However you should go through section 3.3.1 first to understand the criteria for
selection of the bad bands before creating the image subset in section 3.3.2.

3.3.1 Setting Bad Bands

As noted above, some bands are set to zero during Level 1B1 processing. The
zeroed bands are 1-7, 58-76, 225-242.

Bands 56-57 in the VNIR overlap with bands 77-78 in the SWIR region.
Bands 77-78 have usually been found to be noisier than the corresponding
bands in the VNIR.

To check which bands to keep and which to ignore in the VNIR-SWIR
overlap region:
15

Load bands 56, 57, 77, 78 as grayscale images in separate displays.
Link the 4 displays using Tools > Link > Link Displays from one of the
main image displays. In the Link Displays window > select yes for
the four Display #s and click OK.

Examine the four images by scrolling to different regions of the image.
Decide which bands are noisier.

If the zero data bands and the VNIR-SWIR overlap (bands 56,57 or 77,78?)
are ignored, there are 196 unique calibrated bands (bands 8-57 and bands 79-
224), which can be retained for processing.

Within the 196 unique bands there a number of atmospheric water vapour
bands that absorb most of the solar radiation. These can be identified by
examining the radiance spectra (see Figure 9). The strongest water vapour
bands occur between 1356 nm and 1417 nm, 1820 nm and 1932 nm, and
beyond 2395 nm. Since these bands contain little or no information about the
surface, they are usually removed from hyperspectral data.

However some atmospheric correction programs such as ENVI FLAASH do
require bands centred near 1380 nm in the strong water vapour wavelengths
for masking clouds (especially high altitude clouds). Therefore bands 123-125
(1376, 1386, and 1396 nm) will be retained in the image.

Ignoring the strong water vapour bands 121-122, 126-127, 167-178, and 224
leaves a subset of 179 bands for atmospheric correction and further
processing. The 179-band subset includes bands 8-57, 79-120, 123-125, 128-
166, 179-223.

The zeroed bands and strong water vapour bands will now be set as bad and
will be ignored in further processing.

From the ENVI main toolbar > select file > Edit ENVI Header > select the
filename for the radiance data set that you just created.
In the Header Info window > click on Edit Attributes > Select Bad Bands List
In the Edit Bad Bands List values window > hold down the Ctrl key and click
on bands 1-7, 58-78, 121-122, 126-127, 167-178 and 224-242. Click OK, OK.
These bands have now been set as bad bands.

Note (1): In the bad bands list the good bands are selected and the bad bands
are not selected. There are various ways of achieving this but take care you do
not make the good bands bad and vice versa)

Note (2): The image display windows close after the bad bands have been set.
Reload the image by selecting the RGB bands in the available bands list or just
right click in the available bands list and select Load True Color to <new>

16
3.3.2 Creating the 179-band subset with re-calibration
Since the 179 bands described in 3.3.1 will be used in the following sections,
it is useful to create a new image containing only these bands. This will reduce
data volume and speed up the processing.

The method used here involves processing from the original data again and
applying the re-calibration using the Gain/Off option on the Workshop menu.
The 179 bands may be chosen here at the same time as the data are scaled to
radiance times 100.

The ascii file required for creating the 179-band subset of radiance times a
factor of 100 directly from the original 242-band level 1B1 image is called
recal_242to179.txt and is provided under \Ancillary data\Gain-offset files.

From the ENVI main toolbar > select Workshop > Gain/Off using
ASCII file
Read the module description in the Calibrating with an ascii file
window and click Go.
Select the input image file (same as before).
Next select the gain/offset (.txt) file containing the gain and offset
values for all bands. Use the file recal_242to179.txt from the
directory \Ancillary files\Gain-offset files.

Select an output image filename and click open to start the calibration.

3.3.3 Examine Radiance Spectra

Examine the radiance spectra by plotting their spectral profiles. Look for
atmospheric features (see figure below and the figures in the document
Discussions around Hyperion Data:Background Notes for the Hyperion Data
Users Workshop provided to you for this workshop for typical atmospheric
features you will be able to recognise in the radiance spectra).

Collect spectra from different targets in the image and compare. Can you
differentiate between different cover types easily?

Examine the spectra in the VNIR-SWIR overlap bands (i.e. the continuity
between bands 57 and 79).

17

(Figure adapted from Hyperion data Users Guide. Note that this is a very
bright feature and would reach to 14000 in the scaled data you have.)

3.4 Fix Out of Range Data

During the re-scaling of the signed integer data to radiance times 100 in Section 3.2,
some bright targets in the scene such as cloud tops could reach values higher than
+32767. Such data would wrap around to 32768 or similar negative values. Some
Hyperion level 1 data from the early part of the EO1 mission were in unsigned integer
format and scaled to radiance x 100, and if such data are converted to level 1B1
signed integers, integer wrap around could also occur for bright targets.

The wrapped around data can be set to saturated +32767. The method used in this
workshop is based on setting a threshold negative value below which all values will
be set to +32767, while valid negative values are preserved. A threshold of 1000 can
be used in most cases.

To fix any out of range data in the 179-band image:

From the ENVI main toolbar > select Workshop > Fix Out of Range Data
Read the module description in the Fixing out of range data in Hyperion Data
window and click Go

Select the 179-band image file and click OK
Check the file information displayed in the OK to get on and fix out of range
data? Window and click Yes.
18

In the Set Threshold for valid/invalid data window, set values less than 1000
to +32767, and values greater than +32767 to +32767. Click OK. Select an
output filename (the default file name is input filename + _zfix).

Once the processing is complete, the Image Processed result OK? Window
shows up indicating the number of changes made. Click Yes to finish the
processing.

4 IMAGE NOISE & DIMENSIONALITY

4.1 Data Dimensionality and SNR

The MNF Transform is used to determine the inherent dimensionality of image data,
and to segregate noise in the data especially spatial noise. Examination of the MNF
bands gives an idea of how much useful information and noise are present in the data.

Since the VNIR and SWIR data in Hyperion are from two different spectrometers,
they have different noise structures. The MNF transform should therefore be applied
separately to the VNIR and SWIR bands.

Perform an MNF transform on the VNIR bands in 179-band radiance image.

From the Main ENVI window select > Transform > MNF Rotation > Forward MNF >
Estimate Noise Statistics From Data.

Select the filename. Use spectral subset option to select the VNIR bands only (the
first 50 bands in the 179 band subset, or bands 8-57 of the original Hyperion bands).
Click Enter.

In the Forward MNF Transform Parameters window > select no for select subset
from eigenvalues and enter 50 for number of output MNF bands. Click ok. It will
take a while for the MNF transform to finish.

Load and display the MNF bands as grayscale and RGB combinations.

Animate the MNF bands to see the good MNF bands and noise, streaking, limb
brightening etc.

How many good bands are there in the VNIR MNF image?

Repeat the MNF transform on the SWIR bands (the last 129 bands in the 179 band
subset or bands 79-120, 123-125, 128-166, 179-223 from the original Hyperion
bands).
Examine the good MNF bands in the SWIR and compare with the number of good
MNF bands obtained from the VNIR.

What do you conclude about the limb brightening and the streaking in VNIR and
SWIR?
19

4.2 Fixing outlier pixels

Treatment of outlier pixels or spikes in the data is an important step prior to the
application of processing steps such as destreaking. Presence of outliers in the data
will affect the column statistics for data bands and hence influence the destreaking
based on statistical balancing.

The method for fixing outlier pixels is based on using median and mean absolute
deviation (MAD) statistics for detecting outliers and replacing them with the median
value from a neighbourhood of pixels. The routine can be run in three modes. One is
"global" the second is "local/global" and the third is (fully) "local".

In the global method pixels in each band are tested as to whether the absolute value of
the difference of the pixel with the median of the band is more that a given fraction of
the band MAD.

In the local/global method pixels in each band are tested as to whether the absolute
value of the difference of the pixel with the median in a LOCAL neighbourhood is
more than a given fraction of the band MAD.

In the (fully) local method pixels in each band are tested as to whether the absolute
value of the difference of the pixel with the median in a local neighbourhood is more
than a given fraction of the MAD in the same LOCAL neighbourhood

In each case pixels that fail the test are replaced by the median in the local
neighbourhood.

A rough rule is to have the fraction of the MAD used in the "global" case (eg 12)
about double in size what would be selected for a local/global filter (eg 6) and again
the local/global about double in size to the value that would be selected for the (fully)
local case (eg 3).

Global is used to detect and fix very bad outlier data. Local/global is used to de-spike
images in a conservative way. Local removes local salt and pepper noise or speckle
and modifies the data significantly. Global de-spiking is the most conservative and
local the least

Notes:
Outlier fixing has been introduced as even Level 1R data have significant outliers that
can even introduce streaks during de-streaking if not removed. There are also
many cases where removing salt and pepper noise improves the results significantly.
However, this routine, unlike others in the set, requires the use of spatial tiling and
hence results in a BSQ file. If input files are not BSQ then the processing is slow. The
destreaking in the following sections outputs BIL files from the BSQ files created
from outlier fixing.

To fix any outlier pixels in the image created in section 3.4:

20
From the main ENVI menu select > Workshop > De-spike outliers and click
go.

Select the input file (179 band image from section 3.4) and click ok.

For BIL files the warning window Input file structure is not optimal will show
up. Click ok to continue.

Check the image information listed in the OK to get on and fix the outliers?
(Y/N) window and click Yes.

In the Set Parameters for data outlier fixing? window select Local/Global and
set Threshold value to 6 and Window size to 3. Click OK to continue.

Select a name for the output image and click OK to finish the processing.

4.3 Streaking

If you display various bands (such as 56 and 57 in the VNIR and 97 and 186 in the
SWIR) it will be clear that there are many vertical streaks occurring as a noise pattern
in the data. Using the Equalise enhancement is a good way to see how these are
distributed in the image.

The Band Animation feature in ENVI can also be used to quickly examine all of the
individual bands of the image and identify bands/lines most affected by streaking and
other effects.

From the main image display > select Tools > Animation

In the Animation Input Parameters window > select all bands under Animation Input
Bands and Full Scene under Spatial subset > click OK

The animation movie starts once all frames have been loaded.

You can adjust the animation speed by clicking on the up and down arrows. The other
buttons on the animation window can be used to stop or pause the animation, or
change the direction in which the bands are displayed, etc.

To manually scroll through the bands, click on the stop button to pause the animation,
then click /drag the slider to look at individual bands.

De-streaking radiance data

Two methods will be used for removing pixel streaks or vertical stripes from images,
(1) Global destreaking, and (2) Local Destreaking. Both these methods use
statistical balancing to modify the data values. Gains and offsets are applied to each
pixel (column) in a band to match the column means and/or standard deviations to a
reference mean and standard deviation. Global destreaking uses the whole image
mean and standard deviation for each band as the reference values, while local
21
destreaking uses reference means and standard deviations from a local neighbourhood
of columns.

An ENVI-IDL procedure has been prepared for global and local destreaking. The
procedure Column stats d_streak is included in the ENVI Workshop menu. A
powerful feature of this ENVI-IDL procedure is that it allows selected bands to be
destreaked and other bands remain unchanged. Thus the VNIR and SWIR bands can
be destreaked separately but without the need to cut and stitch back the VNIR and
SWIR images before/after the processing. The VNIR bands can be destreaked first
and then the output image can be used to destreak the SWIR bands. The desteaking
can also be applied to any number of selected bands and can be reapplied to any bands
which are not fixed in a previous destreaking operation.

4.3.1 Global Destreaking

For details of the Global Destreaking method see the section Streaks in Hyperion
data and options for their removal in the document Discussions around Hyperion
Data: Background Notes for the Hyperion Data Users Workshop).

In global destreaking, the whole image mean and standard deviation as well as the
column means and standard deviations are calculated for each band. Three options for
global de-streaking are available in the module: additive, multiplicative, and linear. In
the additive method the data are adjusted so that column means are all equal to the
whole image mean for each band, but the column standard deviations are unchanged.
In the multiplicative method the column means are made equal to the whole image
mean but the standard deviations may also change. In the linear method the column
means and standard deviations are both made equal to the whole image means and
standard deviations for each band.

While the linear method may perform best in many cases, the additive or
multiplicative methods may work better in certain cases. It is left to the user to
determine which option reduces most streaking in their images.

If you select the sample image supplied for the Tumbarumba site, try global
destreaking using the linear method (as described below) and also with the additive
method. Compare the results (especially over the large area of water in the scene).
Which method works better in this case?

Apply the linear method for global destreaking as follows:

From the ENVI main menu select > Workshop > Column stats d_streak > Go

In the Select file to get Column Stats window > Select the 179-band radiance
image created in section 4.2 and click OK.

Check the file information in the OK to get on and compute the statistics and
click Yes. (Note: If the program finds outlier pixels in the image it will put up
a warning window indicating the number of bands which may have outliers. If
this happens you should go back and fix the outliers as shown in section 4.2
above).
22
When prompted to save the statistics in a file select Yes > Enter an output
filename and click Save. Click OK once the statistics have been saved.

In the De-streaking Method window select Global and Click OK.

In the D_Streaking using Global statistics window select Linear and Click
OK.

In the Get Output Range window enter lines 1 to 1400 and click OK.

Then enter an output file name for your globally destreaked image and click
Save to start the destreaking process.

Compare the globally destreaked and original images.
Display individual bands from the original and globally de-streaked radiance
images. Select some bands which showed streaks in the original image and
compare with the globally destreaked image. Are the streaks gone?

To compare the effect of de-streaking on the spectra:

Link the image displays. (From one of the main image displays select Tools >
Link > Link Displays. In the Link Displays window > select yes for the
Display #s and click OK.)

Display spectral profiles for the de-streaked and original radiance images.

Moving the scroll box to different areas on either of the linked images will
extract the spectral profiles of the same pixel for both images.

In the Spectral Profile window > Click on options > New Window Blank. This
will bring up a blank window on the screen. Then right click in one of the
spectral profiles to display the annotation (pixel reference (X,Y) > left click
and drag the annotation into the blank window to export the spectral profile.

The other spectrum can then be dragged into the blank window for
comparison. (Note: To make your spectra stand out, change their color by
selecting Edit > Data Parameters > Color from the spectral profile windows).
To clear the exported spectra right click on their annotations.

Compare spectra from pixels which showed streaks in the original image with
the spectra from the destreaked image. Are there any differences in radiance
values?

Also compare spectra from pixels which did not show any streaking in the
original image. Are these spectra also modified?

Perform an MNF transform on the globally de-streaked radiance image using the
method described in section 4.1.
23

Animate the MNF bands and compare with the MNF bands of the original
radiance image from section 4.1. How many good MNF bands are there now
compared to before de-streaking?

What happens to the spectral smile radiance effect after global destreaking ?
(Compare the first few MNF bands).

Another way to examine the effect of global destreaking on the data is to look at the
difference images between the destreaked and original data. To create the difference
image:
From the ENVI main window select > Basic Tools > Band Math

In the Band Math window > enter the expression float(b1-b2) and click OK.

In the Variables to Band Pairings window > click on the variable B1 > then
click on Map Variable to Input File.

In the Band Math Input File window select the original 179-band radiance
image from section 4.1 and click OK.


In the Band Math Input File window select the globally destreaked 179-band
radiance image and click OK.

Enter an output file name for the difference image and click OK in the
Variables to band Pairings window.

Load individual bands of the difference image and examine how the data have
been changed across the image.

Perform a MNF transform on the difference image and examine the bands.

From the above results what can you conclude about global destreaking. Does
it have any adverse effect on the data in addition to removing streaks and the
smile effect?

4.3.2 Local destreaking

In local destreaking the reference statistics are estimated locally. The first step
involves detection of outlier pixels using set thresholds based on median values, and
pixels above the specified thresholds are modified (as done in section 4.2). Following
the replacement of outlier pixels, a local smoothing filter is applied. The smoothing
filter widths can be defined in both the pixel and band dimensions. The reference
means and standard deviations are calculated from the pixels defined by the filter
width and the column data are adjusted to match column statistics to the local
statistics. Usually, filters in the pixel dimension are used for local destreaking prior to
24
atmospheric correction of the image data. Local smoothing using filters in the band
dimension can be applied after atmospheric correction for smoothing the data.

Apply local destreaking to the 179-band radiance image as follows:

From the ENVI main menu select > Workshop > Column stats d_streak > Go

In the Select file to get Column Stats window > Select the 179-band radiance
image created in section 4.2.1 and click OK.

Check the file information in the OK to get on and compute the statistics and
click Yes. When prompted to save the statistics in a file select Yes > Enter an
output filename and click Save. Click OK once the statistics have been saved.

In the De-streaking Method window select Local and Click OK.

In the D_Streaking the Image Using Local statistics window select >
Threshold and click OK.

In the Locating outlier columns using Local statistics window set the filter
size in the sample direction , Size_X as 5 and leave the filter size in
wavelength direction, Size_W as 0. Click OK.

In the Edit mean and SD thresholds window > set Thresh_mn = 2 and
Thresh_SD = 2. Click OK.

Check the number of changes to the means and SDs and select Yes to
continue or No to go back and change the threshold settings. Select Yes
once the outlier replacements are acceptable.

When prompted to save the changed pixels List to a file select Yes and then
provide a filename with a .csv extension and click Save.. (The csv format
allows the file to be opened later in a spreadsheet such as MS Excel).

The next step is the local destreaking. In the Get the output range of lines to
d_streak select the number of lines to which you want to apply the local filter.
Usually all lines of the image are selected by default, but you can apply the
filter to a subset of the image only. Select the first and last lines and click OK.

Then enter the output filename for the locally destreaked image and click
Save.

Compare the locally destreaked and original images.
Display individual bands from the original and locally de-streaked radiance
images. Select some bands which showed streaks in the original image and
compare with the locally destreaked image. Are the streaks gone?

To compare the effect of de-streaking on the spectra:

25
Link the image displays. (From one of the main image displays select Tools >
Link > Link Displays. In the Link Displays window > select yes for the
Display #s and click OK.)

Display spectral profiles for the de-streaked and original radiance images.

Moving the scroll box to different areas on either of the linked images will
extract the spectral profiles of the same pixel for both images.

In the Spectral Profile window > Click on options > New Window Blank. This
will bring up a blank window on the screen. Then right click in one of the
spectral profiles to display the annotation (pixel reference (X,Y) > left click
and drag the annotation into the blank window to export the spectral profile.

The other spectrum can then be dragged into the blank window for
comparison. (Note: To make your spectra stand out, change their color by
selecting Edit > Data Parameters > Color from the spectral profile windows).
To clear the exported spectra right click on their annotations.

Compare spectra from pixels which showed streaks in the original image with
the spectra from the destreaked image. Are there any differences in radiance
values?

Also compare spectra from pixels which did not show any streaking in the
original image. Are these spectra also modified?

Perform an MNF transform on the locally de-streaked radiance image using the
method described in section 4.1.

Animate the MNF bands and compare with the MNF bands of the original
radiance image from section 4.2.1. How many good MNF bands are there now
compared to before de-streaking?

What happens to the spectral smile radiance effect after local destreaking ?
(Compare the first few MNF bands).

Another way to examine the effect of local destreaking on the data is to look at the
difference images between the destreaked and original data. To create the difference
image:
From the ENVI main window select > Basic Tools > Band Math

In the Band Math window > enter the expression float(b1-b2) and click OK.


In the Band Math Input File window select the original 179-band radiance
image from section 4.2.1 and click OK.

26

In the Band Math Input File window select the locally destreaked 179-band
radiance image and click OK.

Enter an output file name for the difference image and click OK in the
Variables to band Pairings window.

Load individual bands of the difference image and examine how the data have
been changed across the image.

Compare with the difference image obtained in section 4.4.1 for global
destreaking.

Perform a MNF transform on the difference image and examine the bands.

Compare with the MNF bands of the difference image for the global
destreaked image obtained in section 4.4.1

From the above results what can you conclude about local destreaking. Does it
have any adverse effect on the data in addition to removing streaks?

If there is time you can try local destreaking separately on the VNIR and SWIR
bands. Try small filter sizes (1, 2, or 3) in the VNIR and wide filter sizes (10, 15, 20)
for the SWIR. To destreak the VNIR bands select a spectral subset of these bands
(first 50 bands in the 179 band image) when entering the input image file. The
destreaking will be applied to the selected bands only and the remaining SWIR bands
will be written to the output file unchanged. This output file can then be used for
destreaking the SWIR bands by selecting the remaining 129 bands (bands 51-179) in
the input file.

Examine MNF and difference images to determine the best filter widths to use for the
VNIR and SWIR.

27
5 ATMOSPHERIC CORRECTION

Atmospheric Correction is a very big topic. In these notes there will be two currently
available methods used to apply this transformation. One is a code called ACORN
(Atmospheric Correction Now) and the other one called FLAASH (Fast Line-of-
sight Atmospheric Analysis of Spectral Hypercubes). ACORN is provided by
AIGLLC and FLAASH is produced by the group that develops MODTRAN.

Note: The atmospheric correction procedures outlined below are only for
demonstrating the functionality of the ACORN and FLAASH codes, using mostly the
default settings for input parameters. It is left to the user to experiment with the codes
and optimise their atmospheric correction results by referring to the user manuals for
ACORN and FLAASH.

5.1 Atmospheric Correction using ACORN
Section 5.1.1 describes the input data requirements for running ACORN. If you are
already familiar with using ACORN, you can go straight to Section 5.1.2 to start
using ACORN with the data provided.

5.1.1 Ancillary data required for ACORN

Input image file
Input image must be in units of radiance (W/m
2
/sr/m) or radiance with gains
and offsets applied. (Gain and offset values for all bands must be provided in a
separate file)
Image data must be 16-bit integer (either big endian (NETWORK) or little
endian (HOST or INTEL) formats)
Image interleave must be Band Interleaved by Pixel (BIP) or Band Interleaved
by Line (BIL)

(Most hyperspectral image data are supplied in the above formats e.g.
HYMAP, HYPERION)

Spectral Calibration File
A 2-column ASCII file containing center wavelength (in nanometers) of each
band in the first column, and the full-width-half-maximum (FWHM) values in
the second column. (Note: If ACORN is run as an ENVI plugin, these data are
automatically retrieved from the image header files)

Gain File
An ASCII file containing values (multiplicative factors) for each band that
converts the image integer values to radiance in watts per square meter per
steradian per micrometer (W/m
2
/sr/m). The factors used to convert the image
integer values into image radiance values need to be known before preparing
the gain file. If the image file is already in radiance units, the gain values for
each band would be 1 and in our case the gain values will be 0.01 as the data
are these radiance units multiplied by 100.

Offset file
28
An ASCII file containing numbers that must be added to the image radiance
values after the gain file has been applied. For most datasets the offset values
are zero.

(Note: the offset values can be added into a second column of the gain file,
and the appropriate column numbers can be specified for gain and offset
values when the gain file is called upon in ACORN).

A gain and offset file can be created in MS Excel or any other spreadsheet
package and saved as an ASCII file. The ASCII file should have at least 2
columns of data (gain, offset) with no header rows. This gain file must have
exactly the same number of bands as the image file you have prepared for
atmospheric correction.

Image location and acquisition parameters

Image acquisition altitude (km): (in kilometres above sea level). The altitude
from which the image was acquired should be known. (For satellite sensors
this is a fixed value, e.g. For Hyperion it is 705 km, for airborne sensors the
flying height varies and this data should be obtained for each scene). Note:
The maximum value that ACORN accepts is 100 km. This will not create a
problem since above 100 km the effect of the atmosphere is negligible.

Image mean elevation (meters): The average ground elevation (in meters
above sea level) of the scene must be known.

Image latitude and longitude. The centre latitude and longitude of the scene
must be known (in degrees, minutes, seconds or in decimal degrees). (+ values
for North and East, - values for South and West).

Image date (dd:mm:yy): The day, month, and year when the image was
acquired must be known

Image average time (UTC): The time of day when image was acquired
should be known (hour, minute, second). The time should be converted from
local standard time or daylight saving time into UTC time.

Estimate of fixed water vapor amount (mm): An estimate of atmospheric
water vapor around the time of image acquisition must be provided. (in units
of precipitable liquid water between 1-50 mm). This can be obtained from site
meteorological data. Typical values would be 15 mm for dry regions and 25
mm for humid regions. ACORN calculates water vapor amounts on a pixel-
by-pixel basis from the image itself. The user-entered estimate will only be
used if the calculated values are found to be internally inconsistent.

Image atmosphere visibility: An estimate of visibility at time of image
acquisition must be provided (e.g. 20 km for hazy conditions, or 100 km for
clear sky). ACORN calculates visibility from the image and will only use the
user-entered value if the calculated values are found to be internally
inconsistent.
29

5.1.2 Running ACORN Atmospheric Correction

From the main ENVI menu select > Basic Tools > Calibration Utilities > ACORN >
ACORN Hyperspectral Correction

In the ACORN Input File window > select the Radiance Image File (use the locally
destreaked 179 band radiance image) and click OK.

In the Enter ASCII filename containing Gain Values window > select the gain file
acorn_gain_offset.txt from \Ancillary_Data\Gain_offset files directory and click
Open.

In the ACORN Gain and Offset Values window > Enter 2 for Gain Column and 3
for Offset Column > click OK.

The ACORN Control File Parameters window appears.

(The image acquisition/location data for the sites are provided in the same directories
as the image files (e.g. \Coleambally\Image Data) as an ASCII file called
Location_Acquisition_data(site name).txt. Open this file and enter the data in the
ACORN Control File Parameters window).

In the ACORN Control File Parameters window > enter Image Acquisition Altitude
(km) > Image Mean Elevation (Meters) > Image Date > Image Time (UTC)
(hh:mm:ss) > Scene Center (Longitude and Latitude, + for N and E, - for S and W).

Select Yes for Acorn Estimate Visibility? > Enter 60 for Atmosphere Visibility
(km)
Select 1140 nm for Derive Water Vapor > enter 18for Fixed Water Vapor (mm)

For Artifact Suppression click on Type 1, 2, and 3.

30

Choose an Output ACORN Reflectance Filename and check that you have entered
everything correctly. Click OK to start the atmospheric correction process. It may take
a few minutes for the processing to complete.

Once the Atmospheric Correction is complete, the reflectance image is automatically
loaded in the Available Bands List.

(Note: To re-run ACORN with previously saved parameters use the Restore button
and the saved ACORN control file to automatically enter all input parameters)
31
5.2 Atmospheric Correction using FLAASH
Section 6.2.1 describes the input data requirements for FLAASH. If you are already
familiar with using FLAASH you can go straight to Section 5.2.2 to start using
FLAASH with the workshop data.

5.2.1 Input parameters required for FLAASH

When FLAASH is started from the ENVI main menu by selecting Basic Tools > Calibration
Utilities>FLAASH or Spectral > FLAASH, the FLAASH Atmospheric Correction Model
Input Parameters dialog appears as shown below:

Input Radiance Image and Radiance scale factors

The first step in using FLAASH is to select the input radiance image and to set the radiance
scale factors.

The input radiance file must have a BIL or BIP interleave, and should contain calibrated
hyperspectral radiance data scaled into 2-byte integers. The images spectral range must span
either 1050-1184 nm or 770-870 nm at 15 nm or finer resolution. Additional wavelength
coverage is required for aerosol retrieval.

If wavelengths and FWHM values are not defined in the input images header file, you will be
prompted to read these values from an ASCII file. The ASCII file should contain its data in
columns. Be sure to correctly specify the column and units for each parameter when reading
them.

32
After you select the input image, the Radiance Scale Factors dialog appears. Either a single
scale factor (a constant) for all bands can be entered, or an array of scale factors (one per
band) can be read from an ASCII file.

Note: Scale factors must be chosen such that the input image divided by the scale factors
converts the two-byte integer data into floating point radiance values in units of W/cm
2

/nm/sr. If the image file is already in units of W/cm
2
/nm/sr, the gain values for each
band will be 1 and in our case the gain values will be 1000 as the data are radiance
units (W/m
2
/m /sr) multiplied by 100.
(Note: 10 W/m
2
/m /sr = 1W/cm
2
/nm/sr).

Scene and Sensor Information
Scene and sensor information includes the scene center location (lat/long), the average ground
elevation of the scene, the sensor type, the sensor altitude, and the flight date and time. These
data let FLAASH determine where the sun was in the sky and the path the suns light took
through the atmosphere to the ground and back to the sensor.

Image latitude and longitude. The centre latitude and longitude of the scene
must be known (in degrees, minutes, seconds or in decimal degrees). (+ values
for North and East, - values for South and West).

Sensor Type. The name of the hyperspectral sensor that collected the input radiance
image must be known. The sensor type is used by FLAASH to set the Instantaneous
FOV (field of view). At present FLAASH allows the selection of sensors from
AVIRIS, HYDICE, HYMAP, CASI_82, CASI_84, and UNKNOWN. For
HYPERION, the UNKNOWN sensor type should be selected and then its IFOV
should be entered in the advanced settings dialog.

Sensor Altitude (in kilometres above sea level). The altitude from which the
image was acquired should be known. (For satellite sensors this is a fixed
value, e.g. For Hyperion it is 705 km, for airborne sensors the flying height
varies and this data should be obtained for each scene).

Ground Elevation (in kilometres above sea level). The average ground
elevation (in kilometres above sea level) of the scene must be known.

Flight Date. The day, month, and year when the image was acquired must be
known

Flight Time. The time of day when image was acquired should be known
(hour, minute, second). The time should be converted from local standard time
or daylight saving time into UTC or GMT time.

Atmospheric Model
From the Atmospheric Model button menu, one of the standard MODTRAN model
atmospheres should be selected. A model whose standard column water vapor amount
is similar to, or somewhat greater than, that expected for the scene should be selected.

The standard column water vapor amounts (from sea level to space) for each model
atmosphere are given in the Table 1. If no water vapor information is available, a
33
reasonable atmosphere may be selected according to the known or expected surface
air temperature, which tends to correlate with water vapor. If the temperature is
unknown, an atmosphere may be selected from Table 2, which is based on a seasonal-
latitude surface temperature model.

Table 1: Column water vapor amounts and surface temperatures for the MODTRAN
model atmospheres.

Table 2: Selection of MODTRAN model atmospheres based on latitudinal/seasonal
dependence of surface temperature.

34

Aerosol Model
From the Aerosol Model button menu, one of the standard MODTRAN aerosol/haze
types should be selected (eg., rural, urban, maritime, tropospheric).

Aerosol Retrieval
This option allows the retrieval of aerosol and estimation of a scene average visibility
using image dark pixels. Setting aerosol retrieval to no will use the initial visibility
value.

Initial Visibility Value
An estimate of visibility (in kilometres) at time of image acquisition must be provided
(e.g. 20 km for hazy conditions, or 40-100 km for clear sky). The initial visibility
value is assumed for the atmospheric correction if the aerosol is not being retrieved.

A reasonable value must be entered for visibility even if you have set the Aerosol
Retrieval parameter to Yes, because the Initial Visibility value will be used if the
aerosol cannot be retrieved.

Spectral Polishing
Spectral polishing is a post processing option for smoothing the output reflectance
image. Residual noise and artifacts are reduced to make the spectra appear more like
the true spectra of the surface. If spectral polishing is set to yes, then a smoothing
window should be specified in Width (number of bands).

Width (number of bands)
The width of the spectral polishing window is entered here. A larger number
generates more smoothing. A value of 9 is recommended for typical 10 nm-resolution
35
hyperspectral sensors (such as AVIRIS and Hyperion). A value of 2 provides minimal
smoothing but removes odd-even spectral band imbalances. Odd polishing widths are
slightly more computationally efficient.

FLAASH Advanced Settings
Clicking on the Advanced Settings button in the FLAASH input Parameters dialog
brings up the FLAASH Advanced Settings dialog. The advanced settings inputs can
be used to adjust the modelling parameters, viewing geometry, and run controls.

For the purposes of this workshop we will use the ISAACS multiscatter model and
15cm-1 modtran resolution. These settings allow a faster processing time. [However
for more accurate results the DISORT multiscatter model with 8 or 16 streams and a
modtran resolution of at least 5 cm-1 should be used. It may take several hours for the
processing to finish with these settings, but more accurate results are obtained
especially in low visibility and hazy scenes]. The IFOV for Hyperion (0.043) will also
need to be entered. All other parameters will be left with their default settings. You
may select yes to save the flaash output diagnostic files. For more details on the
various parameters in the advanced settings, refer to the FLAASH users guide.

5.2.2 Running FLAASH Atmospheric Correction

From the main ENVI menu select > Basic Tools > Calibration Utilities > FLAASH.

In the FLAASH Atmospheric Correction Model Parameters dialog > click on Input
Radiance Image > select the locally destreaked radiance file (179 bands).

In the Radiance Scale Factors dialog > click on Use single scale factor for all bands
> and enter a value of 1000 for the Single scale factor. (A value of 1000 will convert
36
the input radiance image (W/(m
2
.m.sr) x 100) into the standard FLAASH input
radiance units of W/(cm
2
.nm.sr).

Click on the Output Reflectance File button and enter the filename and path for the
output reflectance image.

Click on the Output Directory for FLAASH Files and set the directory path where all
FLAASH files except the output reflectance file will be written. These files include
the column water vapor and cloud images, the journal file, and optionally the template
file.

Click on the Rootname for FLAASH Files button and enter the prefix to be added to
each FLAASH file produced during the session. Just enter the name, do not include a
directory path here.

Enter all the scene and sensor information (Scene centre longitude/latitude, Sensor
Altitude, Ground elevation, Flight date and time). This information is provided in the
ASCII file Location_Acquisition_data in the same directories as the image files (e.g.
\Coleambally\Image Data). Note: The values for sensor altitude and ground
elevation should both be entered in kilometres.

For Sensor Type > select UNKNOWN.

Select Mid-Latitude Summer for Atmospheric Model.

Select Rural for Aerosol Model.

Set Aerosol Retrieval to Yes. Set Initial Visibility to 60 km.

Set Spectral Polishing to No.

(This output with no polishing would be useful for initial comparison with the
ACORN results. You can re-run FLAASH later with various spectral polishing
settings to see if there are any improvements. To do this you can simply use the
Restore button on the FLAASH Atmospheric Correction Model Parameters dialog to
re-use all the other input parameters set previously).

Click on the Advanced Settings button to bring up the FLAASH Advanced Settings
dialog.

Enter a value of 0.043 for Instantaneous FOV (mrad). (Hyperion has an IFOV of
0.043).

Select Yes for Automatically Save Template File.

Select Yes for Output Diagnostic Files.

Leave all other parameters unchanged (i.e. use the default settings). Click OK to close
the FLAASH Advanced Settings dialog.
37

In the FLAASH Atmospheric Correction Model Parameters dialog > click Apply to
start the FLAASH processing.

It will take a few minutes for the FLAASH processing to finish. The reflectance file
will be loaded in the Available Bands Window once the processing is complete.

6 VALIDATION OF DERIVED SURFACE REFLECTANCES

Display the reflectance images in true colour RGB.

[Note: The reflectance values have been multiplied by 10,000 in the ACORN and
FLAASH images. (e.g. a value of 4000 would represent a reflectance of 0.4 or 40%)].

Examine spectral profiles of various surface features. Note any bands with large
spiky features. Many of these will be in regions of strong atmospheric absorptions
and can be set as bad bands.

6.1 Stable Band Selection

From the 179 band image a further reduced set of stable bands could be selected for
further analysis. The basis for selection of these stable Hyperion bands and the set
of stable bands are outlined in the document Discussions around Hyperion Data:
Background Notes for the Hyperion Data Users Workshop.

A set of 155 stable bands identified for Hyperion data in that document are:

Hyperion Bands Wavelengths
(nm)
10-57 448-926
81-97 953-1114
101-119 1155-1336
134-164 1488-1790
182-221 1972-2365

Check this selection and its rationale quickly by using the ENVI animate facility
one frame at a time and looking at the bands excluded. They are generally deep
atmospheric water vapour bands.

Edit the header file for the radiance image to keep the above set of bands as good
bands and set the remaining bands as bad bands. (See Section 3.2.1 for procedure on
setting bad bands).

This 155-band radiance image will be used for all further processing.

Display and link the ACORN and FLAASH reflectance images. Extract spectral
profiles from the two images and compare for different cover types.

38
6.2 Setting up Ground Spectra in a Spectral Library

A number of spectra measured on the ground with an ASD Field Spectroradiometer
are available for some of the sites. These are provided as ASCII files in the
ASD_Spectra sub-directories for each site. The first column is wavelength in nm,
and remaining columns are reflectance (0-1) for various surface materials. The
corresponding image pixel coordinates (x,y) for each ground spectra are also given in
these ASCII files (see the column names for the (x,y) values)

To compare the ground spectra with the image spectra, the ground spectra will need to
be imported into an ENVI spectral library and convolved to the Hyperion bands.

First create a spectral library of the ground spectra using the following steps:

From the main ENVI window > select Spectral > Spectral Libraries > Spectral
Library Builder.

In the Spectral Library Builder window > select input ASCII file to select the Input
Spectral Wavelength > select the ASCII file from the ASD_Spectra subdirectory in
the Enter File Containing Output Wavelength window and click open.

In the Input ASCII file window select 1 for the wavelength column and click OK.

In the Spectral Library Builder window > select Import > From ASCII File.

In the Enter ASCII Plot Filenames window > select an ASCII file from the
ASD_Spectra subdirectory and click Open.

In the Input ASCII File window > select column 1 for the x-axis column and the
remaining columns for the y-axis columns > enter 1 for the X Data Multiplier and
10000 for the Y Data Multiplier (Factor of 10000 here will transform reflectance
values to the 0-10000 scale for easy comparison with the image reflectance spectra).
Click OK.

Repeat the above 3 steps to import all spectra from the ASD_Spectra subdirectory
for the site.

In the Spectral Library Builder window > select Options > Edit Endmember Names.

In the Edit Endmember Names window > click on each spectrum
(ASD_Spectra.txt:C2.etc.) and type in their original names (have the ASCII files
open in notepad or wordpad and use the column names for the spectra to edit the
library endmembers) > Click OK.

In the Spectral Library Builder window > select File > Save Spectra As > Spectral
Library > Enter a filename in the Output Spectral Library Name window > Click OK
to create the spectral library.

Use Spectral > Spectral Libraries > Spectral Library Viewer to look at the spectra
from the library you just created.
39

6.3 Resampling the Spectral Library to Hyperion Bands

The spectral library (with 2151 spectral bands) will now be convolved or binned to
the 155 stable Hyperion Bands. (See the section on Theory of Binning in
Discussions around Hyperion Data:Background Notes for the Hyperion Data Users
Workshop for details on binning from high spectral resolution sensors to broad
bands).

The band pass function for the binning of ASD spectra to Hyperion wavelengths has
been set as a User Defined Filter Function. This User Defined Filter Function has
been created using a similar procedure to that used described later in section 7.2.1.1.
The filter function is a spectral library of binning weights (2151 rows x 155 columns).
The procedure for creating the Filter Functions is the same as creating a spectral
library from ASCII files (where the binning weights for each Hyperion band are
stored as individual spectra).

Use the following procedure to convolve the spectral library you just created in
Section 6.2 to the equivalent 155 stable Hyperion bands:

From the main ENVI window select Spectral > Spectral resampling > In the Spectral
resampling input file window > select the spectral library you created in Section 6.2
and click OK.

In the Spectral Resampling Parameters window > Select User Defined Filter
Function and enter an output filename for your resampled library (e.g.
ASD_to_Hyperion155) > click OK.

In the Input Filter Function Spectral Library select the file ASD_TO_HYPERION
from the \Ancillary data\User Defined Filter Functions subdirectory. Click OK to
create the new library.

6.4 Comparing Ground and Image Spectra

Use Spectral > Spectral Libraries > Spectral Library Viewer to display the spectra
for various ground points from the 155-band spectral library created in section 6.3.

Extract the image reflectance spectra for the same pixels. From the main image
display use Tools > Profiles > Z-profile (spectrum) to display an image spectrum.
Then use Tools > Pixel Locator from the main image display to bring up the pixel
locator window. Enter the pixel (x) and line (y) numbers and click apply to display
the spectra of individual pixels in the image spectral profile. Use the (x,y) values from
the spectra names given in the ASCII files in the ASD_Spectra subdirectory to get
the corresponding image spectra.

Compare the image and library spectra by dragging a library spectrum into the image
spectral profile window. (Right click in the spectral library plot window to display the
spectra names. Then left click on a library spectrum name and drag it to the image
spectra profile window). Compare the image and library spectra in terms of
reflectance amplitude, noise and shape and position of spectral features.
40

7 Advanced processing

7.1 Spectral Smile

For a detailed discussion of the spectral smile effect in Hyperion data, refer to the
section on Spectral Smile (pages 9-11) in the document Discussions around
Hyperion Data: Background Notes for the Hyperion Data Users Workshop.

The spectral smile is a low frequency effect that is caused by the variation in central
wavelength and FWHM across a line of pixels. This effect in pushbroom sensors such
as Hyperion is caused by the non-uniform spectral response across the elements of the
detector arrays in the VNIR and SWIR regions. The spectral smile effect is more
pronounced in the VNIR that the SWIR wavelengths in Hyperion Data.

The central wavelengths and FWHMs across the 256 pixels for all 242 Hyperion
bands are available from the pre-launch calibration data. In the Hyperion level 1B1
data, the central wavelengths and FWHMs of the central pixel (pixel 128) are
provided in the image header files. For more recent level 1R data the image header
files provide the average values of the central wavelengths and FWHMs across the
256 pixels. Most applications based on broad spectral channels would not be affected
too much by the spectral smile effect. However, many hyperspectral applications of
the data dealing with fine spectral features such as wavelength position, slope and
inflection points of atmospheric and surface absorption features would be affected by
non-uniform central wavelengths and FWHMs across the 256 pixels.

The smile effect appears in higher order MNF bands (see MNF bands 1 or 2 from
section 4.1). Another way to check for the smile effect is to look at the band
difference images around atmospheric absorption features such as the Oxygen A
absorption near 762 nm. Try the following band difference between bands 40 and 42
(752 nm and 772 nm) on either side of the oxygen A feature near 762 nm. Use the
radiance image created in section 4.1.

Smile effect across Oxygen A absorption feature near 762 nm
Basic Tools > Band Math
Enter the expression float (b1-b2) and click ok.
Then select band 40 (752 nm) for variable b1 and band 42 (772 nm) for b2
Enter a filename to save your result and click ok.
Load the output image as a grayscale image and examine the variation in
brightness across the 256 pixels.

Some form of correction for the spectral smile effect is therefore important. One
approach for reducing the smile effect would be to apply atmospheric corrections
separately on each of the 256 individual columns of data, using central wavelengths
and FWHM for each column. However this method would be computationally
intensive. Another approach is to interpolate the image radiance data to the same set
of central wavelengths and FWHMs across all 256 pixels of the image swath. The
interpolation method would be tried out in this workshop.

41

7.1.1 De-smiling Hyperion data

In this workshop the radiance data from individual pixels in each spectral band are
interpolated to the common set of wavelengths defined by the average wavelength of
all 256 pixels. The method reads in the radiance data for each pixel, and uses the
central wavelength values for each pixel from the SpectralL0_revA_dat file (the
ASCII file in this program uses the transpose of the SpectralL0_revA_dat file) as the
input set of wavelengths for all bands, and interpolates the radiance to the output set
of wavelengths (read in from a separate file containing the average wavelengths
across all 256 pixels). The cubic spline interpolation method is used here.

In the normal processing sequence for Hyperion data, the de-smiling is best applied
following the fixing of outlier pixels (section 4.2) and before applying the destriping
(destreaking) procedure (section 4.3).

From the ENVI main menu select > Workshop> Interpolate Wavelengths (de_smile)
In the Hyperion L1B1 input file window select the 179-band image created in 3.4 above and
click OK.

(The input image should be in BIL format only. The method does not work with other file
types at present).

Then select the De_smile output file name and click ok. [Note: The save to memory option
does not work in the current version so users are advised to save the output to a file].

Next Select the column wavelengths (CWL) (.txt) file and click ok. The file
cwl(179).txt is provided in the \Ancillary data\Hyperion_Smile directory on CD1.
(This file contains the central wavelengths for each of the 256 pixels in columns:
(pixel 1 in Column 1, . to pixel 256 in column 256). The rows represent the 179
bands in ascending order.)

Next Select common wavelengths (AV_CWL)(.txt) file and click OK. The file
avwvl(179).txt is provided in the \Ancillary data\Hyperion_Smile directory on CD1.
(This file contains a single column of average central wavelengths for the 179 bands
across the 256 pixels).

The processing will take a while to finish.

7.1.2 Effect of De_smiling on MNF transform

The MNF band 1 (or 2) or the band difference between bands 40 and 42 usually show
the radiance effect caused by the spectral smile.

The MNF transform of the de-smiled data can be used to evaluate the reduction in the
spectral smile effect following the de-smiling by interpolation.

Perform a MNF transform on the VNIR bands in the 179-band de-smiled radiance
image from section 7.1.1.
42
From the Main ENVI window select > Transform > MNF Rotation > Forward
MNF > Estimate Noise Statistics From Data. Select the filename and subset to
first 50 VNIR bands and click Enter.

In the Forward MNF Transform Parameters window > select no for select
subset from eigenvalues and enter 50 for number of output MNF bands.
Click ok. It will take a while for the MNF transform to finish.

Perform a MNF transform on the SWIR bands.

Load and display the MNF bands as grayscale images. Load the MNF bands from
section 4.1. Compare the MNF bands and note the changes (especially in VNIR Band
1 or 2) caused by the de-smiling process.

Are the streaks still present in the MNF bands?

Also try the band difference between bands 40 and 42 and compare with the results
obtained in 7.1 above.

7.2 Broad And Narrow Band Comparisons

7.2.1 Synthesis of LANDSAT and ALI bands from Hyperion Bands

A set of spectral bandpass functions for the binning of ASD bands to Hyperion, ALI
and LANDSAT ETM bands are provided in the directory \Ancillary
Data\Binning_Weight_Matrices. The three ASCII files for Hyperion, LANDSAT and
ALI sensors are ASD2Hyp.txt, ASD2ALI.txt, and ASD2ETM.txt respectively. The
file ASD2Hyp_179bands.txt is also provided for binning ASD data to the subset of
179 Hyperion bands used in this workshop. These bandpass functions are all based on
the ASD set of wavelengths at 1 nm step from 350 to 2500 nm.

These binning weight matrices can be conveniently used by first setting them up as
User Defined Filter Functions (i.e. Spectral libraries) in ENVI.

7.2.1.1 Building User Defined Filter Functions

The user defined filter functions are created in the same way spectral libraries are
created in ENVI.

From the main ENVI window > select Spectral > Spectral Libraries > Spectral
Library Builder.

In the Spectral Library Builder window > select Input ASCII File to select the Input
Spectral Wavelength.

In the Enter File Containing Output Wavelength window select the ASCII file
ASD2ETM.txt and click open.

In the Input ASCII file window select 1 for the wavelength column and click OK.
43

In the Spectral Library Builder window > select Import > From ASCII File.

In the Enter ASCII Plot Filenames window again select the file ASD2ETM.txt and
click Open.

In the Input ASCII File window > select column 1 for the x-axis column and the
remaining columns for the y-axis columns > enter 1 for the X Data Multiplier and
1 for the Y Data Multiplier. Click OK.

In the Spectral Library Builder window > select Options > Edit Endmember Names.

In the Edit Endmember Names window > click on each spectrum (ASD2ETM..txt:C2,
.etc.) and type in the LANDSAT band names at the end (i.e. replace the C2,
C3,in the names with Band 1, Band 2) > Click OK.

In the Spectral Library Builder window > select File > Save Spectra As > Spectral
Library > Enter a filename (e.g. ASD_to_LANDSAT) in the Output Spectral Library
Name window > Click OK to create the spectral library.

Repeat the above procedure to create the filter function for Hyperion and ALI bands
using the ASD2Hyp_179bands.txt and ASD2ALI.txt ASCII files.

(Note: For ASD to Hyperion we will only use the file ASD2Hyp_179bands.txt for
creating the 179 bands used in this workshop. The file ASD2Hyp.txt is also provided
should you need to create a filter for binning ASD to the full 242 Hyperion bands).

Use Spectral > Spectral Libraries > Spectral Library Viewer to plot the bandpass
functions for the Hyperion, LANDSAT and ALI bands that you have just created.

7.2.1.2 Creating Filters for Binning Hyperion data into LANDSAT ETM
and ALI bands

The filters created in section 7.2.1.1 above are base filters that can be used to bin
ASD spectra into a number of broader bands.

To bin Hyperion image data into broad multispectral bands such as ETM and ALI, we
would need filter functions based on Hyperion bands (ie. Hyperion to ETM and
Hyperion to ALI filters). Such filter functions can be created using the base filters and
the ENVI module Generating Binning Formulae provided in the Workshop Menu.

To create the filter for binning Hyperion to LANDSAT ETM bands:

From the main ENVI window select Workshop > Generating Binning
Formulae and click Go.

Then select the base bandpass filter (this would be the ASD to Hyperion filter
(spectral library) created earlier. Click OK.

44
Review the base filter information displayed on screen and click yes to
continue.

Then select the target filter (ASD to ETM spectral library) and click ok.

The Stability of the Approximation window comes up next. Examine the
eigenvalues and condition number and click ok.

In the Type and File output Settings window select Simple binning type
and save the filter and error files. Click OK.

Then select output filenames for the filter and error files when prompted for
each one and click open to save each file.

Repeat the above procedure to create the filter for binning Hyperion to ALI
(use the ASD to Hyperion filter from 7.2.1.1 as the base and the ASD to ALI
filter from 7.2.1.1 as the target).

From the main ENVI menu select Spectral > Spectral Libraries > Spectral
Library Viewer to plot the bandpass filter functions for the Hyperion to
LANDSAT and Hyperion to ALI bands that you have just created.

7.2.1.3 Examining Filter Statistics

The Spectral Library Viewer in ENVI provides an option to view plots of filter
weights as a function of wavelength. For a more detailed examination of filter
statistics, the workshop menu item List Filter Properties can be used. This ENVI
module computes a range of filter statistics, which can be saved as a CSV file and
examined in a spreadsheet such MS Excel. The statistics computed for each filter
band are the Sum of Filter weights, Mean wavelength, Standard Deviation, Minimum
Value, Maximum Value, Central Wavelength, FWHM for each Filter Band (true), and
FWHM-p for each Filter Band (fast form).

To compute the filter statistics:
From the main ENVI menu select Workshop > List Filter Properties and click
Go.
Select your input filter file (ENVI Spectral Library) and click OK.
Check the filter information listed on screen and click yes to continue.
Click yes to save statistics to a file.
Enter a filename for the statistics and click Save to continue. Click Ok and
then yes to get statistics for another filter or no to finish.

To view the saved statistics file, open in a spreadsheet such as MS Excel and
examine the statistics for each output band of the filter.

7.2.1.4 Binning Hyperion bands into Landsat7 and ALI using ENVI
spectral resampling

To bin the Hyperion Bands to Landsat:

45
(NOTE: For this part you should use the locally destreaked 179-band Hyperion
radiance image created in Section 4.3.2).

From the main ENVI window select Spectral > Spectral resampling.

In the Spectral resampling input file window > select the radiance image (179 bands)
and click OK.

In the Spectral Resampling Parameters window > Select User Defined Filter
Function and enter an output filename for your binned image (e.g.
Hyp179_to_ETM) > click OK.

In the Input Filter Function Spectral Library select the filter you created in section
7.2.1.2 for binning Hyperion to Landsat. Click OK to create the synthesised Landsat
Bands.

Repeat the above procedure to create ALI bands from Hyperion.

Open and display the synthesised LANDSAT and ALI images. Look at grayscale and
RGB combinations. Also display the PAN bands. Note that the radiance data are
scaled as were Hyperion radiances.

Try some simple transformations such as band ratios of NIR and RED bands and
NDVI.

If there is time try some simple classifications (supervised and unsupervised) to map
the distribution of different cover types.

In theory, if you also have the Landsat and/or ALI data for the same time the
corresponding pixels should have the same values to within 5% after allowance for
calibration and scaling. When you get your data in the future you can test if this is so.

7.2.1.5 Binning Hyperion bands into Landsat7 and Ali using the Workshop
Program-Apply Filter to Images.

Repeat the binning from section 7.2.1.4 using the ENVI Workshop menu Apply
Filter to Images.

This procedure resamples the hyperion data into Landsat ETM and ALI bands by
applying the filters created in section 7.2.1.2 without any further modifications to the
binning filters.

The ENVI resampling as applied in section 7.2.1.4 appears to modify the filter and
resample the data at the same time.

Compare the resampled images from sections 7.2.1.4 and 7.2.1.5 to see what
differences are caused by the two different methods. Use band differences and
comparison of band central wavelengths to determine if there are significant
differences in the two types of resampling.
46

{Binning is an ongoing area of research at present. Further work will determine if the
ENVI spectral resampling method needs to be modified to provide accurate
resampling.}

8 GEOMETRIC CORRECTION OF HYPERION DATA

As described in Discussions around Hyperion Data:Background Notes for the
Hyperion Data Users Workshop, the Hyperion geometry is straightforward to map to
a geographic coordinate system. The delivered Hyperion image data are not
georeferenced and users will have to use data for their own sites (GCPs, maps, etc.)
for geometric corrections.

This may be discussed if anyone wishes to do so, otherwise this section will not be
covered in this workshop.

9 INFORMATION EXTRACTION DEMONSTRATIONS

For this part of the workshop there will be some demonstrations of applications and
information products derived from processed Hyperion data. Areas covered will be
generally from Vegetation, Minerals and Water applications. The exact topics to be
covered at a particular workshop will be announced at the start of the workshop.
47
APPENDIX: NOTE ON ENVI BAND MATHS TO OBTAIN CALIBRATED
IMAGE DATA

The rather complicated expressions used in the band math example to derive a single
factor for scaling between the image data and radiance arises because of the issues of
scaling and IDL function behaviour.

The basic Hyperion data are recorded to 10 bits and output as digital numbers in the
range 0-4095. These are proportional to current or voltage in the amplified detector
output. These are dark corrected (including subtraction of offsets used to scale the
data for recording), calibrated and the data files are maintained as integer data so that
storage is controlled. A real valued image would be twice the size of an integer image.
However, in order to retain the data precision the integer results of the calibration are
multiplied by 40 in the VNIR and 80 in the SWIR. Otherwise information recorded by
Hyperion especially over bright areas such as cloud tops - would be lost.

In a similar way, if we were to divide the VNIR and SWIR by 40 and 80 and store the
results as integers in an image then the precision would be lost. Hence by scaling back
up by 100 in the VNIR and the SWIR the precision is retained and the scaling made
uniform for the different wavelengths. As described before, uniform scaling is
essential for binning and useful for other purposes.

The reason Hyperion processing does not scale by 100 is that in this case very bright
cloud tops would saturate in the VNIR and emission by lava flows will saturate in the
SWIR (perhaps even at a multiple of 80). Neither of these cases is likely to be of
interest in the applications being described here so we have settled on 100. If these
other cases are of interest then you should review the method.

To maintain maximum precision we would like the data to be transformed in reals and
the result rounded to integer. The objective is that the output image be BIL and signed
Integer (-32768 to 32767). However, in Band Math, the decisions are not left to us as
users. We have had to accept that the data output is BSQ and that we will have to
convert the file before using it in ACORN or FLAASH.

The data type has to be carefully forced by the choice of expression. The following
file data types result from the expressions shown:

Expression Result
B1/40 Integer but b1=41 to 79 gives 1
B1/40.0 or float(b1)/40.0 Real
Round((float(b1)/40.0) Long Integer but b1=21 to 60 gives 1
Round(100.0*(float(b1)/40.0)) Long Integer
Fix(100.0*(float(b1)/40.0)+0.5) Integer and rounded

The final rather complex expression provides a data type that can be used in FLAASH
and ACORN provided you transform back from BSQ to BIL.
48
References

1. J. Pearlman, S. Carman, C. Segal, P. Jarecke, and P. Barry, Overview of the
Hyperion imaging spectrometer for the NASA EO-1 mission, in Proc. Int.
Geoscience and Remote Sensing Symp. (IGARSS01), Sydney, Australia, 2001.

2. S. G. Ungar, Overview of EO-1, the first 120 days, in Proc. Int. Geoscience and
Remote Sensing Symp. (IGARSS01), Sydney, Australia, 2001.

3. Jupp, D.L.B, B.Datt, J. Lovell, S.Campbell, E.A. King, et al. (2002) Discussions
around Hyperion Data: Background Notes for the Hyperion Data Users
Workshop, CSIRO Earth Observation Centre, Canberra.

49

Hyperion Data Processing Instructions

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Hyperion Data Processing Instructions

Uploaded by

Copyright:

Available Formats

CSIRO Office of Space Science & Applications

Earth Observation Centre

You might also like