Available online at www.sciencedirect.

com

Acta Astronautica 54 (2004) 455 461 www.elsevier.com/locate/actaastro

Automated detection of orbital debris in digital video data from a telescope

Thomas J. Heberta; , John Africanob , Gene Stansberyc
a Dept.

of Electrical and Computer Engineering, The University of Houston, Houston, TX 77204-4793, USA b Boeing North America, Houston, TX 77058, USA c NASA Johnson Space Center, Houston, TX 77058, USA Received 8 August 2000; received in revised form 19 December 2002; accepted 20 March 2003

Abstract As part of a program to measure, monitor, and predict the orbital debris environment the Orbital Debris Program O ce of the NASA Johnson Space Center (JSC) collects video data through a 3-m zenith-staring liquid mirror telescope located in New Mexico. This paper presents methods and results from a PC-based digital video processing system that automates the detection and measurement of orbital debris and meteors in these videotapes. Results using the automated system were compared to those from two trained observers in reviewing 40 h of video. These results demonstrate that the computer-automated system outperforms the combined results from two trained observers, achieving a 17 percent improvement in the detection rate of orbital debris per tape and a three-fold improvement in the detection rate of meteors. Published by Elsevier Ltd.

1. Introduction For the purpose of determining the orbital debris environment, the Orbital Debris Program O ce at the NASA Johnson Space Center makes use of a number of radar facilities as well as two optical telescopes, the Liquid Mirror Telescope (LMT) and the CCD Debris Telescope (CDT). The zenith-staring 0:46 eld of view (FOV) LMT is located near Cloudcroft, NM. It has a 3-m rotating parabolic dish that uses mercury as the re ective surface. Under favorable viewing conditions, stars of greater than 17.0 stellar magnitudes are observable.
Corresponding author. Tel.: +1-713-743-4448; fax: +1-713743-4444. E-mail address: thebert@uh.edu (T.J. Hebert).

Some debris objects re ect sunlight well, but have a weak radar signature. The reverse is also true. Radar and optical telescopes o er somewhat di erent samples of the orbital debris population. Optical data is important because it provides an additional set of measurements that are independent of radar. While radar is better at detecting debris in low earth orbits, optical telescopes hold the upper hand for debris at very high altitudes. A more complete picture of the orbital debris environment is possible if both radar and optical measurements are used. These combined observations provide quantitative information that can be used to estimate the number of debris particles per size in low earth orbit (LEO). This distribution provides the basis for estimating risks of debris impacts associated with space shuttle ights,

0094-5765/$ - see front matter Published by Elsevier Ltd. doi:10.1016/S0094-5765(03)00171-1

456

T.J. Hebert et al. / Acta Astronautica 54 (2004) 455 461

the space station, satellite orbits, and other space activities. The NASA-JSC LMT is operated during the hours just before dawn and just after sunset. During these times, the sky is dark, while orbital debris above the earths shadow is still illuminated by sunlight. Under favorable weather conditions, objects of su cient size and re ectivity relative to their altitude can be observed as regions of heightened brightness traversing the FOV at constant velocity. In order to archive these observations for later analysis, up to 4 h of digital video are recorded per 24-h period. Prior to the completion of the method presented in this paper, each digital videotape was manually reviewed by two reviewers. The process of reviewing 23 h tapes that primarily show stars drifting slowly across the FOV is a tedious one. Each reviewers performance varies on a day-to-day basis, tending to decline over time. Further, reviewer performance varies throughout the course of a 23 h tape. Therefore, this task it ideal for computer automation. The development of algorithms for detecting orbital debris in digital video data is closely related to the detection of moving objects in sequences of images from a telescope. In [1], image triplets are processed to detect near earth objects. A computer system employing four CPUs is used to search four image triplets in parallel. Several minutes of computation are required to process an image triplet. In [2], an automated algorithm is used to search sequences of images from a 1-m telescope for debris in geosynchronous orbit. The estimated size and orbit of detected debris is used to construct and validate models characterizing the debris in orbit [36]. 2. System hardware The front end of the video data from the telescope is a Sony Digital Video (DV) format [7] camcorder that digitally records an intensied image from a micro channel plate as compressed 720 640 pixel video at 29.97 frames per second onto a Sony DV format magnetic tape. The DV format is a variant of motion JPEG compression scheme. The basic hardware system is a dual-processor 500 MHz Pentium III PC. The compressed video segments are loaded from digital tape onto the computer. One hour of DV compressed

video stripped of audio comprises about 12-Gb of storage, so an 18-Gb drive is su cient to hold both an hour of video plus the output from the detection software (SW). A 9-Gb drive is su cient as a storage unit 1 h of compressed video plus the detection 2 output. The PC-systems are equipped with two 18-Gb ultra-wide SCSI drives and two 9-Gb ultra-wide SCSI drives. The transfer from digital videotape to computer is done via an IEEE 1394 interface using automated SW. The transfer of compressed video to the computer is done without loss of information.

3. The video data Upon decoding, each image pixel is associated with three 1-byte values, one each for red, green, and blue. The DV format compresses the red, green, blue (RGB) information into a xed 120,000 bytes per image frame. Each image frame is actually composed of two elds; an even eld comprising the even numbered horizontal lines and an odd eld. Fields are recorded sequentially at 59.94 elds per second. A weighted sum of these three values yields a single intensity value per pixel. The transfer of DV video from digital magnetic tape onto is controlled via a dialog-based software application written in Visual C++. The user interface contains user-instructions, progress-indicators, and error-reporting. When the video transfer is initiated, the software creates the appropriate structures, arrays, and pointers to structures and data bu ers. Flags dening the audio sampling frequency and audio format are initialized. Resources are allocated, the channel is opened, and the number and IDs of nodes on the bus are determined. The VCR is then prompted to play. The receiver is opened and the processing of I=O events is initialized. When event signals indicate that the data bu ers are loaded, the compressed audio bytes are stripped and the remaining compressed video bytes appended to the computer video le. When the appropriate number of frames has been stored in the rst compressed video le, the le is closed, the second video le is opened and video transfer continues without interruption. When completed, the VCR is automatically prompted to stop and all resources are released.

T.J. Hebert et al. / Acta Astronautica 54 (2004) 455 461

457

through the LMT. Note how the elongation of the object increases with decreasing altitude. A low altitude object crosses the FOV much faster than one at high altitude. A fast moving object (low altitude) traverses 1 a number of image pixels within the 30 s over which the CCD integrates one image frame. During this time, the object contributes intensity to all image pixels that it crosses, resulting in elongation. High altitude (slow moving) objects above 4000 km have no elongation, as they move no more than one pixel within the time it takes to integrate one image frame. Note that the elongated debris objects consist of an alternating pattern of bright and non-bright lines. To understand this, 1 recall that each frame is composed of a 60 s even-line 1 eld and a 60 s odd-line eld. This is because the object activates the frames even-line eld during the 1 rst 60 s and it activates the frames odd eld during 1 the second 60 s. Fig. 2 shows a thin strip of image pixels along the path of a debris object in a sequence of 50 frames. In each image strip, two slanted lines bracket the debris object. Note how the brightness of the object changes from frame-to-frame. Most debris objects are of irregular shape, and many tumble as they cross the FOV. These observations support the following model for the kth frame f(x; y; k), f(x; y; k) = feven (x; y; k) + fodd (x; y; k); where feven (x; y; k) and fodd (x; y; k) are the even and odd elds of the kth image frame such that feven (x; y; k) = 0 for all odd y and fodd (x; y; k) = 0 for all even y. Let heven (x; y; x0 ; y0 ) denote the even-eld two-dimensional (x; y) point spread function (PSF) for a debris object that is centered at image coordinates (x0 ; y0 ), dened such that heven (x; y; x0 ; y0 ) = 0 for all odd y. Then feven (x; y; k) =
iFOV tk+0:5 tk

Fig. 1. Six debris objects at six di erent altitudes that were observed through the LMT are shown. Note how the elongation of the object decreases with increasing altitude.

4. Modeling the digital video data A wide range of algorithms for detecting debris/meteors in digital video from a telescope is possible. Any algorithm, even the simplest, will meet with some level of success. Before formulating an algorithm for detecting debris/meteors, it is important to rst determine the characteristics and variability of the signature presented by debris/meteors in the digital video data. Debris visible in an image frame appear as faint/bright clumps of pixels. Fig. 1 shows six di erent debris objects at six di erent altitudes observed

si heven (x; y; xi; t ; yi; t ) dt

x; n t + x n ; y; n t + yn )dt

+ +

tk+0:5 tk tk+0:5 tk

dn (t) heven (x; y;

[beven (x; y; t) + neven (x; y; t)] dt;

458

T.J. Hebert et al. / Acta Astronautica 54 (2004) 455 461

tk+0:5 . The odd eld of the kth image frame is dened in similar fashion.

5. Overview of an algorithm for automated debris detection Except for the problem of attention span, human observers are adept at detecting debris in video segments. Before designing a computer-automated algorithm for detecting debris, a good question to ask is what visual cues are human observers keying on? Firstly, human observers do an excellent job of ignoring the star eld, so observers perform a mental removal of the star eld. Secondly, if a feint debris object crosses the FOV in, say, 60-frames but is only intermittently observable in a few frames, the human observer is still able to detect it. What cues make this possible? It is necessary for the debris object to be brighter than the sky-background. But an even stronger cue for the human observers seems to be that the object (even if intermittent) locations must be consistent with uniform linear motion. Based upon the image frame model above, the debris detection and tracking algorithm consists of six steps. Step 1: Perform the preliminary computation of the image characteristics. Before the detection process begins, two subroutines are executed to determine characteristics of the image frames. The rst subroutine determines which image pixels are within the camera FOV. A small optimization problem is solved to yield the circle of minimum radius and its location in the FOV such that all non-zero pixels within the processed binary image are inside the circle. Given this circle, a vector with the indices of these pixels is generated. In all steps that follow, only pixels within this FOV are processed, reducing all computation time by about 20%. The second preliminary subroutine determines the rate at which the stars cross the FOV and the camera rotation angle. Step 2: Estimate the star eld and sky-back ground. Update an estimate of the star eld plus skybackground s(x; y;::: and subtract it from f(x; y; k) k) to form an image f(x; y; k).

Fig. 2. A sequence of thin strips from 50 image frames showing the path of a single debris object. In each image strip, two short slanted lines bracket the debris object. Note how the brightness of the object changes from frame-to-frame.

where si is the intensity of the ith star located at image coordinates (xi; t ; yi; t ) at time t in the FOV, dn (t) is the intensity of the nth debris object at time t (dn (t) = 0 if no debris present), x; t and y; t are the horizontal and vertical velocity in pixels-per-frame of the debris object, beven (x; y; t) is the sky-background intensity image at time t, neven (x; y; t) is the background noise image at time t, and the camera CCD forms the even eld of the kth frame during the interval from tk to

T.J. Hebert et al. / Acta Astronautica 54 (2004) 455 461

459

The star eld plus sky-background component of f(x; y; k) is given by s(x; y; k) =

tk+0:5 tk

beven (x; y; t) dt
tk+0:5

object at altitude a (as in Fig. 1) and is oriented at ::: the appropriate angle for inclination . Form h (x i; y j; am ; n ) for 7 altitude bins am , 100 inclinations n , and 8000 image locations (i; j). h (x i; y j; am ; n )
::: tk+0:5 tk tk+0:5 tk

+
iFOV

tk tk+1

si heven (x; y; xi; t ; yi; t ) dt

= +

heven (x; y; heven (x; y;

x; n t

+ x n; + x n;

y; n t

+ yn ) dt + yn ) dt;

+
iFOV

tk+0:5

si hodd (x; y; xi; t ; yi; t ) dt:

x; n t

y; n t

To form an estimate s(x; y; k) of s(x; y; k), let the rate of drift of the star eld through the eld of view be such that it takes r frames for a star to drift horizontally by a distance of 1 pixel. If no debris/meteors were detected in the k 4r, k 4r + 1, k 5r, and k 5r + 1 frames, then the rst order star eld plus sky-background image is formed as 1 s(x; y; k) = [f(x 4; y; k 4r) 4 + f(x 4; y; k 4r + 1) + f(x 5; y; k 5r) + f(x 5; y; k 5r + 1)]: Additional minor modications are used to improve s(x; y; k). If debris/meteors were present in any of the necessary frames, a shifted and processed version of s(x; y; k 1) is used for s(x; y; k). Given estimate s(x; y; k), removal of s(x; y; k) from f(x; y; k) then involves several steps. First, form ::: f(x; y; k) = f(x; y; k) s(x; y; k). Then zero-out pix::: els in f(x; y; k) that are within the stars in s(x; y; k) (dened as pixels 1:4 of the average:::pixel value in s(x; y; k). Zero-out negative values in f(x; y; k). Find the maximum value and average non-zero value in ::: ::: f(x; y; k). Zero-out f(x; y; k) pixels that are below the average non-zero value and clip pixel values above (0:85 maximum + 0:15 ave-non-zero + 0:5). Finally apply a 2 2 max lter. Step 3: Apply a library of matched lters. Image ::: f(x; y; k) only contains debris signatures plus residual noise that can be quite signicant. To form a model of ::: f(x; y; k), consider ::: set of spatially invariant debris a response functions h (x; y; a; ) whose shape (centered at x; y) has the appropriate elongation for a debris

where x; n ; y; n ; x n ; yn are chosen to match the signature from an object at altitude am , inclination n , and visible at image coordinates i; j. This constitutes ::: a library of 5,600,000 matched lters. Since f(x; y; k) only contains debris signatures plus residual noise, it can be modeled as f(x; y; k) =
i ::: j :::

di; j; m; n; k h (x i; y j; am ; n ) + n(x; y; k);

:::

where di; j; m; n; k = 0 if no debris with altitude am and inclination n is visible at image coordinates i; j in ::: the kth frame. Noise n(x; y; k) is not assumed to be ::: zero mean. Each h (x i; y j; am ; n ) is then the matched lter for objects within altitude bin am at inclination n and located at image location (i; j) in ::: frame k. Functions {h (x i; y j; am ; n )} are the set of matched lters that sample the space of all possible optical signatures from debris within the FOV of the telescope. The values i; j; m; n; k =
x;y

f(x; y; k)h (x i; y j; am ; n )d xdy

:::

:::

are indicators of the presence or absence of the corresponding debris. A maximally e cient computation strategy taking advantage of operations common to subsets within the lter library is used to compute the 5,600,000 matched lter outputs for each image frame. Step 4: Find the maximum matched lter output for each altitude bin. Compute the seven maximum matched lter outputs. max; m k = maximumi; j; n {i; j; m; n; k }, one for each of the seven altitude bins indexed by m. Theremax; fore, each m k has associated with it a particular inclination k and a location ik ; j k in frame k. At the kth frame, the history of values

460

T.J. Hebert et al. / Acta Astronautica 54 (2004) 455 461

max; k k k max; kN kN kN (m k ; m ; im ; jm ); : : : ; (m kN ; m ; im ; jm ) where N = 39 are maintained in memory. Step 5: Estimate sample means and variances. Upmax; date the sample mean and variance for m k in each max; k altitude bin. Sample means { m } and standard demax; max; viations { m k } for m k in each altitude bin are max; k updated using past m outputs

constrained minimization [8,9]

MINIMIZE with respect to xm ; ym ; x; m ; y; m k k (im xm )2 + (jm ym )2 + kp kp (im p x; m xm )2 + (jm p y; m ym )2 + kq 2 kq 2 (im q x; m xm ) + (jm q y; m ym )

max; k = m

1 28

k15 j=k42

max; m j

max; k m

1 28

k15 j=k42

max; [m j

max; k 2 ] m

Step 5: Apply a dual-statistic hypothesis test to determine whether a debris object is present in the current plus past frames. For each altitude bin, compute two statistics. The rst statistic tests the null hypothesis that a debris object is not in that altitude bin, the second tests the null hypothesis that an object is in that altitude bin. If the rst statistic is within its critical region and the second is not, then the hypothesis that a debris object is present is accepted. The rst statistic is a function of the output of the maximum max; matched lter m k for the corresponding altitude max; max; bin, its estimated mean m k , and variance m k when no debris is present. If a debris object is present, the computed value for this statistic is thus dependent upon the brightness of that object in the current max; frame. Under the null hypothesis that m k is a typical value for frames where no debris object is present, max; max; max; (m k m k )= m k is a simple unit normal random variable. The null hypothesis is tested using an 85% one-sided condence interval. To compute the second statistic, the algorithm checks all combinations of the current plus any two past (inclination, image position)s for their t to a parametric model for an object moving linearly and uniformly accordn n ing to im = (n k) x; m + xm and jm = (n k) y; m + ym for some motion parameters x; m ; y; m ; xm ; ym . max; k k k Using the current (m k ; m ; im ; jm ) plus each max; kp kp kp max; kq pair (m kp ; m ; im ; jm ); (m kq ; m ; kq kq im ; jm ) from the past 39 frames, rst check kp kq whether the inclinations m and m di er from k m by less than 10 . If so, nd the parameters x; m ; y; m ; xm ; ym that yield the best t by solving the

where the minimization is constrained to the region of velocities x; n ; y; n that lie within the minimum and maximum for objects within the mth altitude bin. Under the null hypothesis that the same debris object is visible in the current kth frame plus both the pth and qth frames, these three locations should line up well with x; m ; y; m ; xm ; ym to within the resolution of the sampling in image location. This sampling is done via the matched lters, so the resolution in image location for each altitude bin is one half the length of the non-zero portion of the corresponding matched lter, designated here as rm . Under the kl null hypothesis, the values (im l x; m xm )= rm kl and (jm l y; m ym )= rm are well-modeled as zero-mean unit-variance Gaussian random variables. The sum of the resulting six squared terms is a chi-squared variable with two degrees of freedom (rather that six) as four parameters x; m ; y; m ; xm ; ym are estimated from the values. This second statistic is used to test the null hypothesis at the 75% condence level. The two statistics are statistically independent, so the condence level for the combined two-statistic test is [1:0(1:00:85)(1:00:75)]100%=96:25%. Step 6: Track detected debris objects. At each frame, the Tracking process is in one of ve states. State (1) No object is being tracked, no objects are visible in the current frame. State (2) No object is being tracked, but an object is detected in the current frame. The object is assigned a label and tracking is initiated. State (3) An object is being tracked and it has been detected in the current frame. The estimate of its motion is updated. State (4) An object is being tracked, it has not been detected in the current frame, and its location is estimated to be within the FOV. A check is made to determine if it is a false detection, whereby the tracking process is returned to state (1). State (5) An object is currently being tracked, it has not been detected in the current frame, and its location is estimated to be outside the FOV. The tracking process is returned to state (1).

T.J. Hebert et al. / Acta Astronautica 54 (2004) 455 461 Table 1 Comparison of debris detected by human reviewers and by automated software (SW) in 52 h of digital video data Detected by reviewers and SW Detected only by reviewers Detected only by SW Detected by neither (correlated target found post-review) 105 7 27 1

461

7. Results and conclusions Results using the automated system were compared to those from two trained observers in reviewing 40 h of video. There were three reviewers for 9-h of video, two reviewers for 40-h of video, and one reviewer for 3-h of video. Table 1 shows the numbers of debris objects detected by the human reviewers and by the automated software. These results demonstrate that the computer-automated system outperforms the combined results from two trained observers, achieving a 17 percent improvement in the detection rate of orbital debris per tape and a three-fold improvement in the detection rate of meteors. References
[1] S.H. Pravdo, D.L. Rabinowitz., et al., The near-earth asteroid tracking (NEAT) program: an automated system for telescope control, wide-eld imaging, and object detection, Astronomical Journal 117 (1999) 1666. [2] W. Flury, A. Massart, et al., Searching for small debris in the geostationary ring: discoveries with the Zeiss 1-metre telescope, ESA Bulletin, Nov (2000) 92100. [3] H. Krag, P. Beltrami-Karlezi., et al., ProofThe extension of ESAs MASTER model to predict debris detections, ACTA Astronautica Journal 47 (2000) 687697. [4] H. Krag, J. Bendisch., et al., Debris model validation and interpretation of debris measurements using ESAs proof tool, ACTA Astronautica Journal 48 (2001) 373383. [5] J.-C. Liou, P. Anz-Meador, et al., The new NASA orbital debris engineering model ORDEM 2000, Orbital Debris Quarterly News 5(4) (2000) 1. [6] T.J. Hebert, E.G. Stansbery, N.E. Hartsough, P. Jones, A Complete Approach to Processing Optical Observations by the Liquid Mirror Telescope to Infer the Space Debris Environment, Report: NASA-JSC-29443 April 2001, pp. 154. [7] Sony HD Digital VCR Conference; Specications of Consumer-Use Digital VCRs using 6.3 mm Magnetic Tape, 1994 1996. [8] Titterington, General structure of regularization procedures in image reconstruction, Astronomy and Astrophysics 144 (1985) 381387. [9] T.J. Hebert, R.M. Leahy, Statistic-based MAP image restoration from Poisson data using Gibbs priors, IEEE Transactions on Signal Processing 40 (9) (1992) 22902303.

For the algorithm described above, one dualprocessor 500 MHz Pentium III PC requires less than 12 h to decode the video frames, detect the debris/meteors, and archive the results found in 1 h of compressed video. 6. The Event-Review software The Event-Reviewer software is an IDL-based SW application. With this application, the user is able to review each debris/meteor event, add comments, and perform photometric analysis. This software contains two user interfaces. Viewing of debris/meteors events is done through the rst interface. The interface contains a graphics windows for displaying video. Controls are provided for image brightness and playback speed. A click on a perform photometric analysis button initiates the second interface. Here, the cataloged star eld for the telescope view is displayed. This star eld is extracted from the Hubble Space Telescope Guide Star Catalog of stars brighter than 16th magnitude. Buttons to pan through the star eld are provided. Based upon star correspondences, a six-parameter forward mapping from star right ascension (RA), declination (DEC) to image frame (x; y) coordinates is computed. This mapping includes a lens distortion model. The magnitude of the debris/meteor is computed and a log-linear mapping between integrated intensity and magnitude is computed using a least squares criterion based upon the four cataloged stars nearest in integrated intensity to the debris/meteor. The visual magnitude of the debris/meteor is computed from this log-linear mapping.