Tennis Touching Point Detection based on High Speed Camera and Kalman Filter

Weixin Wu Dept. of Electrical and Computer Engineering Clemson University

Abstract
This paper introduces the tracking of tennis ball and touching point detecting using high speed camera and Kalman filter. Due to the motion blur of the target, position measurement is corrupted with noise. The segmentation algorithm also fails when the tennis when the ball is passing the line and especially when bouncing on the line. To address the problem, principal components analysis and Kalman filter are used for the measurement and estimation respectively. The position of tennis ball and line is reconstructed for on-line/outside detection. Experimental results demonstrate the effectiveness of the proposed method.

1. Introduction
The goal of the project is to automatically track the tennis ball and measure if the bouncing point is on the line or outside the line, which is also known as good shot or bad shot. To achieve the goal we need to detect the line and the tennis balls dimension, position, bouncing point and reconstruct the relative position of the line and the ball. Three steps are required. The first step is segmentation, which includes line segmentation and tennis ball segmentation. The second step is tennis ball position measuring, tracking, diameter calculating and bouncing point detection. The third step is tennis ball reconstruction using above parameters and on-line/outside calculating. However, two main problems exist in the process. The first problem is blurring along the motion direction. Movement of the tennis ball is recorded using high speed camera with certain frames per second (fps). The camera we have provides maximum of 1000 fps recording rate, however, due to the exposure time that CMOS needs to be charged, the video recorded with 1000 fps is not bright enough for further image processing. To balance the video brightness and minimum speed requirement to get accurate measurement, we selected 50 fps as the suitable speed. As the shutter speed is not ideally fast, blurring exists in the moving direction[1]. To address the problem the blurring caused to tennis ball position and diameter measuring, principal components analysis (PCA) is used to calculate the center point and diameter. However the measurement of position in the motion direction is still corrupted with measurement noise. The second problem

is segmentation failure when tennis ball crossed the line or bouncing on the line. Due to the grayscale image, the tennis ball and line tend to merge into each other. It brings difficulty for segment and for direct measurement of position, which is shown in Figure 5 and Figure 6. To address the above two main problems, Kalman filter[2] is used to track center point at each moment, especially at the bouncing point. Experimental results demonstrate the effective tracking results based on PCA and Kalman filter. The organization of the rest of the paper is as following: The analysis of the problem , Kalman filter and algorithm are introduced in section2. In section 3 some experimental results are proposed and conclusion is given in section 4.

2. Method
Three steps of the project are introduced in this section. The first step is line segmentation using double threshold and tennis ball segmentation using frame differentiating. The second step is tennis ball center point measurement, diameter calculating based on PCA, and bouncing point tracking based on Kalman filter. The last step is tennis ball reconstruction and on-line/outside measurement.

2.1. Line and tennis ball segmentation, motion blurring analysis, center point detection and diameter calculating based on PCA.
The line is segmented using double threshold in order to suppress noise and acquire sharp edge of line for further processing. Due to the specific lighting condition and recording rate, the high and low threshold value are chosen as 200 and 50 respectively. Tennis ball is segmented based on frame differentiating, since the shutter speed is not ideally fast, motion blur exists and the segmented target is not round but a jaggy blob. We assume the blurring only happens in the motion direction but not the perpendicular motion direction, shown in Figure 1. We use PCA to find the Eigenvector and Eigenvalue. The maximum Eigenvalue and corresponding Eigen vector represent the motion direction; the minimum one represents the perpendicular direction. So the maximum distance of the blob in the perpendicular motion direction is tennis balls diameter. The diameter is measured every time, the maximum value is used as the true value of diameter.

Figure 1. Motion blur and PCA

Figure 2. Position reconstruction

The center point of the blob is taken as the center point of the tennis ball. Obviously this measurement is corrupted with noise, known as measurement noise. However, above method fails when the ball crosses the line and bouncing on the line because in grayscale image, the intensity of tennis ball and line is almost saturated. It means they merge into each other at these moments. The segmentation results are more than 100 pixels away from the true position. To address this problem, Kalman filter is used to predict the center point. 2.2.

Kalman filter and bouncing point reconstruction and online/outside detection

In this project, the tracking is based on only current frame and previous frame, which can be taken as Markov process. The movement of tennis ball can be decomposed as free dropping movement in vertical direction and movement with constant speed in horizontal direction. Even though the direction between gravity and velocity in vertical direction changes instantly after the bouncing and friction may apply to tennis ball, what we really interested is the movement before the bouncing, so the system can still be regarded as linear before that moment. The noise is presumed as Gaussian-distributed, so Kalman filter is a good option for this problem. The System transition functions are expressed as following[3, 4]:

in which T is sampling time 1/50 sec, g is gravity constant 9.8 m2/s. The reconstruction of tennis ball bounce uses center point and diameter acquired previously, which is shown in Figure 2. The center touching point between tennis ball and ground is selected as the lowest point of the ball, and touching area is select as an eclipse.

2.3. Algorithm
The algorithm consists of 6 parts shown as flow chart in Figure 7. 3.

Experimental results

Two sets of experimental results are presented as following. Figure 3 and 4 is for the situation that the ball is bouncing on the line; Figure 5 and 6 is outside the line.

Figure 3. (From left to right) Segmented image before and after the bounce for on-line situation.

Figure 4. (From left to right) Kalman filter output of center point in vertical direction, horizontal direction, reconstructed image and original image for on-line situation.

Figure 5. (From left to right) Segmented image before and after the bounce for outside situation.

Figure 6. (From left to right) Kalman filter output of center point in vertical direction, horizontal direction, reconstructed image and original image for outside situation.

4.

Conclusions

From Figure 3 we can find the segmentation when the ball is crossing the line is almost failure. However, from Figure 4 and Figure 6 we can see the Kalman filter prediction weights more prediction than measurement. And in last two pictures in Figure 4 and Figure 6 we can see the prediction of Kalman filter is quite close to the real situation, which demonstrate the effectiveness of the algorithm. However, this work is limited as the results may not tell how credible it is in probability distribution. To improve this, particle filter can be used in future, so it can be used to test the situation that the tennis ball has a little overlap on the line. Whats more, some other method of deblurring algorithm can be used to directly measure the tennis ball position, which may produce better result. 5.

References

1. Image Processing: Analysis and Machine Vision, M Sonka, V Hlavac, R Boyle Brooks/Cole, Thomson Asia Pte Led. 2. An Introduction to the Kalman Filter, Greg Welch and Gary Bishop, http://www.cs.unc.edu/~welch/media/pdf/kalman_intro.pdf. 3. Kalman Filtering, Dan Simon, http://www.innovatia.com/software/papers/kalman.htm. 4. Kalman Filter from Wikipedia, http://en.wikipedia.org/wiki/Kalman_filter. 5. J. Canny, A computational approach to edge detection, PAMI, 8(6):679-698, 1986.

Start

The last frame?

Line segmentation

Ball segmentation

Evector and Evalue calculation using PCA

Center point calculation and diameter calculation

Kalman filtering

Bounce reconstruction and on-line/outside detection

End Figure 7. Algorithm flow chart