You are on page 1of 6

Situation-Driven Control of a Robotic Wheelchair to Follow a Caregiver

Yuki Kinpara, Elly Takano, Yoshinori Kobayashi and Yoshinori Kuno


Abstract Recently, several robotic/intelligent wheelchairs that have autonomous functions to get to some goals or user-friendly interfaces, have been proposed. Although it is desirable for wheelchair users to go out alone, caregivers often accompany them. Therefore it is important to consider reducing caregivers load and support their activities. Hence, we have proposed a robotic wheelchair that can move with a caregiver side by side by observing his/her behaviors such as body position and orientation. This way of motion enables easy communication between the wheelchair user and the caregiver. However, caregivers cannot always move with the wheelchair side by side. For instance, they may step forward of the wheelchair to open a door, and they may step aside when pedestrians are approaching in narrow corridors. To cope with these situations we propose a new robotic wheelchair that can move with a caregiver collaboratively depending on the situation. A laser range sensor is used for observing the environment around the wheelchair. When any obstacles are found in its travelling direction, the wheelchair gives up moving with the caregiver side by side and follows him/her behind. When the caregiver stops at a door, the wheelchair waits for him/her to open the door and follows him/her while avoiding obstacles. The wheelchair returns to move side by side with the caregiver when it enters wide space. We propose a motion control method based on the potential eld method, which enables the wheelchair to take an appropriate position with respect to the caregiver depending on the situation.

we can track the caregivers body orientation by considering its shape so that the wheelchair can make a turn when the caregiver is going to change his/her moving direction. Based on these observations of the caregiver, our wheelchair can move with the caregiver side by side (Fig.1).

Fig. 1.

Concept of our robotic wheelchair moving side by side

I. I NTRODUCTION In the last decade, a variety of robotic/intelligent wheelchairs have been proposed to meet the need in aging society. Their main research topics are autonomous functions such as moving toward some goals while avoiding obstacles, or user-friendly interfaces [1], [7], [8], [9], [11]. Although it is desirable for wheelchair users to go out alone, caregivers often accompany them. Therefore we have to consider not only autonomous functions and user interfaces but also how to reduce caregivers load and support their activities in a communication aspect [2]. From this point of view, we have proposed a robotic wheelchair moving with a caregiver side by side based on the visual and laser sensing techniques [5]. We have developed an integrated sensor which consists of a laser range sensor and an omni-directional camera and attached it on the rear pole of our wheelchair so that the caregivers upper body can be observed by both sensors. This integrated sensor can track the behaviors of the caregiver such as positions and orientations of both body and head by using the Rao-Blackwellized particle lter [10]. Especially,
Y. Kinpara, E. Takano, Y. Kobayashi and Y. Kuno are with the Department of Information and Computer Sciences, Saitama University, Saitama city, Saitama 338-8570, Japan {kinpara, elly, yosinori,

kuno}@cv.ics.saitama-u.ac.jp

However, our previous system has a problem in an actual use. Caregivers cannot always move with the wheelchair side by side. For instance, they may step forward of the wheelchair to open a door, and they may step aside when pedestrians are approaching in narrow corridors. To cope with these situations we propose a new robotic wheelchair that can move with a caregiver collaboratively depending on the situation. When any obstacles are found in its travelling direction, the wheelchair gives up moving with the caregiver side by side and follows him/her behind. When the caregiver stops at a door, the wheelchair waits for him/her to open the door and follows him/her while avoiding obstacles. The wheelchair returns to move side by side with the caregiver when it enters wide space. For local obstacle avoidance we employ a potential eld where the goal position is adaptively changed depending on the situation. Based on these techniques our wheelchair tries to understand what the caregiver is going to do and takes appropriate response. There is a great deal of research on path planning for autonomous mobile robots. The potential eld [4] and the graph theory [14] are commonly used by various systems. Although we have to consider the limitation of movement by nonholonomic vehicles such as wheelchairs, we can also employ these methods by assuming the shape of nonholonomic vehicles as a circle including the whole body. However, when we control the wheelchair in a narrow space, we have to consider the shape of its body in detail. Alhough there are several approaches to dealing with this shape problem by

using the conguration space [13], [6], [15], their processing speed is not enough for real-time systems. Seki et al. [12] proposed a method for practical obstacle avoidance based on the potential eld for a nonholonomic vehicle with a rectangular body to move toward the goal. The repulse forces are calculated in two points located in both front and rear of the vehicles body. The rear forces are inverted and transferred to the front. Then, they are integrated with the front repulse forces and the attractive force to drive the vehicle toward the goal while avoiding obstacles. By using a laser range sensor to measure the distance between obstacles and the outline of the vehicle, the shape of the vehicles body is naturally incorporated in this obstacle avoidance framework. Although they conrm their method only by simulations, we employ their approach for avoiding obstacles in our robotic wheelchair. Unlike their approach, the goal position of the wheelchair is always around the caregiver nearby. Thus, a major weakness of the potential eld, the local minimum problem, is not serious in comparison to its benet from a low computational cost. II. OVERVIEW OF O UR ROBOTIC W HEELCHAIR Fig.2 shows an overview of our robotic wheelchair. For autonomous function, we use a laser range sensor (UTM30LX by Hokuyo Electric Machinery). The laser range sensor can measure in 270 and 30m in the distance. We attach the sensor on the top of the rear pole for observing circumstances including the caregiver. Our wheelchair can be controlled with a joystick controller. We attach an extension unit to the joystick for controlling the output signal. This unit is connected to a laptop PC with USB and is controlled by our software. This output signal is controlled using two output voltage values: a front-back control voltage and a left-right control voltage. These voltage values are controlled using the extension unit that includes a microcomputer (H8-3048). In addition, we replace the lever of the joystick controller with a touch-sensitive one. When a wheelchair user touches the lever of the joystick controller, the control commands of the user overrides the signals from the computer even when the wheelchair moves autonomously. This function enables handicapped users to use our wheelchair safe and reliably.

III. M OVING WITH A C AREGIVER S IDE BY S IDE A. Tracking caregiver The laser range sensor is placed horizontally on the caregivers shoulder level so that the contour of the caregivers shoulder can be observed as a part of an ellipse. We use an ellipsoid as the model to track the caregivers position. We assume the coordinate system represented with X and Y -axes aligned on the ground plane. Then, the model of the tracking target is represented with the center coordinates of the ellipse [u, v] and its rotation angle . These parameters are estimated in each frame by the particle lter [3] framework. Details of tracking method are described in [5]. We evaluate samples based on the observations of the laser range sensor. Distance data captured by the laser range sensor are mapped onto a 2D image plane (what we call a laser image) and used for the likelihood evaluation of the body. The likelihoods of the samples are evaluated from the contour similarity between the model and the caregivers upper body that is partially observed by the laser range sensor. The contour observation model is shown in Fig.3(a). The likelihood of each sample is evaluated, as shown in Fig.3(b), by the maximum distance between the evaluation points and the nearest distance data by: wt,laser = exp
(i) (i)

d2 max d

(1)

where wt,laser is the likelihood score based on the laser image, dmax is the maximum distance between the evaluation points and the nearest distance data. At each time instance, once the distance image is generated from the laser image, each distance dn is easily obtained. d is the variance derived from dn . These likelihood evaluation procedures are repeated for each sample. The estimation of the state at time t is calculated as the expectation of the weights over samples.

(a) Contour observation model Fig. 3.

(b) Evaluation based on maximum distance

Likelihood evaluation by laser images

B. Side-following mode
Laptop PC in the back pocket Laser range sensor

touch-sensitive joystick controller

Fig. 2.

Overview of our robotic wheelchair

Our robotic wheelchair usually moves with the caregiver side by side. The speed and direction of the wheelchair are controlled by the information of the caregiver (Fig.4). We take the X axis in the left-right direction of the wheelchair and the Y axis in the front-back direction. The initial position (u0 ,v0 ) of the caregiver is assumed to be the standard location of the caregiver (Fig.4(a)). The speed is controlled based on the difference (u,v) of the current caregivers position from the initial position. If the difference is positive, the

speed is increased, whereas if it is negative, the speed is decreased. When the current position is not different from the previous position, the system considers that the caregiver is moving at the same speed and the wheelchair is following him or her at the same speed. The moving direction is controlled based on the caregivers position and body direction. When the caregiver turns his or her body near the wheelchair, the wheelchair also turns in the same direction. If the caregiver turns toward the wheelchair, the wheelchair slows down and waits a bit for the caregiver because the wheelchair will be in the caregivers path if it does not slow down. We call this following style the sidefollowing mode.

(a) Integration of forces

(b) Repulsive potential area (a) Initial position Fig. 4. (b) Difference cue Fig. 5. Potential eld using front and rear repulsive forces Wheelchair moving by difference cue

IV. O BSTACLE AVOIDANCE WITH THE P OTENTIAL F IELD Although the default operation mode of our wheelchair is the side-following mode described in the previous section, it is not always desirable that the wheelchair moves with the caregiver side by side. For example, if any obstacles are found in the path where the wheelchair is going, the wheelchair should avoid them rather than keep the side position. Or, if any pedestrians are approaching the wheelchair and the caregiver in narrow corridors, it might be better for them to move in a line. Therefore, we propose the function that the wheelchair avoids obstacles by changing its following position with respect to the caregiver. In particular, the wheelchair changes its following position to the back of the caregiver because it is the most reliable space where the wheelchair can avoid collisions. We call this following style the back-following mode. A. Potential Field Method We use the potential eld method to realize the backfollowing mode. The basic concept of the potential eld is that the robot is assumed as a particle immersed in a potential eld generated by the goal and the obstacles in the environment. The goal generates an attractive potential while each obstacle generates a repulsive potential. A potential eld can be considered as an energy eld and the gradient of the position in which the robot is immersed represents the force that drives it toward the goal while keeping it away from the obstacles. This concept is briey summarized in the equation: F (q) = Fatt (q) +
i

sive force from the i-th obstacle, and F (q) is the resultant force, at the position q. If the caregivers position is set as the goal in this method, the wheelchair may move toward the caregiver while avoiding obstacles, thus realizing the back-following mode. However, there is a problem when we apply this method to our wheelchair as it is. For example, when a wall is in one side from the wheelchair, the wheelchair cannot move straightforward to the goal even when the goal is near by the wall because the repulsive force from the wall may affect the moving direction of the wheelchair. Therefore, we employ the method proposed by Seki et al. [12]. As shown in Fig.5(a), the repulsive forces are calculated in the two points located in both front and rear of the wheelchair. The rear forces are inverted and transferred to the front, and then integrated with the front repulsive forces and the attractive force. By doing this, the resultant force indicates the moving direction suitable for a nonholonomic vehicle such as a wheelchair to approach the goal. Also this method naturally incorporates the shape of the wheelchair by producing the repulsive forces based on the distance between the obstacles and the outline of the wheelchair. Thus the wheelchair can go through the narrow way while avoiding obstacles. This method is briey summarized in the equation: F (q) = Fatt (q) + kf
if

Ff rep if (q) kr
ir

Frrep ir (q), (3)

Frep i (q),

(2)

where Fatt (q) is the attractive force, Frep i (q) is the repul-

where Ff rep if (q) is the repulsive force from the if -th obstacle located in front of the wheelchair, Frrep ir (q) is the repulsive force from the ir -th obstacle located in back of the wheelchair, kf and kr are the parameters representing the proportions of the effects between the front and rear forces so that kf + kr = 1.

We apply this method to controlling the wheelchair in the back-following mode. The original method needs distance data in all 360 directions. However, our wheelchair has a laser range sensor that can measure only in 270 . To make up for the lost backward 90 range, we adjust weight kr so that the repulsive forces can keep balance when the wheelchair moves along a wall. However, if we consider the repulsive forces within the circle around the wheelchair, the distance between the wall and the wheelchair may affect much the balance between the front and back repulsive forces. Thus we consider the repulsive forces in the ellipsoidal range as shown in Fig.5(b). This restriction of range is reasonable since the wheelchair follows the caregiver from behind and does not need to consider obstacles located a bit far in both sides. In this way, our robotic wheelchair can follow the caregiver while avoiding obstacles. B. Mode switching The wheelchair changes its operation modes between the side-following mode and the back-following mode based on the existence of obstacles around the wheelchair. When the wheelchair moves in the side-following mode, the wheelchair searches its front area (Fig.6(a)). If an obstacle (for example, a pedestrian) exists in the front area, the moving mode is changed to the back-following mode. Then the wheelchair follows the caregiver behind. When the wheelchair moves in the back-following mode, the wheelchair searches the lateral area of the caregiver (Fig.6(b)). So long as the obstacle (pedestrian) exists in the lateral area, the wheelchair moves in the back-following mode. If the obstacle no longer exists, the moving mode is changed to the side-by side-mode again.

200 Y[cm] wheelchair caregiver


mo ve

pedestrian

250

ov

mov

300

350 200

250

300

350

400

450

X[cm] 500

Fig. 7.

Wheelchair avoiding moving obstacles

mode when a pedestrian came close and then returned to the side-following mode after passing the pedestrian. Note that a weakness of the potential eld method, which is known as the local minimum problem, does not affect our system because the goal position in the potential eld is usually near by the wheelchair. In addition, even if the wheelchair falls into the local minimum position in the potential eld, the goal position may soon be changed by the movement of the caregiver. This implies that the wheelchair has a chance to go out of the local minimum position. Thus, our system can take only benets of the potential eld method. V. A DAPTIVE B EHAVIORS D EPENDING ON THE S ITUATION We further modify the wheelchair so that it can collaboratively move with a caregiver in various situations such as going through a door and getting into an elevator. In such situations, the caregiver needs to open the door or to press the button. The wheelchair should wait during the caregiver s these actions and then follow him/her when the door is open or the elevator comes. We introduce the insensitive mode connecting the side-following and back-following modes to enable the wheelchair to move as mentioned above in such situations. A. Mode switching by incorporating the insensitive mode When the caregiver stops for a while, the wheelchair turns into the insensitive mode. In this mode, the wheelchair does not respond to the motion of the caregiver while s/he is in the insensitive zone. In the current implementation, if the caregiver stops for 5 seconds, the insensitive modes take over, and the wheelchair does not respond to the caregiver while the caregiver is within 80cm distance from the wheelchair. The wheelchair returns to the back-following mode when

(a) Front area of wheelchair Fig. 6.

(b) Lateral area of caregiver

The area wheelchair searches

C. Experimental results on obstacle avoidance We performed experiments to conrm the effectiveness of our wheelchair that can avoid obstacles. We examined if the wheelchair was able to avoid obstacles such as pedestrians. As shown in Fig.7, our new version of robotic wheelchair successfully avoided a pedestrian approaching the wheelchair. The pictures in Fig.7 shows a sequence of actions when the wheelchair avoided the pedestrian. The wheelchair turned into the back-following mode from the side-following

s/he goes out of the insensitive zone and no obstacle exists in front. Fig.8 shows the transitions among the three modes. When the caregiver walks in an open space, the wheelchair moves in the side-following mode. When any obstacles are detected in the path where the wheelchair is going, the wheelchair turns into the back-following mode. When the caregiver stops for a while, the wheelchair turns into the insensitive mode. When the caregiver goes out of the insensitive zone, the wheelchair turns into the back-following mode.

Side-following mode

Stop for a while Insensitive mode

Fig. 9.

An example situation of going through doors

Detect open space

Detect obstacle Stop for a while

VI. C ONCLUSION We have proposed a robotic wheelchair that can move with a caregiver collaboratively depending on the situation. In an open space, it moves with the caregiver side by side. This enables easy communication between the wheelchair user and the caregiver. When any obstacles are found in the path where it is going, the wheelchair changes its motion to follow the caregiver behind while avoiding collision by using the potential eld method. We have conrmed the ability of the wheelchair by performing operation experiments in laboratory settings. In addition, we exhibited the wheelchair in several exhibitions including ROBOTECH and the International Home Care and Rehabilitation Exhibition, both held in Tokyo Big Sight in 2010. The wheelchair was successfully demonstrated during the convention periods and a large number of people tried the operation. In future work, we are planning to add another motion mode. In a crowded space, the wheelchair cannot use the back-following mode, not to mention the side-following mode. It may need to avoid people independently and meet the caregiver again. The wheelchair needs to nd the caregiver and track him/her again to realize such a motion mode. Also, we are planning to perform real eld experiments such as in day-care centers and museums to conrm the effectiveness of our robotic wheelchair. This work was partly supported by JST, A-Step (221Z02410) and KAKENHI (22243037). R EFERENCES
[1] E.S. Boy, C.L. Teo, and E. Burdet, Collaborative wheelchair assistant, IROS2002, vol.2, pp.1511-1516, 2002. [2] T. Iwase, R. Zhang, and Y. Kuno, Robotic wheelchair moving with the caregiver, SICE-ICASE International Joint Conference 2006, pp.238243, 2006. [3] M. Isard and A. Blake, Condensation - conditional density propagation for visual tracking, International Journal of Computer Vision, vol.29, no.1, pp.5-28, 1998. [4] O. Khatib, Real-time obstacle avoidance for manipulators and mobile robots, International Journal of Robotics Research, vol.5, no.1, pp.9098, 1986. [5] Y. Kobayashi, Y. Kinpara, T. Shibusawa, and Y. Kuno, Robotic wheelchair based on observations of people using integrated sensors, Proc. IROS2009, pp.2013-2018, 2009.

Back-following mode Going out insensitive zone

Fig. 8.

Mode transition model

B. Going through doors The combination of these three modes enables the wheelchair to go through doors with a caregiver. The caregiver stops in front of the door for a while. Then, the wheelchair turns into the insensitive mode and does not follow the caregiver. Thus, the caregiver alone can move forward and open the door. While s/he keeps the door open, the wheelchair can pass through the door. We conrmed that the wheelchair is able to go through a narrow door with 90cm width since it follows the caregiver in the back-following mode with the obstacle avoidance capability. Fig.9 shows an example of this situation. After going through the door, the wheelchair turns into the sidefollowing mode when it reaches an open space wide enough for moving side by side with the caregiver. We also conrmed that the wheelchair can get in an elevator with a caregiver in the same way as in going through a door. In the current system, the caregiver needs to stop intentionally to change the wheelchair into the insensitive mode. This intentional stop seems to be unnecessary. Regardless of the caregivers motion, the wheelchair cannot move forward any more in front of doors or elevators if they are not open since they are detected as obstacles. And then, when they are open, the wheelchair starts moving and passes through them since the obstacles disappear. Such scenarios seem to work but cannot work reliably in real situations. Actually, the wheelchair must stop much earlier in front of doors than against other obstacles so that it can give enough space for the caregiver to open them. In order to make the caregivers intentional action unnecessary, the wheelchair needs to have the capabilities of recognizing the environments and understanding the situation in more details. This is left for future work.

[6] K. Kondak and G. Hommel, Computation of time optimal movements for autonomous parking of non-holonomic mobile platforms, ICRA2001, vol.3, pp.2698-2703, 2001. [7] Y. Kuno, N. Shimada, and Y. Shirai, Look where youre going: A robotic wheelchair based on the integration of human and environmental observations, Robotics and Automation, vol.10, no.1, pp.26-34, 2003. [8] S.P. Levine, D.A. Bell, L.A. Jaros, R.C. Simpson, Y. Koren, and J. Borenstein, The NavChair assistive wheelchair navigation system, Trans. Rehabilitation Engineering, vol.7, pp.443-451, 1999. [9] J. Min, K. Lee, S. Lim, and D. Kwon, Human-friendly interfaces of wheelchair robotic system for handicapped persons, IROS2002, vol.2, pp.1505-1510, 2002. [10] K. Murphy and S. Russell, Rao-Blackwellised particle ltering for dynamic bayesian networks, Sequential Monte Carlo Methods in Practice, Springer-Verlag, pp.500-512, 2001. [11] Y. Satoh and K. Sakaue, An omnidirectional stereo vision-based smart wheelchair, Journal on Image and Video Processing, vol.2007, 87646, 2007. [12] H. Seki, S. Shibayama, Y. Kamiya, and M. Hikizu, Practical obstacle avoidance using potential eld for a nonholonmic mobile robot with rectangular body, International Conference on Emerging Technologies and Factory Automation, pp.326-332, 2008. [13] M. Strobel, Navigation in partially unknown, narrow, cluttered space, ICRA99, pp.29-34, 1999. [14] I. Ulrich and J. Borenstein, VFH*: Local obstacle avoidance with look-ahead verication, ICRA2000, pp.2505-2511, 2000. [15] J.P. Laumond, P.E. Jacobs, M.Taix, and R.M. Murray, A motion planners for nonholonomic mobile robots, Trans. on Robotics and Automation, vol.10, no.5 pp.577-593, 1994.

You might also like