Humanoid Drawing Robot

A. Srikaew, M. E. Cambron, S. Northrup, R. A. Peters II, D. M. Wilkes, and K. Kawamura Center For Intelligent Systems Vanderbilt University Nashville, TN 37235
srikaewa@vuse.vanderbilt.edu

This paper discusses a drawing robot. ISAC, a Dual Armed Humanoid Robot, explores the relationship between technology and creativity. The problem of tracking and mimicking an artist is analyzed. Control of the end-e ector with force sensors is described. Some initial results are given.
Keyword:

Abstract

Drawing Robot, Humanoid, Color Tracking, Force Sensors

1 Introduction
Entertainment is an interesting application for humanoid robots. ISAC, a Dual Armed Humanoid Robot 1, 2], explores the relationship between man's creativity and automation in the eld of visual art. ISAC has the ability to observe an artist in the process of drawing, and to mimic the motions of the artist. The robot can record the motions and then reproduce it's own \robotic" interpretation of the drawing. Because of the nature of it's soft arms, ISAC is unlikely to reproduce the drawing exactly. Each time it reproduces the artist's actions the robot will add its own subtle variations. Through experimentation with the robot, the artist can learn how to manipulate the robot so that the added variations occur within the artist's plans for his work. Moreover, since the robot reproduces the artist's physical motions in creating the drawing, the robot's motions are a performance in themselves. A humanoid service robot designed for entertainment di ers signi cantly from an industrial robot. Such a robot must physically interact with a human being. Therefore, it must be safe; its design must minimize the possibility of injuring a person. A humanoid robot will not be of much use if programmed to perform a xed set of relatively simple, repetitive tasks. It must take commands from a person { in fact it must work
with a person { to perform a wide variety of complex tasks, each of which could be unique to the moment.

Not only must it be purposive to accomplish speci c tasks, but also it must be exible and reactive to cope

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

with a highly dynamic environment. Such reactive exibility requires the use of complex sensing such as 3D vision, force feedback, and touch. Sensing, at this level, involves much more than the gathering of sensory information. It involves the understanding of sensory information within the context of the problem at hand. In those ways, a human service robot must be able to sense and act much like a human being. There are many examples of robots being used for entertainment. The movie industry uses animatronics to bring to life the fantasies of writers in movies such as Jurassic Park. There are several examples of robots playing musical instruments 3, 4], however, drawing robots are rare. An example of a drawing robot is ARRON 5]. ARRON is the creation of Dr. Cohen at the University of California at San Diego. It is a highly developed drawing machine capable of creating impressive drawings. ARRON consists of 2 parts. First, an arti cial intelligence program creates a drawing in software. This drawing could be printed or displayed on a monitor. The second part of the system is a Cartesian robot that draws the picture. The Cartesian robot is basically a plotter. However, the interaction between the artist and ARRON only occurs at the source code level. Another example of a drawing robot is Drawbot 6] which uses a small industrial robot to draw pre-programmed shapes such as a simple star. In this paper we will discuss many of the parts needed to make ISAC mimic an artist. We examine how we track the artist's hand and translate that data to a drawing plane. We also examine the force control needed for drawing.

2 ISAC

2.1 Softarms
Conventional robot arms, actuated by electric motors, are usually massive and rigid, and therefore dangerous for use in close proximity to people. Impedance control can provide the manipulator with some compliance, but a software error can result in a dangerous machine. An alternative is to use manipulators with exible links or joints. The McKibben arti cial muscle is an excellent actuator for the arms of a human service robot. The arti cial muscle is pneumatic. When depressurized it is a exible cylinder, but an increase in air pressure causes the actuator to expand radially, contract axially, and become sti er. A pair of these actuators attached to either side of a robot arm (in the plane of motion) work in opposition to move the joint, as do the muscles in a human arm. A robot arm that uses McKibben arti cial muscles can be made to be

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

Figure 1: Humanoid Robot lightweight, strong, and naturally compliant. These attributes make the arms safe for human contact. In fact, they are often called \softarms" for those reasons. Softarms 7] have some disadvantages. They exhibit signi cant hysteresis and other non-linearities. They are neither as fast nor as precise as arms actuated by electric motors. However, human-service work typically requires neither great speed nor great accuracy. Thus, the advantages signi cantly outweigh the disadvantages.

2.2 Vision System

Vision is necessary for a human service robot. An e ective visual input system for a humanoid robot is an active stereo color camera head. This has two color video cameras mounted on independent pan motors to permit stereo vergence. Additional motors give the head pan and tilt motion. The pan-tilt-verge motion of the system mimics that of a human being. Many vision problems are solved by this system 8]. In particular, the tracking of an object in stereo facilitates segmentation and distance estimation while greatly reducing clutter, hence the complexity, of the images. The use of color can also greatly simplify object recognition and tracking.

3 Mimic Hand System: Visual Tracking

The drawing robot is designed to perform human-like drawing if the human hand/arm movement can be tracked. In order to mimic human-hand movement, some knowledge about the hand is needed. The objective of the visual tracking module is to provide information (such as position) about the hand to the robot system. This tracking system is composed of a camera head 9] with two color cameras, a color image acquisition

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

Figure 2: ISAC's 4 DOF camera head. board and two Pentium PCs for color image processing, hand tracking, and camera control. Figure 2 shows two color CCD cameras mounted on a pan/tilt/verge system. The hand tracker utilizes the 2-D position of the hands in the image planes and subsequently controls the camera head to center the hand in the camera views. After the target is xated, the 3-D position can be computed and used by the robot system to nd the position of the hand relative to the robot coordinate system. Details of each module are discussed in the following sections.

3.1 Object Color-Based Tracking

The goal of the color segmentation module is to locate a hand in a color image and guide the camera head to center it in the camera views. To accomplish this, color models were created for the colors of the hand. The color segmentation is then performed to separate the pixels into possible hand-color pixels and non-hand-color pixels by the color model. This color segmentation method was proposed by Barile in 10]. In the segmentation stage, pixels from the input image are tested to determine if they fall inside a pre-de ned RGB color model space. From this process, a binary mask image is created for use by later stages. In the binary mask image, the white pixels represent a color that matches the color model and black pixels represent a color that does not match the color model. In order to perform segmentation of the hand, the color model is created o -line by manually segmenting several images of the hand. Because the color of the hand can be considered as a skin-tone color, the problem then becomes skin-tone color segmentation. Figure 3(a) shows an example of the resulting distribution of skin-tone pixels in RGB -space. The parametric model of the data is illustrated in Figure 3(b).

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

(a)

(b)

Figure 3: (a) Distribution of skin-tone pixels in RGB space (reprinted from 2]) (b) Example parametric slices for skin-tone model (reprinted from 2]). The outline of the segmentation process can be seen in Figure 4. When an image is captured, it is segmented and produces a binary mask image. Pre-processing such as low-pass ltering may be applied before the image is segmented to reduce noise introduced by the CCD camera. Pixels in the resulting image are then examined to see if they are in the color model. The corresponding pixel location in the binary mask image is set to 1 if it falls within the color model, otherwise, it is set to 0. After all pixels have been examined, further image processing techniques such as erosion and dilation are then performed on the output image to remove isolated spurious pixels that belong to the background.

Figure 4: Color segmentation process. From the mask image, the location of the hand must be determined. In our system, this is approximately computed by nding the centroid of the skin-tone pixels under an L1 norm criterion. This criterion uses the median value instead of the mean value of the coordinates from the segmented pixels, i.e., median of all xand y-coordinates. This approach yields the following bene ts: (i) it is less sensitive to noise compared to

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

the rst moment centroid method, and (ii) it is not computationally expensive which allows the system to run in real-time. Because the system do not perform any \blob" detection, restrictions, however, have to be made, for example, it is currently assumed that there is only hand in the image.

Figure 5: Example of skin-tone segmentation with centroid shown.

Figure 6: Camera control scheme.

3.2 Camera Tracking

Given the location of the skin-tone centroid in the mask image, the camera tracker then moves the camera head to guide the centroid towards a dead zone in the camera view. The dead zone is de ned to be a circular area in the center of the image view. The size of the dead zone indicates the accuracy of the camera xation point. The smaller the area of the dead zone, the more accurate. The cameras move according to a direction vector generated by the distance from the skin-tone centroid to the center of the image view. The amount of the cameras' movement is proportional to the magnitude of the direction vector. The y-component of the direction vector controls the tilt motor and the x-component controls the verge motors. Once the target has reached the dead zone, the tracker is acknowledged that the target has been xated and then stops moving

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

Figure 7: Camera geometry. camera head. This behavior-based method allows the system to focus the robot's attention using the active vision. The camera control scheme is shown in Figure 6. Once the left and right cameras have xated the target, the 3-D position of the hand can be determined geometrically. The world coordinates of the hand in the camera frames can be determined as follows (see Figure 7) for L R ; for R L ; B B = tan 1 depth tan L ] A = tan 1 depth tan R ] A x = p cos( P + j A j) x = p cos( P j A j) y = p sin( P + j A j) y = p sin( P j A j) z = depth sin( T ) z = depth sin( T ) depth where B = BaseLine and p = cos A 2 * (x; y; z)Camera translation (x; y; z)Robot )

The 3-D position of the hand can be estimated using the camera xation point. The camera angles being controlled, however, can produce errors which lead to inaccuracy in the 3-D position estimator. These errors are introduced by the verge cameras which were altered in-house to reduce cost by using inexpensive hobby servo motors. In order to compensate these errors, the neural-network-based camera-robot calibration are then used. The o -line training process is done by xating cameras at the robot's end-e ector (using the color tracker described above). Next, the inverse kinematics of the camera angles is calculated from the position of the end e ector to estimate the true angles of the cameras. This new angle set is then trained as the output of a neural network using back-propagation with the actual camera angles read from the motors as the input. This provides a simple, robust, and inexpensive camera-robot calibration system. The calibration can be done automatically by moving the robot arm, and using the camera to track the robot's end-e ector,

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

(a)

(b)

Figure 8: (a) ISAC Humanoid Drawing Robot (b) Gripper with a pen. recording corresponding points and training. The result of the calibration is fairly accurate within the robot workspace. The system simply mimics the user's hand movement and utilizes the knowledge from tracking, i.e., trajectory of the hand, to generate 2-D points used for drawing. This can be done by mapping the 3-D hand trajectory to the 2-D robot drawing plane. The robot arm, which its gripper is holding a pen above the drawing plane, then moves corresponding to this 2-D trajectory to perform drawing. The robot hand is kept moving at the same level above the drawing paper. This allows the robot to draw a simple sketch, for example, a square or a triangle. Figure 8 shows the setup for the drawing robot.

4 Force Control
To make a successful drawing, the marker must stay in contact with the drawing surface while ISAC draws. Force sensors at attached to the gripper are used to sense the force applied by the marker on the drawing surface. Force control must be used to ensure that the marker neither presses too hard nor leaves the surface when a line is being drawn. This section will detail the force control used by ISAC to ensure proper drawing surface contact. The free space trajectory control of robots has been performed successfully with industrial and research robots for many years. The rubbertuator-based robots designed by Bridgestone were intended to perform industrial painting tasks where the robot would follow a trajectory in free space 13]. Force control presents a much more challenging task. The challenge is to achieve both a desired force and a desired trajectory while

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

the end-e ector is contacting a rigid surface. Recently, much research has been dedicated to the problem of achieving a desired force and trajectory simultaneously. There are two main techniques implemented to solve the force/position problem. The rst approach is hybrid force/position control. In this technique, only position or force is controlled at any instance of time. The second approach is impedance control, in which both force and position are controlled simultaneously. The force control used for ISAC utilizes the second approach. Impedance control is used because of the nature of the task. The drawing task requires ISAC to keep the marker in contact with the writing surface while following a 2D trajectory across the drawing surface. The impedance controller implemented in ISAC is derived from the position-based impedance control work of Heinrich, Sepehri, and Thorton-Trump at the University of Manitoba 11].

Figure 9: Block Diagram of the Position-Based Impedance Controller The basic premise of position-based impedance control is to make corrections to the desired position based upon the output of a second order mass, damper, spring impedance model (refer to Figure 9). If the actual force is less than the desired force, the output of the impedance controller, should move the desired position in the direction which increases the actual force. If the force is too high, the impedance controller, should move the desired position in a direction that will decrease the force. The desired position is input to a PID controller that attempts to achieve the new desired position. The PID servo-loop operates at a period of approximately 20 msec. The impedance control loop operates at a period of about 100 msec. The impedance model was programmed to follow the second order mass, damper, spring system 11]: mxd + b(xd xa ) + k(xd xa ) = fd fa _ _ where m, b, and k are the mass, damping, and spring constants and the subscripts a and d denote actual and desired values. Although ISAC was programmed to use a second order model, testing showed that a rst order model worked well. The reduced order was implemented by simply using a mass of zero in the implementation. Figure 9 shows a block diagram of the force and position controller.

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

10

An additional aspect of the force control implementation for ISAC is that a force threshold is to be met before ISAC will invoke the position-based impedance control. This allows ISAC to switch automatically from traditional control techniques while following a free space trajectory to the position-based impedance control when the drawing surface is contacted.
600 x 400

Actual Position 200 **** Computed Position

Object Position

200

400

z 600 0 50 100 Sample Points 150 200 250

Figure 10: Camera tracking results.

5 Results
ISAC's goal was to observe an artist and attempt to reproduce the artist's motion as a drawing. This required three major tasks: i) Tracking the artist's hand ii) Transform motion from 3D image coordinates to 2D drawing coordinates iii) Control the end-e ector's position and force to draw the picture. ISAC, successfully mimicked the artists by reproducing drawings of simple shapes. The tracking results relied upon accurately estimating the 3D position of the artist's hand. Figure 10 shows the results of estimation of 3D points. In this graph the actual location versus the estimated position is shown in the X,Y , and Z directions. The maximum error in each direction are X = 59:3mm, Y = 84:9mm, and Z = 61:7mm. The 3-D tracking was su ciently accurate to generate the 2-D points for drawing a square, or a triangle. However, due to the non-linear behavior of the softarm, the resulting drawing, as we expected, was not a perfect square or triangle. The softarm was controlled by a PID controller which did not attempt to correct for nonlinear actuator response. Future, implementations will include a nonlinear controller to achieve better control. The goal of keeping the marker on the drawing surface was, however, quite successful. The drawing performance showed that the position-based impedance controller worked very well. The majority of the

submitted to IASTED International Conference on Robotics and Manufacturing, July 26-31, 1998.

11

time, the marker stayed in contact with the writing surface. The points at which the marker lost contact with the drawing surface were due primarily to the error in the PID position controller. During ISAC'S drawing, the force data showed only a slight oscillatory behavior. Some oscillation is expected due to both the spring like behavior of the McKibben actuators and the spring/damper behavior of the impedance model. The degree of oscillation was small. It did not cause the marker to lose contact with the drawing surface.

6 Conclusion
We successfully mimicked an artist drawing simple shapes. The artist hand was tracked as she drew a simple drawing. The hand was tracked using skin-color segmentation. ISAC was able to estimate the position of the artist hand and translate these positions in order to reproduce an interpretation of the drawing. Force sensors were used to keep a constant pressure as ISAC drew an image. Entertainment robots are an interesting application. As technology continues to improve, robots will become more exible.

References

1] K. Kawamura, D. Wilkes, T. Pack, M. Bishay, and J. Barile, \Future robots for home and factory," in Proceedings of the First International Symposium on Humanoid Robots, (Tokyo, Japan), pp. 53{62, Oct. 1996. 2] D. Wilkes, W. Alford, R. Pack, T. Rogers, R. Peters II, and K. Kawamura, \Toward socially intelligent service robots," to appear in Applied Intelligence, 1998. 3] L. Lange, \The theremin strikes a chord with engineers," EE Times, pp. 1,156, July 1997. Issue 964. 4] http://www.kajitani.mce.uec.ac.jp/mubot/. 5] P. Monaghan, \An art professor uses arti cial intelligence to create a computer that draws and paints," The Chronicle of Higher Education, pp. 27{28, May 1997. 6] http://robix.com/drawbot.htm. 7] M. Iskarous and K. Kawamura, \Intelligent control using a neuro-fuzzy network," IEEE Transactions on Rehabilitation Engineering, 1995. 8] D. Ballard, \Animate vision," Arti cial Intelligence, vol. 48, pp. 1{27, Feb. 1991. 9] S. Asaad, M. Bishay, D. Wilkes, and K. Kawamura, \A low-cost, dsp-based, intelligent vision system for robotic application," in Proceedings of 1996 IEEE International Conference on Robotics and Automation, (Minneapolis, Minnesota), Apr. 1996. 10] J. Barile, M. Bishay, M. Cambron, R. Watson, R. Peters II, and K. Kawamura, \Color-based initialization for human tracking with a trinocular camera system," in Proceedings of 1997 IASTED International Conference on Robotics and Manufacturing, vol. 1, (Cancun, Mexico), May 1997. 11] B. Heinrich, N. Sepehri, and A. Thornton-Trump, \Position-based inmpedance control of an industrial hydraulic manipulator," IEEE Control Systems, vol. 17, pp. 46{52, Feb. 1997. 12] N. Hogan, \Stabel execution of contact tasks using impedance control," in Proceedings of the IEEE International Conference on Robotics and Automation, pp. 1047{1054, 1987. 13] Bridgestone Corporation, Japan, Rubbertuators and Applications for Robots, 1986. Technical Guide No. 1.