A Computer Vision System for Automatic Knowledge-based Conguration of the Image Processing and Hierarchical Object Recognition

A. S. Vu Department of Computer Science Forschungsgruppe Bildverstehen (FGBV) Technische Universit t M nchen a u Orleansstrae 34, D-81667 Munich, Germany E-Mail: vu@informatik.tu-muenchen.de

Abstract
In this paper a computer vision system is described which congurates automatically programs for image processing and supports the recognition of objects. The system runs in two phase. In the rst phase based on the primitives (curved edges, corners etc.) and the explicit specication of the content of the image given by the user a sequence of operators will be generated and all their free parameters will be computed adaptivelly. In this phase the system uses a rule base composed of knowledge of visual processing operators, their parameters, and their interdependence. In the second phase a hierachical object model is formulated and edited by the user based on the primitives selected in the rst phase. The system editor is specially provided for this purpose. Using the hierachical object model facilitates a rapid interpretation of the result obtained from the previous image processing for the subsequent object recognation.

1. Introduction
As is well known, the knowledge-based automation of image processing has been considered as a very useful facility in various areas, e.g. manufacturing, medicine, or robotics. Recently several systems were developed for this goal. Matsuyama classied them ([5]) into 2 groups: expert systems for image processing and image understanding systems. The major purpose of the rst group is the automation of segmentation, while that of the second one is the interpretation of visual scenes. In several systems both these purposes are performed as well, where the segmentation is the preprocessing of the subsequent interpretation, e.g. the system developed by Ender in [3].

The conguration of a sequence of operators is the central objective of all systems of the rst group. That is, select suitable image operators from an existing bibliothek and then adapt their parameters. Due to the complexity of performing this procedure most of systems restrict themselves to solving a specic problem and therefore, to a certain number of image operators. In [3] Ender tried to solve the adaption of parameters values of a predened sequence of operators, which is particularly applicable for the processing of greyscale images of machinary objects. Liedke and Bl mer developed in [4] a conguration system with o a small number of operators, thereby the set of available sequences of operators is foreseen. Based on the given requirements an appropriate sequence of operators is designed and the adaption of parameters values is follwed. The system introduced by Messer in [6] requires the specication of interested objects, which is inputed by the user via interactive marking on the image. First of all, two masks types, the region- and contour-mask, are created for generating all possible sequences of operators. The image program is the sequence of operators which provides the best result with respect to the marked masks. As a result this system allows merely the recognition of single unstructed objects. Systems of the second group are focused on the automatic image interpretation using object models. The segmentation result has to be presented in an appropriate symbolic data structure, i.e. a structure consisting of primitive symbols. For example Ender used in [3] circle, line and 900 corner as primitive symbols. The performance of the image interpretation in terms of processing speed depends heavily on the structure of object models. In [1] Bunke discussed several presentations of object models such as graphs, semantic nets and frames. In [7] Radig dened a powerful formalism , the relational structure, which allows the description of arbitrary n-objects relation.

2. System overview
Our aim is to develop a system, which performs the conguration of image programs using the expert knowledge base and provides the possibility to recognize structured objects using hierarchical object models. This system supports the user in the image segmentation as well as in the image interpretation. The user in turn should give the required specication and information about the investigated image and interested objects. Apart from that the system will automatically operate by itself and provide the results of both processes, the conguration and the image interpretation. Figure 1 presents the structure of our system ACHI (Automatic Conguration of image processing and Hierarchical image Interpretation). It introduces the essential system components and data, which are either used or generated during the processing.
User Interface

properties of the selected primitive types are however specied by the user, e.g. length of lines, size of corners. A conguration oriented on the selected primitive types can be viewed as successful if the obtained segments are of those types. During the conguration the system applies the knowledge of image operators, their parameter and their relations, which are inherently embedded in the HALCON system. The conguration results are: an image program for the image segmentation, which consists of a sequence of parameterized operators, a segmentation image, which is obtained by applying the above image program, and primitives corresponding the selected primitive types, which are derived from the segments of the segmentation image. An interactive adjusting of the image program carried out by the user could improve the segmentation result . This can be achieved by HALCON. The image interpretation recognizes interested objects in the segmentation image. As it employs exclusively with symbols, it needs: the symbolic primitives, which are obtained in the output image of the segmentation. a object model, which is edited by the user using the selected primitive types. The selected primitive types affect the extracting of primitives as well as the building of the object model. While the rst process can be achieved by ACHI, the latter can be performed step-by-step via the interaction with the user. An object model editor is already implemented for this task. The hierarchical buiding of object model is the crucial point of the image interpretation. It enables the description of structured objects. The selected primitive types are assigned to the lowest layer of the hierarchy. The objects to be recognized are ordered to the highest layer. Between two these layers are middle layers, where intermediate objects are created. Intermediate objects of one layer are created exclusively by those of the next lower layer. This hierarchy requires the ability of the system in abstract building. The image interpretation is performed based on the extracted primitives and creats progresively according to the hierarchical object model all intermediate objects, layer by layer, until the interested objects are formed. The result of the image interpretation is then presented to the user in a objects description le.

Primitive Types

Image Program

Object Model

Image

Configuration

Segmentation image Primitives (Symbols)

Interpretation

Result

Image Informations

Primitives

Image Operators Knowledge Base

Hierarchical Model Building

Data stream Data

Derivation stream Data block

Control System components

Dependency

Figure 1. System Overview

For the conguration the following data will be used: the image to be investigated, the information of the image is either given by the user or estimated by the system itself, e.g. image noise or brightness of the object compared to the image background, and the primitive types should be selected by the user. Each primitive type represents one class of elementary image portions, such as lines, corners or circles. The estimation of some image informations can be achieved by the computer vision system HALCON. HALCON (see [2]) contains a large number of image operators and the knowledge of their applications. Primitive types play an important role in our concept. They are the connecting elements between the image segmentation and the image interpretation. The system provides the user with a set of predened primitive types. The

3. Primitive Types
In our system the primitive types play an exclusively important role. In the conguration they affect the selection of operators and the adaption of their parameters values and serve as evaluation meassures of the congurated image program as well. They are also the basic elements in building the object hierarchy. A primitive type is a symbolic description of similar image portions. It exhibits several typical properties, e.g. a line-type is characterized via 4 properties: two coordinates of the start point, direction and length of the line. These properties will be set with calculated values as soon as the segmentation image is transformed in primitives of the selected primitive types. In order to be able to describe the structured object model, the set of the primitive types should be sufciently voluminous. In ACHI two kinds of primitive types are dened: simple primitive types and

each of these sequences the best parameterized sequence of operators will be selected as the resultant image program. An image program (sequence of operators) in our system represents an image processing procedure of the image segmentation. It consists of 6 steps in following order: 1. Input of images in System. 2. Selection of the conguration region. 3. Pre-processing. 4. Segmentation. 5. Post-processing (improving the segmented image). 6. Extraction of the primitives. Processing methods are classied into 3 techniques: contour-oriented, region-oriented and texture-oriented segmentation.

complex primitive types. The primitives of simple primitive types can be directly extracted from the segmentation image. The complex primitives are subsequently constructed based on the simple ones. In contrast to the simple primitive types, there are extra methods for the complex primitive types to calculate their properties values. Therefore, the primitives of this class are formed in the second step, after the simple ones are already extracted. Let us illustrate this concept via a simplied example : A complex type composed from 2 parallel lines (PL) are constructed based on the simple line-type (L). The rst method of PL check, under what condition a primitive of this type can be formed: 2 derived primitives L1 and L2 can construct a PL if their directions are nearly identical and their regions in this direction are overlapped. The second method calculates the properties values of PL, the distance of two lines, the direction (the same as the direction of two lines) and the length (equal to the length of the overlapped regions of two lines). In this way the number of primitive types can be extended and the designed system therefore is open to further developments.

Hereby operators (in the knowledge base) according to the processing steps (horizontal) and the processing methods (vertical) are arranged in groups (the 6 above stated processing steps can be splitted internally in smaller ones). The sobel operator for instance belongs to group preprocessing and contour-oriented segmentation. The forming of sequences of operators runs through all 6 processing steps. The selection of operators in each step is controlled by a rule base using the data given in Fig. 1 as well as the knowledge of operators and their mutual dependences for the conguration. This reduces the conguration space and thus, the number of sequences of operators under consideration of: the selected primitive types: e.g. in order to nd circlelike primitives only operators of region-oriented segmentation are considered. the image informations: e.g. for a slightly noisecorrupted image (specied by the user in conjunction with the system) the pre-processing step for noise suppression should be skipped. the users requirements: due to the time restriction all time-comsumed operators are likely excluded. the knowledge: several combinations of operators which are frequently applied by the image processing experts for certain processing steps are inherently established. E.g. for edge extraction (pre-processing) a combination of mean-operator and dynamic thresholdoperator is usually applied.

4. Conguration of image programs

The conguration of an image program is a dynamic process. First of all, it constructs all possible sequences of operators. Through the adaption of operator parameters values and the evaluation of obtained results corresponding to

Depending on the users requirements the congurated sequences of operators differ in length. In several processing steps either none, or an operator or a combination of operators can be selected by the system. As soon as the sequence of operators are selected some of operators parameters are already set with values. They are those parameters with the uniquely possible value, e.g. the name of the image. The other parameters are to be adapted. For the parameters adaption the neccessary input data as well as the knowledge of operators are required. In the knowlegde base typical values (default values) of parameters are inherently stored. They are viewed as initial values of parameters if there are no direct values inputed. The stepsize of parameters adjustment and value domains are also recommended here. While selecting the primitive types, the user also species their properties (e.g. lays down the value domains). E.g. only circle-like primitives with medium size (size implies the relation between radius and length or width of the image) are to be extracted. This affects the selection of operators to improve the segmentation, in this case the operator opening circle is selected and its parameter value, the size of the parameter radius of opening circle, have to be adapted. Based on the image informations the domains of parameters values of certain operators are restricted. E.g. the information that the objects to be recognized is relatively dark compared to the background facilitates using the operator threshold with a small threshold value. Or the magnitude of noise in an image inuences the matrix size of selected lowpass lter operators. One the initial parameters values are already laid down the parameterized sequece of operators will be performed and evaluated. A number of image and region properties are dened, e.g. mean, area,compactness of a region, etc. They can be adjusted along with the parameter adaption. The knowledge of the direction of adjustment with respect to the desired change of properties is stored in the knowledge base of HALCON. In case a certain adjustment of a property is to be achieved the system seeks the concerning information in the knowledge base and then carries out the adjustment of corresponding parameters. An example is the operator threshold with two parameters MinGray (p1) and MaxGray(p2), which selects the grayscale region of the image between the gray values p1 and p2. Let F denote the area of this gray-scaled region we can formally describe the direction of adjustment as follows:

the user rejection the system tries to congurate the image program again. An unaccepted image program could improved by the user with HALCON. For this purpose the user can stored the generated image program and then edit with HALCON.

5. Image interpretation
The image interpretation aims to extract the objects to be recognized from an image. Since a good segmentation is an essential precondition for a successful object recognation, the result of the image interpretation depends generally on the obtained primitives. An other critical factor in the image interpretation is the constructing of object models. In ACHI we are in favour of a multi-stage hierarchical model generating. It is identied by that the end object is generated not directly from the primitives, but from the intermediate objects, which are in turns generated layer by layer. For a number of reasons the image interpretation can or should not be extensive at one, but rather performed via a sequence of partial interpretations. An example is the recognition of a rectangular in a noise-corrupted image. After segmetation the broke lines are obtained as the primitives. The searching for full lines as intermediate objects before building the rectangular seems to be a good policy and more sensible than building the rectangular itself directly from the primitives. Hierarchical model building enables not only the description of models and the realisation of image interpretation, but also facilitates the image interpretation of higher complexity. ACHI offers a number of hierarchical abstractions to enable building of objects of a layer from those of the next lower one. Abstraction means the combination of several single detailed objects into abstract ones, as well as the scaling down of number of the set of detailed objects. The following abstractions are dened in ACHI : Aggregation: aggregate combination of different coorperating objects to creat an abstract object. The relation between the detailed objects and the new generated object is the part-of relation. Classication: similar objects are interpreted as elements of one class while maintaining all their individual properties. An object will be assigned to a certain class if its properties coorespond to that of this class. Recursion: recursive combination of a number of similar coorperating object to creat an abstract object. The number of detailed objects is unknown at the beginning. A new object is created composed of the last formed object and a new additional detailed object. This new object thus exhibits the same set of properties as that of its predecessor.

p1 " ^ p1  p2 = F

 and p2 " ^ p1  p2 = F "

At the last processing step of the sequence of operators the primitives will be extracted. The user has the option to reject or accept the extraction result. The domain of the number of primitives can initially specied by the user. By

Selection: selection special objects from a set of objects. According to a certain catergory the selection of objects is achieved. During recursion a lot of intermediate objects are created, which are useless for further processing. Therefore after the recursion a selection is executed, only the largest, also the last created objects in each recursion generating, are selected. The properties of an abstract object can be derived from the properties of the detailed objects, e.g. via line combination, multiplication, division or by taking over unchanged of the detailed objects properties. ACHI provides the user with a model editor to build object models. In order to catch the appearance form of objects to be recognized, whose properties are on a certain tolerant domain, ACHI provides a possibility to dene the conditions for abstractly creating a new object from detailed ones as fuzzy sets. The user then can choose one of a number of options to combine fuzzy conditions. The image interpretation works according principle of the fuzzy (approximated) matching. The input data are the primitives, i.e. the elements of the lowest layer of the object hierarchy. From bottom upwards the intermediate objects are created by this approximated matching, layer by layer. The end object, which is created on the highest layer, presents the result of the image interpretation.

// image input read_image (im1, die3) // Configuration region threshold (im1, im2, 100, 255) shape_trans (im2, im3, rectangle2) reduce_domain (im1, im3, im4) // Segmentation threshold (im4, im5, 0, 50) fill_up_shape (im5, im6, area, 1, 100) // Improvement opening_circle (im6, im7, 15.5) connection (im7, im8) // Primitive extraction select_shape (im8, im9, circularity, and, 0.85, 1.0) smallest_circle (im9, Row, Column, Radius) This in turns created a segmentation image as the result in Fig. 2(c) and a set of primitives: Circle 125.87 Circle 130.27 ... 117.12 220.50 18.84 18.56

6. Example
In this section we describe an example we are working on. It is about tracing the lines of ball-bondings of a microchip in Fig. 2(a).

The hierarchical object model in Fig. 2(d) is described by the user with model editor. First of all, a Recursion is employed to search for 3 circle-primitives on a line, which are then further extended, each time with a new adapted circle-primitive. After that the Selection selects the longest objects. In this case only one line of ball-bondings was found.

Acknowledgements
(a) Primitive Type (b) (c) Object Model (d)

We thank the Gottlieb Daimler- und Karl Benz-Stiftung and the Department of Computer Science, Technische Universit t M nchen, chair of Professor Radig for their support a u in this project.

Figure 2. Example

References
Vie segmentation the primitives of circle type chosen by the user in Fig. 2(b) should be extracted. The circle type exhibits three properties, two coordinates of the centre point and radius, whose value domains are determined initially by the user. The ball-bondings appear dark, what is the image information that should be inputed by the user. The system then generated the following image program: [1] Horst Bunke: Modellgesteuerte Bildanalyse; B.G. Teubner Stuttgart, 1985. [2] Wolfgang Eckstein: Halcon Reference Manual; Department of computer science, Munich University of Technology; October 1997.

[3] Manfred Ender: Ein Beitrag zur automatischen wissensbasierten Konguration von Bildinterpretationssystemen; Fortschritt-Berichte VDI, Reihe 10: Informatik/Kommunikationstechnik Nr. 68, 1987. [4] C-E. Liedtke, A. Bl mer: Architecture of the Knowlo edge Based Conguration System for Image Analysis CONNY; Proceedings of the 11th IAPR International Conference on Pattern Recognition, The Hague, Vol. I, pp. 375-378, IEEE Computer Society Press; September 1992. [5] Takashi Matsuyama: Expert systems for image processing: Knowledge-based Composition of Image Analysis Processes; Computer Vision, Graphics, and Image Processing 48, 1989, pp. 22-49. [6] Tilo Messer: Acquiring Object Models Using Vision Operations; Vision Interface 92, 11.-15. May, 1992, Vancouver. [7] Bernd Radig: Symbolische Beschreibung von Bildfolgen 1: Relationengebilde und Morphismen; Bericht Nr. 90, Uni. Hamburg, 1982.