1 Copyright 2005 by ASME

Proceedings of IDETC/CIE 2005

ASME 2005 International Design Engineering Technical Conferences &
Computers and Information in Engineering Conference
September 24-28, 2005, Long Beach, California, USA
DETC2005/CIE-85612
AN ENGINEERING SHAPE BENCHMARK FOR 3D MODELS


Natraj Iyer
Subramaniam Jayanti
Karthik Ramani
Purdue Research and Education Center for Information Systems in Engineering (PRECISE)
585 Purdue Mall
School of Mechanical Engineering
Purdue University, West Lafayette IN


ABSTRACT
Three dimensional shape searching is a problem of current
interest in several different fields, especially in the mechanical
engineering domain. There has been a large body of work in
developing representations for 3D shapes. However, there has
been limited work done in developing domain dependent
benchmark databases for 3D shape searching. In this paper, we
propose a benchmark database for evaluating shape based
search methods relevant to the mechanical engineering domain.
Twelve feature vector based representations are compared
using the benchmark database. The main contributions of this
paper are development of an engineering shape benchmark and
an understanding of the effectiveness of different shape
representations for classes of engineering parts.
1. INTRODUCTION
Shape-based retrieval of 3D data has applications in
various disciplines such as computer vision [1], artifact
searching [2], molecular biology [3], and chemistry [4]. The 3D
shape searching area has so far been dominated by research in
computer vision and computer graphics, where researchers
have extensively studied the shape matching problem.
Extensive reviews of these methods are available in [5-7].
Recently, there has been a lot of interest in shape-based search
methods for the mechanical engineering domain [8-16]. In this
paper, we will focus mainly on a benchmark for searching
overall shape of engineering objects - the 3D Engineering
Shape Benchmark (ESB). Based on a classification of 801
engineering components, we have evaluated 12 different shape
descriptors that are described in Section 2 of this paper.
One of the few publicly available benchmarks for 3D
models includes the Princeton Shape Benchmark (PSB) [15].
However, models in the PSB represent mostly multimedia
objects for computer graphics and vision applications. Our
focus in this paper is on developing a benchmark specifically
tailored towards the mechanical engineering CAD domain. Our
hypothesis in the development of a benchmark database for
engineering is that shapes in the engineering domain have
different characteristics than multimedia shapes such as trees,
humans and animals. Two key differentiating characteristics of
engineering shapes as compared to multimedia shapes are: (a)
engineering shapes have high genus as compared to multimedia
shapes, and (b) distances between multimedia shapes are more
apparent as compared to engineering shapes. Engineering
shapes often have high genus and contain important features of
various types, including holes, tunnels, cavities, ribs, and
helixes. Their numbers as well as relative positioning are an
important factor in the resemblance of two shapes [33], unlike
multimedia where the overall shape is more important. As a
result, representations developed for multimedia objects may
not perform well for engineering objects. For example,
structure-preserving representations in multimedia such as
shock graphs [34] and Reeb graphs [17] are widely used and
work well for multimedia shapes of Genus-0. However, issues
related to topological sensitivity of Reeb graphs leads to
significant number of false positives in engineering databases
[17].
We hypothesize that the performance of various shape
representations will be different than the results presented using
PSB. The National Design Repository developed by Regli et al.
[11] provides a large database of engineering parts in various
CAD formats. However, currently the repository contains no
classification scheme analogous to the PSB database. An
obvious difficulty with creating a benchmark database is that no
benchmark can encompass the whole range of naturally
occurring or man-made shapes. This is especially true in the
engineering domain since the very nature of engineering
design, and the increasing complexity of product design have
forced engineers to design and manufacture increasingly
complex and creative shapes. In addition, it is difficult to
categorize engineering shapes into semantic classes. For
example, it is easy to see that dinosaurs and humans belong to
2 Copyright 2005 by ASME
different categories of shapes, but it is more difficult to
characterize the shapes in Figure 1 into different categories.


Gimble Ring

Lock Nut Flange

Figure 1: Examples of parts that have similar shape
but different functions.



Figure 2: Parts from the Beds category from the
Princeton Shape Benchmark ([18]).

Another limitation with using PSB for the engineering
domain is that PSB classifies models primarily on the basis of
function, and secondarily based on shape. Most objects created
in the multi-media domain can be classified into a category
such as bed, tree or airplane purely based on their
function; however, in the engineering domain the existence of
many varieties of semi-standard and one-of-a-type components
makes it impossible to give names to objects or to classify them
into different functional categories. For example, in Figure 2
beds with different geometric shapes are in the same category.
However, for the engineering domain a primarily function
based classification does not seem logical because parts with
different functions may have similar shape as seen in Figure 1.
For example, in Figure 1, gimble rings are used in lighting
fixtures for fastening; lock nuts are used primarily for locking,
while flanges are used for connecting two components. One
cannot always explicitly state the function of an engineering
component purely based on its shape. As a result, we believe
that a function-based classification for the engineering domain
is a daunting proposition.
Another major difference between a multimedia database
and an engineering database is the motivation behind the search
process. In the multimedia domain, the search is performed
primarily for reusing the models for different scene creation.
On the other hand, in the engineering domain, search may
allow designers to not only reuse the 3D CAD model but also
associated information (such as manufacturing and analyses)
thereby reducing product development time. As a result, the
search system must be capable not only of distinguishing
between overall shapes, but also consider manufacturing and
local shape features in its similarity metric.
The rest of this paper is organized as follows: Section 2
presents a brief overview of shape representations that are
compared using our database; Section 3 describes the
construction and evaluation of our ESB database along with
evaluation methodologies; Section 4 presents a detailed
analysis of results along with discussion, and finally Section 5
presents the conclusions.
2. SHAPE REPRESENTATIONS
A comprehensive review of various shape representations
and search techniques for 3D shapes is available in [5-7]. Based
on these papers existing shape representations can be classified
into two classes: feature vector based and topology based.
Topology based representations extract information from the
topology of a part such as eigenvalues [8, 12], skeletal graphs
[9, 12] and Reeb graphs [16] for comparison. In this paper we
have only evaluated representations based on feature vectors.
Our future work will include benchmarking of topology based
methods for 3D shape search. We describe below the twelve
feature vectors that we have used for benchmarking against the
ESB.
2.1. Moment Invariants (MI)
The three second-order moment invariants for the model
are stored as a feature vector. Moment invariants are by nature
independent of orientation. For every voxel in the model
translation, rotational and scale invariant second order moments
are calculated as described below.
lmn
are calculated as described in Eq.(1) as:
5 3
000
( , , )
2
l m n
lmn
x y z x y z dx dy dz
l m n

= + + =

(1)
where
lmn
u are central moments after translation, given by
Eq. (2) as:
( ) ( ) ( ) ( , , )
, , 1, 2, 3,...
l m n
lmn
x x y y z z x y z dxdydz
l m n
u

=
=

(2)
The integrals above are approximated by summation of the
contribution of every voxel to the moment. Since the
characteristic function is invariant under rotation, the
characteristic function of the matrix M (Eq. (3)) of translation
and scale invariant second order moments is RST (Rotation-
Scale-Translation) invariant.
200 110 101
110 020 011
101 011 002





=



(3)
After evaluating the characteristic function for this matrix,
the three moment invariants that are calculated are described in
Eq. (4).
1 200 020 002
2
2 002 200 002 020 200 002 101
2 2
011 110
2
3 200 020 002 110 011 101 101 020
2 2
011 200 110 002
2
I
I
I





= + +
= + +

= +

(4)
2.2. Principal Moments (PM)
The principal moments for a 3D model are the three
eigenvalues
xx yy zz
u u u

of the moment matrix M as
shown below in Eq. (5).
3 Copyright 2005 by ASME
200 110 101
110 020 011
101 011 002
0 0
0 0
0 0
xx
yy
zz
M
u u u u
u u u u
u u u u


=



(5)
2.3. Spherical Harmonics (SH)
Spherical Harmonics are a decomposition of a spherical
function by finding the Fourier transform on a sphere [16]. The
theory of spherical harmonics says that any spherical function
( , ) f can be decomposed as the sum of its harmonics as seen
in Eq. (6):
0
( , ) ( , )
0 , 0 2
m
lm l
l m l
f a Y


=

(6)
where
lm
a are the Fourier coefficients and ( , )
m
l
Y are the
solutions to the normalized Laplaces equation in spherical
coordinates. The spherical harmonic coefficients can be used to
reconstruct an approximation of the underlying object at
different levels. Similar to moments, a partial yet accurate
description of the part can be obtained by using a limited subset
of Fourier coefficients. Intuitively, we expect this method to
perform especially well for objects with radial symmetry,
because of the spherical decomposition.
2.4. Surface Area and Volume (SAV)
In a general shape-based search system, shape
representations are required to be independent of translation,
rotation and size. However, in the mechanical engineering
domain, the surface area and volume of a component have
serious implications on the manufacturability of an object. For
the same volume, thin-walled components such as manifolds
and tubular parts often have higher surface area for the same
volume compared to prismatic components. Due to their
relevance to design and manufacturing we include these
representations in our tests.
2.5. Surface Area to Volume Ratio (SVR)
In addition to SAV, we tested a separate feature vector-
surface area to volume ratio (SVR). Our hypothesis was that
this feature will distinguish between thin walled and prismatic
components and can be used to prune the database using a
multi-step search approach.
2.6. Geometric Ratios (GR)
We also included the two aspect ratios for a 3D model in
our tests due to the simplicity of computation and its relevance
to classification. Again the assumption here is that the aspect
ratios will serve as good search filters.
2.7. Crinkliness and Compactness (CC)
Crinkliness and Compactness are two feature vectors used
in [17]. Compactness is defined as the non-dimensional ratio of
the volume squared over the cube of the surface area.
Crinkliness is defined as the surface area of the model divided
by the surface area of a sphere having the same volume as the
model.
2.8. 2.5D Harmonics (2.5D)
2.5D Harmonics is a new feature vector proposed in [18]
for representing 2D engineering drawings and 3D models. In
this work, 3D model is first converted into a set of 2D views
through a robust pose estimation algorithm described in [18-
19]. The intuition for transforming the problem to 2D space is
that many engineering shapes are created using orthographic
projections and hence are amenable to orthographic projections.
Each 2D view is represented as a spherical function by
transforming it from 2D space into 3D space. Then a fast
spherical harmonics transformation is employed to get a
rotation invariant descriptor.
Bounding Box Bounding Sphere
Ray Casting
3D representation
of 2D drawing
2D view

Figure 3: Procedure of converting a 2D view into the
2.5D Harmonics representation.

Thus, the shape searching problem is reduced to several
simple steps, such as sampling, normalization, and distance
computation between the descriptors.
arctan
i
i
d
r

| |
=
|
\ .
(7)
For example, in our tests we used a bandwidth of 64 for the
2.5D spherical harmonics method, i.e., the descriptor of a
drawing contains 64 signatures.
2.9. 3D Shape Distribution (3DS)
Shape distributions represent the shape signature as a
probability distribution sampled from a shape function
measuring the geometric properties of a 3D model [20].
Selection of the shape function is the primary step in this
technique.

Figure 4: Shape Functions used in the calculation of
shape distributions.
Figure 3 illustrates typical geometry based shape functions
as explained below:
A3: Measures the angle among three random points on the
surface of a 3D model
D1: Measures the distance between a fixed point and one
random point on the surface
D2: Measures the distance between two random points on the
surface
4 Copyright 2005 by ASME
D3: Measures the square root of the area of the triangle among
three random points on the surface
D4: Measures the cube root of the volume of the tetrahedron
among four random points on the surface
2.10. Orthogonal Main Views (OMV)
Jiantao and Ramani [18-19] recently proposed a new
method to obtain shape signatures of 3D models after
automatically obtaining their three orthogonal main views.
Subsequently, a statistics based approach represents the shape
of the 2D views as a distance distribution between pairs of
randomly sampled points. The 2D shape distributions thus
obtained are used to compare 3D objects.
Sample equidistant points on
outer contour
2D shape distribution

Figure 5: Procedure of converting a 2D view into the
Orthogonal Main Views representation.

The methods proposed in sections 2.8 and 2.10 have many
valuable properties: transform invariance, efficiency, and
robustness. Experiments show that they can not only be applied
to vector drawings, but can also be applied to scanned
drawings. The insensitivity to noise allows for the users causal
input, thus supporting a freehand sketch-based retrieval user
interface.
2.11. Convex Hull Histogram (CHH)
In this new method we compute the 3D convex hull for a
given model using the Quickhull [25] algorithm. Then we build
a histogram of the pairwise distances based on the points
obtained from the convex hull [26].
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
N
o
r
m
a
liz
e
d
D
is
ta
n
c
e
3D Model
3D Convex Hull Convex Hull Histogram

Figure 6: Procedure of converting a 3D model into the
Convex Hull Histogram representation.

The number of histogram bins is set based on the accuracy
needed for similarity searching. This histogram is also
normalized and stored in the database for comparison. Models
are retrieved based on the L1 norm for similarity searching
[27].
2.12. Solid Angle Histogram (SAH)
The Solid-Angle method measures the concavity and the
convexity of geometric surfaces. It is described in more detail
in [24]. Histograms are usually based on a complete
partitioning of the data space into disjoint cells which
correspond to the bins of the histograms. The three dimensional
data space is divided into axis parallel, equisized partitions.
This kind of space partitioning is especially suitable for
voxelized data, as cells and voxels are of the same shape, i.e.
cells can be regarded as coarse voxels. Each of these partitions
is assigned to one or several bins in a histogram based on
different models. We tested a solid angle based similarity
model in this paper. Below, we give more details about the
solid angle based similarity model.

Figure 7: Computing the solid angle histogram for a
3D model.

Let
, c r
K be a set of voxels that describes a 3D voxelized
sphere with central voxel c and radius r. For each surface-voxel
v of an object o the Solid-Angle value is computed as follows.
The voxels of o which are inside
, v r
K are counted and divided
by the size of
, v r
K , i.e. the number of voxels of
, v r
K . The
resulting measure is called the Solid-Angle value ( , ) SA v r and
can be computed as follows:
,
,
( , )
o
v r
v r
K V
SA v r
K

= (8)
The solid angle value of each cell is transferred into three
bins - surface voxels, inside voxels and no voxels. Due to its
use of the discretization in rectangular coordinates we expect
this method to represent prismatic and flat-thin walled shapes
well.
3. A 3D ENGINEERING SHAPE BENCHMARK (ESB)
This section describes the processes of acquiring 3D
models for the benchmark database, classification of 3D models
and evaluation of shape representations described in section 2.
3.1 Model Acquisition
Since we are benchmarking shape-based search systems,
we classified 3D models primarily based on shape. It is
important to note that we have eliminated duplicates from our
database, indicating that no two models in the database are the
same. This removes bias from the database, since most shape
representations satisfy the condition of self-similarity. The 3D
models in the database were acquired from a variety of sources
including the National Design Repository [25], websites on the
internet [27, 28] and industry. One of the major difficulties in
building benchmark databases for engineering arises due to the
proprietary nature of many engineering designs. While all
standard components have CAD models made freely available
to the researchers, most of the semi-standard and non-standard
designs are not available for a public benchmark database. As a
result, most engineering shape search systems have been tested
on freely available CAD models. However, it is difficult to
estimate the scalability of shape search algorithms methods to
5 Copyright 2005 by ASME
real-world, non-standard components used in industry. In order
to overcome this difficulty, we acquired proprietary designs
from a heavy machinery manufacturer. Of the 801 models in
the ESB, 76 models are proprietary and have a high degree of
shape complexity. We have therefore provided the remaining
725 models for public use and encourage other researchers and
institutes to use ESB for testing new methods and algorithms
for shape based search. We will continue to add models to our
ESB to encompass a wider variety of shapes. Our work is the
first engineering benchmark for search.
Each 3D model in the ESB has CAD files in two different
formats (STL and OBJ) and an associated thumbnail image
(JPG). Models from the ESB can be downloaded along with a
classification schema from our website
http://engineering.purdue.edu/precise/esb.html. Most models in
the ESB are of non-trivial complexity in terms of design and
manufacturing.
3.2. Model Classification
We believe that users of a shape based search system are
likely to search a database of previous parts with some intent in
mind. For the purposes of a benchmark database, shapes can be
classified into the most granular level (e.g. Gears, Handles
etc.). In order to keep our benchmark database as general as
possible, we used the classification methodology developed by
Swift and Booker for the purposes of cost estimation and
process planning [28]. The models for the benchmark database
were classified by six individuals unrelated to the research,
with varying degrees of training in Mechanical Engineering.
Similar to the classification methodology of the Princeton
Shape Benchmark, we provided individuals with thumbnail
images of 3D models for classification. In case of doubts, the
respective 3D models were also provided to users.
A total of 1,391 3D models were partitioned into three
super-classes, namely:
Solids of Revolution: Part envelope is largely a solid of
revolution
Prismatic: Part envelope is largely a rectangular or cubic
prism, and
Flat-Thin Wall: Parts with flat-thin walled sections and
shell-like components
Within each super-class, models were classified into
clusters of similar shapes. A model was included in a particular
category only when the six individuals agreed upon it. This
classification process continued iteratively until all the 1,391
models were exhausted. Trivial models, as well as categories
with less than 4 models were excluded from the ESB. The final
classification consisted of 801 models classified into 42
categories of similar parts such as Discs, T-shaped parts
and Bracket-like parts. A list of super-classes along with their
respective sub-classes is seen in Table 1.
3.3. Evaluation Methodology
We used standard evaluation procedures from information
retrieval, namely precision-recall curves and average precision.
We also retrieved models randomly to ensure that every shape
representation performed better than random retrievals (RDM).
Definitions of precision and recall are illustrated in Figure 8.
Precision-recall (PR) curves describe the relationship between
precision and recall for an information retrieval method. We
used standard techniques of constructing PR curves from the
NIST TREC standards [29].

Table 1: Classification of Models in ESB.

Bracketlike parts 18 Bearing Blocks 7 90 degree bends (*) 41
Clips 4 Contoured Surfaces 5 Bearing like parts 20
Contact Switches 8 Handles 18 Bolt like parts 53
Curved Housings 9 L Blocks 7 Container like parts 10
Doors 7 Long Machine Elements 15 Cylindrical Parts 43
Rectangular Housings 14 Machined Blocks 9 Discs 51
Slender Thin Plates 12 Machined Plates 49 Flange like parts 15
Thin Plates 23 Motor Bodies 7 Gearlike parts 36
Total 95 Prismatic Stock 36 Long pins 58
Rocker Arms (*) 10 More than two openings (*) 9
Slender Links 13 Non 90 bends (*) 8
Small Machined Blocks 12 Nuts 19
T shaped parts 15 Oil pans (*) 8
Thick Plates 12 Posts 11
Thick Slotted plates 20 Pulley like parts 12
U shaped parts 25 Round, Change at end 21
Total 260 Shelled Tubes 16
Spoked Wheels 15
Total 446
Flat Thin Wall Components Prismatic Parts Solids of Revolution


A perfect retrieval retrieves all relevant models
consistently at each recall level, producing a horizontal line at
precision = 1.0. However in practice, precision decreases with
increasing recall. The closer a PR curve tends to the horizontal
line at precision = 1.0, the better the information retrieval
method. If the PR curves for two information retrieval methods
cross each other or are very close to each other, it is difficult to
make judgements about the relative effectiveness of each
method. In this paper, we also attempt to quantify the
performance of various representation methods with respect to
a base method (in this case, 3D shape distributions) as an
Average of Differences (AOD). Although this is not a standard
practice in information retrieval, we find that it gives relevant
results in the context of this paper. We calculated the average
of differences between the precision values of 3D shape
distributions and the method under investigation. This average
performance is expressed as a percentage of the performance of
the base method.

Retrieved
Documents
(Y)
Relevant
Documents
(Z)
Relevant
Retrieved
(X
Precision = X/Y Recall = X/Z
Figure 8: Precision and Recall Calculations.

4. RESULTS
We evaluated the precision at various levels of recall for
each of the 12 shape representation methods to generate PR
curves. We found that all shape representation methods
performs better than the random retrieval method as seen in
Figure 9.
It was found that on average, the two methods based on 2D
views (2.5D Harmonics and Orthogonal Main View)
outperform other methods consistently. This is similar to
conclusions drawn from [15], where the Lightfield Descriptors
based on 2D projections work better than other 3D methods. It
is interesting to note that traditional engineering drawings also
use 2D projections to represent 3D models. Spherical
harmonics and two histogram based methods - Solid Angle
6 Copyright 2005 by ASME
Histogram and Convex Hull Histogram also perform better than
3D shape distributions.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Recall
P
r
e
c
i
s
i
o
n
2.5D
OMV
SH
CHH
SAH
3DS
SAV
CC
GR
PM
MI
SVR
RDM

Figure 9: Precision-Recall Calculations for 12 shape
representations.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Recall
P
r
e
c
i
s
i
o
n
2.5D Harmonics
Orthogonal Main View
Spherical Harmonics
Hull Histogram
Solid Angle Histogram
3D Shape Distribution

Figure 10: Precision-Recall curves with confidence
intervals for the top 6 shape representations.

Table 2: AOD for 12 shape representations with 3D
shape distributions as a reference.
Method AOD
2.5D Harmonics 81.15%
Orthogonal Main View 72.50%
Spherical Harmonics 56.04%
Hull Histogram 33.87%
Solid Angle Histogram 27.00%
3D Shape Distributions 0.00%
Surface Area and Volume -13.17%
Geometric Ratios -14.10%
Crinkliness and Compactness -14.38%
Surface Area Volume Ratio -19.14%
Principal Moments -35.24%
Moment Invariants -41.66%
Clearly, histogram based methods outperform feature
vector based methods such as Moment Invariants and Principal
Moments. This is because histogram based methods capture
more of the shape content than feature vectors. The only
exception is spherical harmonics, which approximates a shape
with 64 harmonic coefficients, thereby capturing more shape
content than other feature vectors.
On an average, the base method for AOD, 3D shape
distributions, performs 5.57 times better than random retrieval.
Table 2 shows the AOD of various methods as a percentage
value. From this point forward, we only consider the top six
shape representations based on the AOD for comparison of
different methods. The results for each super-class are
presented below.
4.1. Flat-Thin Walled Components
For the Flat-Thin walled components class, the methods
based on 2D drawings outperformed other methods.
Surprisingly, 3D shape distributions and Surface Area and
Volume performed better than the rest of the three methods
based on more complex feature vectors, viz., SH, CHH, and
SAH. Although simple, the SAV performs better for this super
class because thin walled components have higher surface areas
and lower volume, and these features are more explicitly
captured in the SAV compared to other point-based methods.
The PR curves for these methods for the Flat Thin Walled
components super class is shown in Figure 11.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Recall
P
r
e
c
i
s
i
o
n
2.5D Harmonics
Orthogonal Main View
3D Shape Distribution
Solid Angle Histogram
Surface Area and Volume
Geometric Ratios
Surface Area Volume Ratio

Figure 11: Precision-Recall curves for Flat-Thin
Walled Components.
4.2. Prismatic Parts
For this super class, there were four methods that
performed consistently better than 3D shape distributions,
namely the two 2D view based methods (2.5D and OMV),
Spherical Harmonics and Solid Angle Histograms. However,
the 2D view based methods did not show a marked
improvement in performance compared to other methods for
this super class. While the 2D view based methods performed
consistently well, two other methods viz. spherical harmonics
(SH) and Solid Angle histograms (SAH) performed comparable
at higher recall levels (after 0.3 and 0.5 recall respectively) as
we had expected. Results can be seen in Figure 12.
4.3. Solids of Revolution
Both methods based on 2D drawings (2.5D and OMV)
performed significantly better than other methods on average.
Spherical Harmonics and the Convex Hull Histogram
performed better than 3D shape distributions. The PR curves
for these methods for the Solids of Revolution super class is
shown in Figure 13.
7 Copyright 2005 by ASME
As with a text based search engine such as Google, users
would like the most relevant 3D models to appear early on in
the search results. This is equivalent to saying that we need
higher precision at low levels of recall. We ranked methods
based on their AODs until a recall value at 0.25. The results are
shown in Table 3. Clearly, the shape representations that hold
more shape content are better at retrieving more relevant
models as compared to the retrieval size. Performing a similar
analysis for each of the three super-classes, we found that SAV
gives better precision than 3D shape distributions for flat thin
walled parts. Not surprisingly, for all three super-classes, both
methods based on 2D views outperformed other methods at low
recall levels.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Recall
P
r
e
c
i
s
i
o
n
2.5D Harmonics
Orthogonal Main View
Solid Angle Histogram
Spherical Harmonics
3D Shape Distribution
Hull Histogram

Figure 12: Precision-Recall curves for Prismatic
Parts.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Recall
P
r
e
c
i
s
i
o
n
2.5D Harmonics
Orthogonal Main View
Spherical Harmonics
Solid Angle Histogram
Hull Histogram
3D Shape Distribution
Crinkliness and Compactness

Figure 13: Precision-Recall curves for Solids of
Revolution.

Table 3: AOD for 12 shape representations until 25%
recall with 3D shape distributions as a reference.
Method AOD
2.5D Harmonics 49.04%
Orthogonal Main View 41.96%
Spherical Harmonics 25.19%
Hull Histogram 23.29%
Solid Angle Histogram 13.28%
3D Shape Distribution 0.00%
Surface Area and Volume -16.35%
Geometric Ratios -18.92%
Surface Area Volume Ratio -29.02%
Principal Moments -29.09%
Crinkliness and Compactness -31.12%
Moment Invariants -31.27%
5. DISCUSSION
We believe that for the engineering domain, it is important
to analyze which shape representations perform well for a
particular part category, which may seem contrary to the views
of researchers in computer vision and graphics. This problem is
partly also because engineering shapes have higher levels of
shape complexity as compared to other shapes. Part repositories
in a single engineering division often hold similar kinds of
parts. For example, a division of a car company that designs
connecting rods will hold similar connecting rods, while
another division that designs steering columns will hold a lot of
similar steering columns.
6. CONCLUSIONS
In summary, we have developed a publicly available
engineering shape benchmark (ESB) for comparing various
shape based search algorithms. ESB includes a set of 725
models in two formats (STL and OBJ) along with associated
JPG images and a classification schema. All this data is
available publicly from our website
http://engineering.purdue.edu/precise/esb.html.
The main contributions of this paper are development of an
engineering shape benchmark and an understanding of the
effectiveness of twelve different shape representations across
three classes of engineering parts. We used 3D shape
distributions as a base method to evaluate the performance of
other methods on the ESB. It was found that two new shape
representations based on 2D views produce consistently better
results than methods based purely on 3D models.
We also evaluated the performance of different methods on
three classes of engineering shapes. It was found that for Flat-
Thin Walled components and Solids of Revolution, the
methods based on 2D projections outperformed other methods
consistently. However, for Prismatic parts, the best four
methods - 2.5D, OMV, SH and SAH perform equally well. The
better performance of 2D view based algorithms for all classes
of shapes reinforces our intuition that engineering shapes
exhibit distinguishing shape features in their 2D views. In
addition, spherical harmonics seems to perform reasonably well
for the prismatic parts and solids of revolutions compared to
other shape representations.
To the best of our knowledge, this is the first attempt at
creating a benchmark database for engineering shapes. We will
continue to add models to the ESB to encompass a wider
variety of shapes so that we can conduct detailed studies
regarding the performance of shape representations across the
whole spectrum of engineering shapes. We hope that our efforts
at building the ESB and conducting studies using ESB for
classes of engineering shapes will give rise to more robust
shape representations.
ACKNOWLEDGMENTS
The initial funding for this project came from the 21st
Century Research and Technology Fund award from the state
of Indiana. We acknowledge Professor Karthik Ramanis
8 Copyright 2005 by ASME
University Faculty Scholar award from Purdue University,
which seeded this project. Supplemental support from the e-
Enterprise Center at Discovery Park, Purdue University is
acknowledged. The discussions and suggestions by Jiantao Pu,
and the implementation of various methods by Alok Bhide and
Josh Olsen deserve special mention.
REFERENCES
1. Pope, A. (1994), Model-Based Object Recognition: A
Survey of Recent Research, Technical Report 94-04,
Department of Computer Science, University of British
Columbia, Canada.
2. Rowe, J., Razdan, A., Collins, D., and Panchanathan, S.,
(2001), A 3D Digital Library System: Capture, Analysis,
Query, and Display, Proceedings of 4th International
Conference on Digital Libraries (ICADL), Bangalore,
India.
3. Kastenmller, G., Kriegel, H.-P., Seidl T, (1998),
Similarity Search in 3D Protein Databases, Proceedings
of German Conference on Bioinformatics, Kln, 1998.
4. Bruno, I.J., Kemp, N.M., Artymiuk, P.J., and Willett, P.,
(1997), Representation and Searching of Carbohydrate
Structures Using Graph-theoretic Techniques,
Carbohydrate Research, Vol. 304, pp. 61-67.
5. Tangelder, J., and Veltkamp, R.C., (2004), A Survey of
Content Based 3D Shape Retrieval Methods, Proceedings
of Shape Modeling International, pp. 145-156
6. Iyer, N., Jayanti, S., Lou, K., Kalyanaraman, Y., and
Ramani, K., (2005), Three-dimensional shape searching:
state-of-the-art review and future trends, Computer Aided
Design, Vol. 37, No. 5, pp. 509-530
7. Cardone, A., Gupta, S.K., and Karnik, M., (2003), A
Survey of Shape Similarity Assessment Algorithms for
Product Design and Manufacturing Applications, ASME
Journal of Computing and Information Science in
Engineering, Vol. 3, No. 2, pp. 109-118
8. McWherter, D., and Regli, W.C., (2001), An Approach to
Indexing Databases of Solid Models, Technical Report
DU-MCS-01-02, Drexel University, Philadelphia, PA.
9. Iyer, N., Kalyanaraman, Y., Lou, K., Jayanti, S., and
Ramani, K., (2003), A Reconfigurable 3D Engineering
Shape Search System Part I: Shape Representation,
Proceedings of ASME DETC 03 Computers and
Information in Engineering (CIE) Conference, Chicago,
IL.
10. Lou, K., Jayanti, S., Iyer, N., Kalyanaraman, Y., Ramani,
K., and Prabhakar, S., (2003), A Reconfigurable 3D
Engineering Shape Search System Part II: Database
Indexing, Retrieval and Clustering, Proceedings of ASME
DETC 03 Computers and Information in Engineering
(CIE) Conference, Chicago, IL.
11. Foster, C., Hayes, E., Ip, C.Y., McWherter, D., Peabody,
M., Shapirsteyn, Y., Zaychik, V., and Regli, W., (2001),
National Design Repository project: A status report, in
International Joint Conferences on Artificial Intelligence
(IJCAI) and AAAI/SIGMAN Workshop on AI in
Manufacturing Systems.
12. McWherter, D., Peabody, M., Regli, W. C., and
Shoukofandeh, A., (2001), An approach to Indexing
Databases of Graphs, Technical Report DU-MCS-01-01,
Department of Mathematical and Computer Science,
Drexel University, Phildelphia, PA, June 2001.
13. Rea, H., Corney, J., Clark, D., Pritchard, J., Breaks, M.,
and MacLeod, R., (2001), Part Sourcing in a Global
Market, Proceedings of ICeCE 01, China Machine Press,
Beijing.
14. Cybenko, G., Bhasin, A., and Cohen, K., (1997), Pattern
Recognition of 3D CAD Objects, Smart Engineering
Systems Design, Vol. 1, pp.1-13.
15. Kriegel H.-P., Krger P., Mashael Z., Pfeifle M., Ptke M.,
Seidl S., (2003), Effective Similarity Search on Voxelized
CAD Objects, Proceedings of 8th International
Conference on Database Systems for Advanced
Applications, Kyoto, Japan, pp. 27-36.
16. Kriegel H.-P., Brecheisen S., Krger P., Pfeifle M.,
Schubert M., (2003), Using Sets of Feature Vectors for
Similarity Search on Voxelized CAD Objects,
Proceedings of ACM SIGMOD Int. Conf. on Management
of Data (SIGMOD'03), San Diego, CA, pp. 587-598.
17. Bespalov, D., Regli, W.C., and Shokoufandeh, A., (2003),
Reeb Graph based shape retrieval for CAD, Proceedings
of ASME Design Engineering Technical Conferences,
Computers and Information in Engineering Conference,
Chicago, Illinois.
18. Ip, C-Y., Lapadat, D., Sieger, L., and Regli, W.C., (2002),
Using Shape Distributions to Compare Solid Models, 7th
ACM/SIGGRAPH Symposium on Solid Modeling and
Applications, Saarbrcken, Germany, pp. 273-80.
19. Shilane, P., Min, P., Kazhdan, M., and Funkhouser, T.,
(2004), The Princeton Shape Benchmark, Proceedings of
Shape Modeling International, Genova, Italy, pp.167-178.
20. Hilaga, M., Shinagawa, Y., Kohmura, T., and Kunii, T. L.,
(2001), Topology Matching for Fully Automatic
Similarity Estimation of 3D Shapes in SIGGRAPH 2001,
ACM Press, pp. 203-212.
21. Kazhdan, M., Funkhouser, T., and Rusinkiewicz, S.,
(2003), Rotation Invariant Spherical Harmonic
Representation of 3D Shape Descriptors, in Proceedings
of ACM/Eurographics Symposium on Geometry
Processing, pp. 167-175
22. Jiantao Pu, and Karthik Ramani, (2005), A 3D Model
Retrieval Method Using 2D Freehand Sketches, accepted
for Fourth International Workshop on Computer Graphics
and Geometric Modeling (CGGM'2005), Emory
University, Atlanta, USA, May 22-25, 2005
23. Jiantao Pu, Kuiyang Lou, and Karthik Ramani, A 2D
Sketch User Interface for 3D CAD Model Retrieval,
(2005), accepted for CAD05, June 20-24, Bangkok,
Thailand, 2005, accepted.
24. Osada, R., Funkhouser, T., Chazelle, B., and Dobkin, D.,
(2002), Shape Distributions, ACM Transactions on
Graphics, Vol. 21, No. 4, pp.807-832.
25. Barber, B.C., Dobkin, D., and Huhdanpaa, H. (1996), The
Quickhull Algorithm for Convex Hulls, ACM
Transactions on Mathematical Software, Vol. 22, No. 4,
pp. 469-483.
26. Boutin, M. and Kemper, G., (2004), On Graph
Matching, PRECISE Technical Report, PRE-TR-2004-1,
Purdue University, West Lafayette IN
27. Kalyanaraman Y. and Boutin, M., (2005), Convex Hull
Histograms for Shape Matching, PRECISE Technical
9 Copyright 2005 by ASME
Report, PRE-TR-2005-1, Purdue University, West
Lafayette IN
28. http://www.designrepository.org
29. http://www.part-solutions.com/
30. http://www.traceparts.com
31. Swift K.G and Booker, J.D, (1998), Process Selection
From Design to Manufacture, John Wiley and Sons, NY
32. http://trec.nist.gov/pubs/trec13/appendices/CE.MEASURE
S.pdf, Proceedings of the Thirteenth Text Retrieval
Conference, TREC 2004.
33. Sotzio, A. and Shamir, A., (2004), Feature Sensitive 3D
Shape Matching, Proceedings Computer Graphics
International, pp. 596599, 2004.
34. Demirci, M., Shokoufandeh, A., Dickinson, S., Keselman,
Y., and Bretzner, L., (2004), Many-to-Many Feature
Matching Using Spherical Coding of Directed Graphs,
Proceedings of European Conference on Computer Vision,
Prague, May, 2004