Minimally Invasive Therapy.

2010;19:211

REVIEW ARTICLE

How to objectively classify residents based on their psychomotor

laparoscopic skills?

MAGDALENA K. CHMARRA1, CORNELIS A. GRIMBERGEN1,2,

FRANK-WILLEM JANSEN1,3, JENNY DANKELMAN1
1
Department of BioMechanical Engineering, Delft University of Technology, Delft, the Netherlands, 2Department of
Biomedical Engineering & Physics, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands,
and 3Department of Gynecology, Leiden University Medical Center, Leiden, the Netherlands

Abstract
In minimally invasive surgery (MIS), a surgeon needs to acquire a certain level of basic psychomotor MIS skills to perform
surgery safely. Evaluation of those skills is a major impediment. Although various assessment methods have been introduced,
none of them came as a superior. Three aspects of assessing psychomotor MIS skills are discussed here: (i) advantages and
disadvantages of currently available assessment methods, (ii) methods to objectively classify residents according to their level of
psychomotor skills, and (iii) factors that inuence psychomotor MIS skills. Motion analysis has a potential to be the means to
deal with assessment of psychomotor skills. Together with classication methods (e.g. linear discriminant analysis), motion
analysis provides an aid in deciding whether a resident is ready to move to the next level of training. Presence of factors that
inuence psychomotor MIS skills results in a high need for standardisation of valid tasks and setups used for the assessment of
MIS skills.

Key words: Laparoscopy, psychomotor skills, training, assessment, motion analysis

Introduction objective assessment of technical performance is in its

It is indisputable that a surgeon needs to acquire a
certain level of manual dexterity to perform surgery
safely. Therefore, surgical organisations (e.g.,
Accreditation Council of Graduate Medical
Education ACGME) are calling for methods to
ensure the maintenance of skills, advance surgical
training, and to credential surgeons as technically
competent (1-3). A variety of training simulators is
commercially available and used to train basic psy-
chomotor skills needed in minimally invasive surgery
(MIS, e.g., laparoscopy) (4-10). Since the introduc-
tion and implementation of those simulators in the
curriculum of surgeons training, there is growing
consensus about the need to assess surgical compe-
tence objectively. Evaluation of MIS skills, however, is
still a major impediment. Although progress has been
made over the past years, at this stage, the science of
infancy (11).
Various scoring methods to assess laparoscopic skills
have been developed (12-14). However, widely used
methods to objectively determine whether a resident
can be called an expert, intermediate, or novice
(according to his/her laparoscopic skills) are still miss-
ing. It is, therefore, necessary to develop such objective
assessment methods and to implement them in the
training curriculum. Ideally, objective assessment
methods address at least two topics: They identify
potential areas of improvement, and they determine
whether a resident is competent to move to the next level
of training (e.g. to the operating room (OR)).
Surgery, and MIS in particular, requires a broad
range of skills (e.g. technical skills, motor skills, cog-
nitive knowledge, surgical judgement, fast acting,
team work, communication skills) (15, 16). Hence,
various objective assessment methods are needed to

Correspondence: M. K. Chmarra, Department of BioMechanical Engineering. Faculty of Mechanical, Maritime and Materials Engineering (3mE),
Delft University of Technology, Mekelweg 2, 2628 CD Delft, the Netherlands, Fax: 31-15-27 84 717; E-mail: m.k.chmarra@tudelft.nl

ISSN 1364-5706 print/ISSN 1365-2931 online 2010 Informa UK Ltd

DOI: 10.3109/13645700903492977

assess those skills. This paper will elaborate only on
topics related to the assessment of basic psychomotor
MIS skills. Therefore, advantages and shortcomings
of current assessment methods will be given. Further,
methods to classify residents as experts, inter-
mediates, and novices, according to their basic
psychomotor MIS skills, will be discussed. Examples
of factors that inuence psychomotor MIS skills will
be given. Finally, a number of recommendations and
possible approaches to improve current assessment of
residents basic psychomotor skills will be proposed.
Assessment of basic psychomotor MIS skills
Presently, one of the most essential and widely
accepted objective measure of residents competence
is the number of performed cases (17). This measure
is easily quantiable (Table I). However, it does not
represent the real competence of a resident. For
example, it has not yet been investigated how many
performance cases are required for competency.
Moreover, it is very likely that different individuals
require different numbers of cases for gaining com-
petency (18).
Besides the number of performed procedures, the
most frequently used objective evaluation measure of
residents MIS skills in the OR is the number of
complications (19-23). This measure is also easily
quantiable (Table I). However, such assessment of
MIS skills based on morbidity and mortality data fails
to represent the actual residents competence, since
each patient and case are always different and cannot
be easily compared.
Evaluation methods based on task-specic check-
lists and global ratings scales (e.g., Global Operative
Assessment of Laparoscopic Skills GOALS, and
Objective Structured Assessment of Technical
Table I. Overview of assessment methods
Assessment of
Training psychomotor
Objectivity environment MIS skills
No. Cases + OR - instrument in 3D
No. Complications + OR -
GOALS - OR, box, VR +/-
OSATS - OR, box, VR +/-
Educator - OR, box +/-
Motion analysis + VR, box + a task
MIS minimally invasive surgery; OR operation room; box box
trainer; VR virtual reality; GOALS Global Operative Assess-
ment of Laparoscopic Skills; OSATS Objective Structured
Assessment of Technical Skills.
Objective classication 3
Skills OSATS (13, 24-28)) have gained a lot of
attention (29, 30). These methods have been vali-
dated in training setups (e.g. skills labs) and showed
that they can be of great value (24-28). However, their
validation and implementation in the OR is difcult,
since it is not clear when the various scores should be
given. It is, e.g., difcult to judge the skills of the rst-
year resident and the fth-year resident, who both
obtained the highest OSATS score, because during
assessment, the evaluator often takes into account
their year of the residency. As a consequence, both
residents have the same score, but their MIS skills are
different.
In many hospitals and training centres, assessment
of MIS skills is often done by surgical educators, who
can be inuenced by, e.g., personal relationships.
Since this kind of assessment is per denition not
objective (30, 31), the emphasis is put on using
trainers (e.g., box trainers and virtual reality (VR)
trainers) for the assessment of individual skills (e.g.,
knowledge of steps of a procedure, basic psychomotor
MIS skills) (32-34).
Various studies showed that basic psychomotor
MIS skills can successfully be assessed by analyzing
motions of MIS instruments (12, 30). Figure 1 shows
an example of typical MIS instrument trajectories of
an expert and a novice resident performing the same
task. A range of measures have been proposed. All of
them are parameters based on time-dependent three-
dimensional (3D) representation of the tip motions of
the MIS instrument together with the rotation of the
instrument around its axis:
. time total time taken to complete the task
. path length the length of the curve described by
the tip of the instrument
. movement economy ideal path length divided by
the actual path length, where the ideal path length is
the straight-line distance between two targets
. deviation from the ideal path the sum of the
differences between the actual and the ideal path
. depth perception total distance travelled by the
instrument along its axis
. accuracy the correctness of placing the tip of the
. rotational orientation the amount of rotation of
the instrument around its axis
. angular area parameter related to the distances
between the farthest left-right and forward-back-
ward positions of the instrument while performing
. volume parameter related to the distances
between the farthest left-right, forward-backward,
and in-out positions of the instrument while per-
forming a task
4 M. K. Chmarra et al.

Expert Novice

-140
-140

zt [mm]
-160
zt [mm]

-160
-180
-180
-200
-200
40 0
40 20
20 0 0 -50
0 yt [mm] -20
-50 -40 -100 xt [mm]
yt [mm] -20
-40 xt [mm]
-100

Figure 1. An example of typical MIS instrument trajectories of an expert and a novice resident performing a positioning task with the left hand.
The camera was held by the resident in his/her right hand.

. motion smoothness parameter which represents a Consequently, there is no agreement whether a

change of the acceleration, etc. (12, 35-41).
Classication based on basic psychomotor MIS
skills
There is a tendency to assess residents performance
in MIS using a few very basic aspects. Mostly, those
are: Time, path length, and penalty points. Such
assessment, however, does not show the actual size
and the nature of the gap between expert and novice
performance (12). From the clinical point of view, it
is important to objectively determine (classify)
whether a resident can be called an expert, intermediate,
or novice (according to his/her MIS skills). In daily
practice, it has been recognised that one performance
measure alone (e.g. time) does not sufciently mea-
sure basic MIS skills. Therefore, basic MIS skills are
often assessed using at least two different outcome
measures (42-44). It has also been found that perfor-
mance measures depend on the kind of performed
exercise (e.g., suturing, cutting, clip applying) (45).
Therefore, different motion characteristics might be
optimal for different exercises.
Using performance measures as an examination (or
classication) requires establishing a passing score
(18). Determining a sufcient score is difcult
because of (at least) two reasons:
. Current assessment is done using multiple assess-
ment measures, and
. there is no gold standard of surgical competency
against which one can judge the validity of com-
petency assessment (18).
resident should be assessed on individual tasks, or
whether a composite assessment of all tasks should be
used (46).
There is a shortage of methods that can be used to
distinguish between residents of varying experience,
especially when ne gradations in experience need to
be detected. The American Board of Surgery and the
Royal Australasian College of Surgeons have both
mandated Fundamentals of Laparoscopic Surgery
(FLS) certication of candidates before they receive
their specialty certication. The FLS is a program
which was developed by the members of the Society of
American Gastrointestinal and Endoscopic Surgeons
(SAGES) and the American College of Surgeons
(ACS) in 2004 (47, 48). The technical skills of resi-
dents are being evaluated in the FLS based on the
McGill Inanimate System for Training and Evalua-
tion of Laparoscopic Skills (MISTELS) score
(47, 49). The score is computed for each individual
task and takes only two performance measures (time
and accuracy) into account (50). The MISTELS
score, however, ignores a lot of information about
instrument movement (e.g. the volume of move-
ment), which is an important issue in MIS. Therefore,
the MISTELS score, although reliable and validated
(51-53), can be seen as a score that only partially
assesses actual psychomotor MIS skills.
Cotin et al. have introduced a standardised score
based on ve motion analysis performance measures
(12). The score is supposed to be used to classify
residents according to their psychomotor MIS skills.
The score is calculated using user-dened weighting
factors. One of the shortcomings of that score is the
 Objective classication 5

fact that the weight factors have not been reported. various tasks (e.g. suturing, placement of beads at
They are, thus, not standardised. It is, therefore, likely predened positions, and cutting predened pat-
that correct implementation of that score in the daily terns), and, since it is a fully objective method,
practice is difcult and can provide some confusion . there is no subjective bias introduced by the user or
when comparing scores of residents following differ- assessor.
ent MIS trainings. Moreover, the classication is
A pilot study showed the potential of this classi-
performed based on the training data of experts
cation method to distinguish between residents
only and does not account for the distribution of
with ne gradation in experience (e.g. experts and
performance measures of intermediates and/or
intermediates) (40). The classication method has
novices (12). Additionally, validation of the score
not been fully validated, and as such, it should not
included comparison of only expert and novice per-
(yet) be used to certify psychomotor MIS skills.
formances. In consequence, it is not known whether
Nevertheless, the potential of that classication
the standardised score is able to detect ne gradations
method lies in its simplicity, generalisability, and
in basic MIS skills.
objectiveness.
A fully objective classication method to differen-
tiate between surgically competent and surgically not
(yet) competent residents based on psychomotor MIS
skills has been introduced by Chmarra et al. (40). Factors that inuence psychomotor MIS skills
This method makes use of principal component
In the literature, the potential of using motion analysis
analysis (PCA) a data reduction technique, and
as an objective tool for assessing psychomotor MIS
linear discriminant analysis (LDA) a classication
skills in training environments has been demonstrated
method. Both PCA and LDA are commonly used in
(35-39). However, extensive research should take
the eld of pattern recognition and can easily be
place in order to allow full and correct exploitation
implemented for classication of residents according
of motion analysis as an assessment tool.
to their psychomotor MIS skills (54, 55). Among
MIS requires psychomotor skills different from
others, the most important advantages of using
open surgery. At Delft University of Technology,
such a classication method are:
three aspects (factors) that differentiate MIS from
. Possibility of using any motion analysis perfor- conventional open surgery have been investigated:
mance measure and/or combination of such Force feedback (56), use of the camera to obtain
measures, the view of the operation eld (45), and specic
. possibility of applying the method to a single spe- movements of MIS instrument obtained due to the
cic task (e.g., suturing) and to a combination of limited number of degrees of freedom (57). The

A. B.

4th DOF
3rd DOF

1st DOF

2nd DOF

Figure 2. The TrEndo tracking system developed at Delft University of Technology (58). The TrEndo is used for guiding and measuring
movements of real MIS instruments in training setups (box trainers and VR trainers). (A) A schematic drawing of the TrEndo. The TrEndo
allows measurement and manipulation of the instrument in four degrees of freedom (DOFs): translation (1st DOF) and rotation (2nd DOF) of
the instrument around its axis, and left-right (3rd DOF) and forward-backward (4th DOF) rotation of the instrument around the incision point.
(B) A box trainer equipped with the TrEndo tracking system.
6 M. K. Chmarra et al.
inuence of those three factors on MIS instrument
movements has been studied. During those studies,
the TrEndo tracking system (Delft University of
Technology, Figure 2) has been used to record move-
ments of real MIS instrument in the box trainer (58).
It has been found that force feedback (factor 1)
inuences psychomotor MIS skills when executed
tasks require applying pulling and pushing forces
(56). For such tasks, a change from a trainer without
force feedback to the one that provides natural force
feedback has a detrimental effect on performance (e.g.
time, path length, depth perception). For that reason,
it has been suggested that training for tasks in which
forces play an important role (e.g. stretching tissue)
should be done using trainers with natural force
feedback. In case of tasks that do not require appli-
cation of above mentioned forces (e.g. eye-hand
coordination tasks), training can be done in trainers
without force feedback.
In MIS, a small camera (factor 2) is used to provide
the surgeon with the view of the operation eld. The
view of the operation eld (e.g. side, angle, scale) can
change when there is a change in the position of the
camera. This kind of change can disturb the eye-hand
coordination of the surgeon (even an experienced
one), especially when combined with replacement
of the camera operator (as in Figure 3). Therefore,
surgeons should be able to perform MIS well in two
situations:
.
When manipulating the camera themselves, and
.
when the assistant manipulates the camera.
A study showed that camera holding (by the resi-
dent him/herself or by an assistant) inuences psy-
chomotor MIS skills (45). Particularly, this has been
observable for novice residents. From the clinical
point of view, it was an interesting nding, because
it showed that residents who are hardly experienced in
MIS should begin their active participation in MIS in
the OR by performing basic techniques while holding
the camera themselves.
Studies on specic movements of MIS instruments
obtained due to the limited number of degrees of
freedom (factor 3) showed that a goal-oriented
M A
P P
Figure 3. A schematic drawing representing three possible test congurations. Left: participant performing a task with the right hand,
and holding the camera in the left hand. Middle: participant performing a task with the right hand, the camera is held by an assistant.
Right: participant performing a task with the left hand, the camera is held by an assistant.
movement in MIS is not performed via the shortest
path, and can be split into two phases: The retracting
phase and the seeking phase (Figure 4) (57). Only in
the seeking phase, experts and intermediates are more
effective than the novices. Therefore, the seeking
phase is characteristic of performance differences
(57). Moreover, analysis of motions in separate phases
of the movement gives great insight into the nature of
the difference in performance of MIS tasks between
expert and, e.g., novice performance. Therefore,
motion analysis can be used to give feedback on
the nature of possible residents limitations, which
have to be improved.
Discussion
Medical education extends over the lifetime of the
surgeon. Therefore, objective assessment methods are
needed to support regular certication and monitor-
ing of the residents progress, re-certication after a
loss of privileges, and maintenance of certication
(59). Although even a simple task in MIS requires
the use of different skills at the same time (e.g.
knowledge of anatomy, eye-hand coordination, pro-
tocol knowledge) (15, 16), only objective methods to
assess and to classify residents based on their psycho-
motor MIS skills have been presented in this paper.
The idea behind that choice lies in the fact that
technical prociency seems to be fundamental to
perform surgery safely (18). This is particularly visible
in MIS, which is executed in a limited working area,
with limited tactile perception, and with difcult
handling of MIS instruments (60).
Although various methods to assess psychomotor
MIS skills have been developed, none of them has
come out as clearly superior. Each of the methods has
some advantages over the other and vice versa. The
same can be said about methods that are used to
classify residents according to their psychomotor MIS
skills.
Motion analysis has a great potential to be used as a
means to objectively assess psychomotor MIS skills.
For example, the study described by Chmarra et al.
M A M
P
 Objective classication 7

P
M

A
B
plane AB

plane ABP

Expert Novice
60 60
y [mm]

y [mm]
40 M M
40
20 20 plane ABP
0 A B 0 A B
0 20 40 60 80 100 120 0 20 40 60 80 100 120
x [mm] x [mm]
0 A B 0 A B
z [mm]

z [mm]

-20 -20 plane AB

-40 -40 M
M
-60 -60
0 20 40 60 80 100 120 0 20 40 60 80 100 120
x [mm] x [mm]

Figure 4. An example of goal-oriented movements of an MIS instrument due to limited number of degrees of freedom in MIS. Top: Plane
ABP (described by points A, B, and P) and plane AB (which passes through points A and B, and is perpendicular to plane ABP), with goal-
oriented movement (AMB) presented in three-dimensional space. Bottom: The typical trajectories for an expert (left) and a novice (right)
performing a goal-oriented movement. The movements are projected in the ABP and the AB planes. A the initial position of the tip of the
instrument; B the end position of the tip of the instrument; M a point, which is used to make a distinction between the retracting and seeking
phase; P incision (pivoting) point.

(57) shows that the thorough analysis of instrument
movements in MIS can be used to indicate the size
and the nature of the gap between experts and novice
performance (12). In order to be able to fully and
correctly exploit motion analysis as an assessment
tool, more research still needs to be done. Moreover,
to apply motion analysis as an assessment tool, it is
necessary to equip trainers with a system that tracks
and records these motions. Ideally, such a tracking
system should be applied at all stages of training (in
the box trainer and VR trainer (during training), and
in the OR (during an operation)) to provide the
resident with automatic detection of correctness of
performed task (a feature currently available only in
VR trainers), and to allow to monitor his/her progress.
It could also be used to give feedback on the nature of
possible limitations of a resident.
The OR is the most realistic (training) environ-
ment, in which residents use real MIS instruments,
and where realistic force feedback is obtained due to
natural instrument-tissue interactions. Similarly, box
trainers also provide residents with a realistic envi-
ronment and force feedback (because real MIS
instruments are used). In case of VR trainers, how-
ever, the environment is less realistic (virtual) and the
MIS instruments are often simulated. Since not many
studies have been done on actual forces applied and
felt during MIS (61), force feedback in VR trainers (if
present) can be very different from that provided by
real instruments. Nevertheless, the advantage of VR
trainers over other training methods is that due to the
use of simulated MIS instruments, the design of the
tracking systems for a VR trainer is simplied. More-
over, in all VR trainers, tracking systems are inher-
ently present (62). The systems that are able to track
real MIS instruments, however, are still in their
infancy, since, e.g., factors such as patient safety,
size of the system, and ergonomics in the OR play
a critical role in designing those systems. Ideally,
tracking systems should be designed so that it is
possible to use them in all training environments.
Currently, however, there are only a few systems
that can track movements of real MIS instruments
in box trainers or in the OR (64). Most of those
systems are still at a prototype or a testing stage.
The systems available on the market, on the other
8 M. K. Chmarra et al.
hand, are very expensive. As a consequence, systems
that track real instruments are mostly used in aca-
demic environments for research purposes (e.g., to
develop new assessment methods) (12, 38, 39,
57, 63, 64).
Assessing residents psychomotor MIS skills in
almost realistic performance situations often results
in complex and not straightforward test design (e.g.,
simulation of anatomy or bleeding in VR trainers).
Besides that, independent of the reality level in a
trainer, it is still a simulation. For that reason, resi-
dents do not behave exactly in the same way as they
would in a real-life surgery. Consequently, assessment
of psychomotor MIS skills can easily be biased.
In the literature, identication (and quantication)
of factors that inuence psychomotor MIS skills have
not yet gained much attention. Nevertheless, when
developing and/or introducing assessment and/or
classication methods, it is necessary to take into
account that those factors can inuence (bias) the
outcomes of those methods. There are a few impor-
tant lessons learnt from studies on identifying such
factors. Lesson 1: There are factors that inuence
motions of MIS instruments. Ignoring those factors
when assessing psychomotor MIS skills will bias that
assessment. Lesson 2: The judgement on psychomo-
tor MIS skills of residents who perform tasks that
require applying pulling and pushing forces (e.g.
suturing) in a trainer without correct force feedback
needs to be done carefully, since there is a high
possibility that motions of MIS instrument(s) are
different than those during executing exactly the
same task in a trainer with natural force feedback.
Lesson 3: The setting (e.g. task, position of the
camera, camera operator, MIS instrument) used
when assessing psychomotor MIS skills inuences
those skills. Moreover, the setting plays an important
role when determining parameters to discriminate
between different levels of performance. Lesson 4:
The concept of shortest path length, which is currently
used to indicate better MIS performance, may not be
a proper concept for analysing optimal movements.
Since no studies have been done to nd out which
motion analysis performance measures truly can
quantify psychomotor MIS skills, it is not sure
whether all of the currently used measures are worthy
to be used.
As said above, there are factors that inuence MIS
instruments movements when performing MIS task
(e.g. presence/absence of force feedback, setup used
for assessment, fatigue and sleep deprivation (65)).
For that reason, it is necessary to take those factors
into account when developing assessment and/or
classication methods. Therefore, it is recommended
to evaluate residents MIS skills only using
standardised and valid tasks and setups, which are
described and dened in detail. Currently, in the
literature on MIS, it is hardly possible to nd a
publication about an assessment method (or valida-
tion study) that reports each detail of the setup (e.g.
position of the monitor, the operator of the camera,
position of the task in a box trainer) and the protocol
used in the study. Without such information, it is
difcult to reproduce the tests or to implement
described assessment methods.
Lack of knowledge on all the factors that have an
inuence on psychomotor MIS skills shows that there
is a need to identify those factors. For example, it is
still not known what kinds of forces are felt and
should be applied when performing MIS. Moreover,
it has not yet been recognised whether the same forces
are (and should be) applied during training of psy-
chomotor MIS skills. Does the camera holding (by an
assistant or by the participant him/herself) inuence
forces applied during performing MIS tasks? When
should application of forces be introduced into the
training curriculum? How exactly are psychomotor
MIS skills inuenced by the presence or absence of
force feedback? Can we measure it? Which motion
analysis performance measures can be used to deter-
mine that? These and many more other questions
need to be answered in future research. Only then will
one be able to develop and implement standard,
correct and objective assessment and classication
methods.
Classication methods are supposed to provide an
aid in deciding whether a resident is competent to
move to the next level of training (e.g. to perform an
operation in the OR) as well as to certify and monitor
progress of residents. Objective assessment methods,
on the other hand, can be used to identify possible
residents limitations, which have to be improved.
There is, therefore, a place for both classication
and assessment methods in MIS. Motion analysis
has a potential to become the means to objectively
assess psychomotor MIS skills. It can be expected that
simple, general, and objective classication methods
(e.g. such as the method based on PCA and LDA) will
be used in the future to determine whether a
residents skills level can be called expert,
intermediate, or novice.
As mentioned in the Introduction, ideally, objective
assessment methods address at least two topics: They
identify potential areas of improvement, and they
determine whether a resident is competent to move
to the next level of training (e.g., to the OR). Iden-
tication of potential areas of improvement of skills
can be done by thorough analysis of instrument
motion such as, e.g., in case of study on specic
movements of MIS instrument obtained due to the

limited number of degrees of freedom (factor 3) (58).
Thus far, however, there are only a few studies that
investigated movements in MIS in detail.
To determine whether a resident is competent to
move to the next level of training, a passing score
needs to be established. As said in the Classication
based on basic psychomotor MIS skills section,
determining a sufcient score is difcult because
current assessment is done using multiple assessment
measures, and because there is no gold standard of
surgical competency against which one can judge the
validity of competency assessment. Therefore, before
the passing score can be determined, it has to be
decided which assessment measures are going to be
used, and the gold standard of surgical competency
needs to be established.
Independent of the kind of method used to assess
MIS skills, performance in one context (e.g. psycho-
motor MIS skills) does not predict performance in
another context (e.g. knowledge of anatomy) very
well. Therefore, motion analysis alone is not a
panacea, but a method that is not capable of dem-
onstrating the actual surgical competence of the indi-
vidual. Therefore, it is required to make a selection of
assessment methods that depends on the skills to be
evaluated, and to use a combination of those
methods.
Any fully objective assessment of MIS skills is more
reliable than a subjective one. When using objective
motion analysis methods to assess psychomotor MIS
skills it is more likely that residents will accept feed-
back on their skills and that they will constructively
incorporate that knowledge in training.
Conclusions
Currently used objective methods to assess and clas-
sify residents according to their basic psychomotor
MIS skills are based on motion analysis. These meth-
ods are supposed to address at least two topics:
Identifying potential areas of improvement of skills,
and determining whether a resident is competent to
move to the next level of training (e.g., to the OR).
Motion analysis has a potential to be the means to deal
with those two topics. However, current lack of gen-
eral knowledge on basic psychomotor skills in MIS
does not (yet) enable to fully and correctly exploit
motion analysis as an assessment tool, e.g. for certi-
cation purposes (e.g. assessment of psychomotor
skills of residents before they receive their specialty
certication). Nevertheless, motion analysis together
with simple, general, and objective classication
methods (e.g. such as linear discriminant analysis)
can already be used to objectively classify residents as
Objective classication 9
experts, intermediates, and novices based on their
psychomotor skills.
Delcaration of interest: The authors have no con-
icts of interest or nancial ties to disclose.
References
1. Park AE, Witzke D. The surgical competence conundrum.
Surg Endosc. 2002;16:5557.
2. Roberts KE, Bell RL, Duffy AJ. Evolution of surgical skills
training. World J Gastroenterol. 2006;12:321924.
3. Ritchie WP. Basic certication in surgery by the American
Board of Surgery (ABS). What does it mean? Does it have
value? Is it relevant? A personal opinion. Ann Surg.
2004;239:1339.
4. Halvorsen FH, Elle OJ, Fosse E. Simulators in surgery. Minim
Invasiv Ther. & Allied Technol. 2005;14:21423.
5. Aggarwal R, Moorthy K, Darzi A. Laparoscopic skills training
and assessment. Br J Surg. 2004;91:154958.
6. Gallagher AG, Lederman AB, McGlade K, Satava RM,
Smith CD. Discriminative validity of the minimally invasive
surgical trainer in virtual reality (MISTVR) using criteria
levels based on expert performance. Surg Endosc.
2004;18:6605.
7. McClusky DAM, van Sickle K, Gallagher AG. Relationship
between motion analysis, time, accuracy, and errors during
performance of a laparoscopic suturing task on an augmented
reality simulator. 12th International Congress of the European
Association for Endoscopic Surgery, Barcelona, O207,
June 2004.
8. Munz Y, Kumar BD, Moorthy K, Bann S, Darzi A. Laparo-
scopic virtual reality and box trainers: is one superior to the
other? Surg Endosc. 2004;18:48594.
9. Schijven M, Jakimowicz J. Virtual reality surgical laparoscopic
simulators. How to choose. Surg Endosc. 2003;17:194350.
10. Sutherland LM, Middleton PF, Anthony A, Hamdorf J,
Cregan P, Scott D, et al. Surgical simulation. A systematic
review. Ann Surg. 2006;243:291300.
11. Darzi A, Datta V, MacKay S. The challenge of objective
assessment of surgical skill. Am J Surg. 2001;181:4846.
12. Cotin S, Stylopoulos N, Ottensmeyer M, Neumann P,
Rattner D, Dawson S. Metrics for Laparoscopic Skills
Trainers: The Weakest Link! Lect Notes Comput Sc
LNCS 2002;2488:3543.
13. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J,
Hutchison C, et al. Objective structured assessment of tech-
nical skills (OSATS) for surgical residents. Brit J Surg.
1997;84:2738.
14. Moorthy K, Munz Y, Dossis A, Bello F, Darzi A. Motion analysis
in the training and assessment of minimally invasive surgery.
Minim Invasiv Ther & Allied Technol. 2003;12:13742.
15. Dankelman J, Chmarra MK, Verdaasdonk EGG,
Stassen LPS, Grimbergen CA. Fundamental aspects of learn-
ing minimally invasive surgical skills. Minim Invasiv Ther &
Allied Technol. 2005; 14:24756.
16. Cuschieri A, Francis SN, Crosby J, Hanna GB. What do
master surgeons think of surgical competence and revalida-
tion? Am J Surg. 2001;182:1106.
17. Park AE, Witzke D. The surgical competence conundrum.
Surg Endosc. 2002;16:5557.
18. Feldman LS, Sherman V, Fried GM. Using simulators to
assess laparoscopic competence: ready for widespread use?
Surgery 2004;135:2842.
10 M. K. Chmarra et al.
19. Moore MJ, Bennett CL. The learning curve for laparoscopic
cholecystectomy. Am J Surg. 1995;170:5509.
20. Morgenstern L, McGrath MF, Carroll BJ, Paz-Partlow M,
Berci G. Continuing hazards of the learning curve in laparo-
scopic cholecystectomy. Am Surg. 1995;61:9148.
21. Lekawa M, Shapiro SJ, Gordon LA, Rothbart J, Hiatt JR. The
laparoscopic learning curve. Surg Laparosc Endosc.
1995;5:4558.
22. Mehrabi A, Yetimoglu CL, Nickkholgh A, Kash A, Kienle P,
Konstantinides L, et al. Development and evaluation of a
training module for the clinical introduction of the da Vinci
robotic system in visceral and vascular surgery. Surg Endosc.
2006;20:137682.
23. Passerotti CC, Nguyen HT, Retik AB, Peters CA. Patterns
and Predictors of Laparoscopic Complications in Pediatric
Urology: The Role of Ongoing Surgical Volume and Access
Techniques. J Urology 2008;180:6815.
24. Vassiliou MC, Feldman LS, Andrew CG, Bergman S,
Leffondre K, Stanbridge D, et al. A global assessment tool
for evaluation of intraoperative laparoscopic skills. Am J Surg.
2005;190:10713.
25. Gumbs AA, Hogle NJ, Fowler DL. Evaluation of resident
laparoscopic performance using global operative assessment of
laparoscopic skills. J Am Coll Surg. 2007;204:30813.
26. Chang L, Hogle NJ, Moore BB, Graham MJ, Sinanan MN,
Bailey R, et al. Reliable assessment of laparoscopic perfor-
mance in the operating room using videotape analysis. Surg
Innov. 2007;14:1226.
27. Aggarwal R, Grantcharov T, Moorthy K, Milland T, Darzi A.
Toward feasible, valid, and reliable video-based assessments
of technical surgical skills in the operating room. Ann Surg.
2008;247:3729.
28. Pellen M, Horgan L, Barton JR, Attwood S. Laparoscopic
surgical skills assessment: can simulators replace experts?
World J Surg. 2010;33:4407.
29. McKinley RK, Strand J, Ward L, Gray T, Alun-Jones T,
Miller H. Checklists for assessment and certication of clinical
procedural skills omit essential competencies: a systematic
review. Med Educ. 2008;42:33849.
30. Moorthy K, Munz Y, Sarker SK, Darzi A. Objective assess-
ment of technical skills in surgery. BMJ 2003;327:10327.
31. Sidhu RS, Grober ED, Musselman LJ, Reznick RK. Assessing
competency in surgery: where to begin? Surgery
2004;135:620.
32. Goff BA, Lentz GM, Lee D, Houmard B, Mandel LS. Devel-
opment of an objective structured assessment of technical
skills for obstetric and gynecology residents. Obstet Gynecol.
2000;96:14650.
33. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W.
Testing technical skill via an innovative bench station
examination. Am J Surg. 1997;173:22630.
34. Dosis A, Aggarwal R, Bello F, Moorthy K, Munz Y, Gillies D,
et al. Synchronized video and motion analysis for the assess-
ment of procedures in the operating theater. Arch Surg.
2005;140:2939.
35. van Sickle KR, McClusky DA, Gallagher AG, Smith CD.
Construct validation of the ProMIS simulator using a novel
laparoscopic suturing task. Surg Endosc. 2005;19:122731.
36. Acosta E, Temkin B. Haptic laparoscopic skills trainer with
practical user evaluation metrics. Stud Health Techn Inform,
MMVR 2005;111:8-11.
37. Cavallo F, Megali G, Sinigaglia S, Tonet O, Dario P. A
biomechanical analysis of surgeons gesture in a laparoscopic
virtual scenario. Stud Health Technol Inform, MMVR
2006;14:7984.
38. Strom P, Kjelling A, Hedman L, Wredmark T, Fellander-
Tsai L. Training in tasks with different visual-spatial compo-
nents does not improve arthroscopy performance. Surg
Endosc. 2004;18:11520.
39. Smith CD, Farrell TM, McNatt SS, Metreveli RE. Assessing
laparoscopic manipulative skills. Am J Surg. 2001;181:54750.
40. Chmarra MK, Klein S, de Winter JCF, Jansen FW,
Dankelman J. Objective classication of residents based on
their psychomotor laparoscopic skills. Surg Endoscop, 2009.
Epub ahead of print.
41. Cesanek P, Uchal M, Uranues S, Patruno J, Gogal C,
Kimmel S, Bergamaschi R. Do hybrid simulator-generated
metrics correlate with content-valid outcome measures? Surg
Endosc. 2008;22:217883.
42. Aggarwal R, Grantcharov T, Moorthy K, Milland T,
Papasavas P, Dosis A, et al. An evaluation of the feasibility,
validity, and reliability of laparoscopic skills assessment in the
operating room. Ann Surg. 2007;245:9929.
43. Moorthy K, Munz Y, Dosis A, Bello F, Chang A, Darzi A.
Bimodal assessment of laparoscopic suturing skills. Construct
and concurrent validity. Surg Endosc. 2004;18:160812.
44. Stefanidis D, Haluck R, Pham T, Dunne JB, Reinke T,
Markley S, et al. Construct and face validity and task workload
for laparoscopic camera navigation: virtual reality versus
videotrainer systems at the SAGES Learning Center. Surg
Endosc. 2007;21:115864.
45. Chmarra MK, Kolkman W, Jansen FW, Grimbergen CA,
Dankelman J. The inuence of experience and camera holding
on laparoscopic instrument movements measured with the
TrEndo tracking system. Surg Endosc. 2007;21:206975.
46. Darzi A, Datta V, Mackay S. The challenge of objective
assessment of surgical skills. Am J Surg. 2001;181:4846.
47. Peters JH, Fried GM, Swanstrom LL, Soper NJ, Sillin LF,
Schirmer B, et al. Development and validation of a compre-
hensive program of education and assessment of the basic
fundamentals of laparoscopic surgery. Surgery 2004;135:
2127.
48. Fundamentals of Laparoscopic Surgery (FLS) Program. Soci-
ety of American Gastrointestinal and Endoscopic Surgeons.
http://www.sprogram.org
49. Ritter EM, Scott DJ. Design of a Prociency-Based Skills
Training Curriculum for the Fundamentals of Laparoscopic
Surgery. Surg Innov. 2007;14:10712.
50. Vassiliou MC, Ghitulescu GA, Feldman LS, Stanbridge D,
Leffondre K, Sigman HH, et al. The MISTELS program to
measure technical skill in laparoscopic surgery : evidence for
reliability. Surg Endosc. 2006;20:7447.
51. Scott DJ, Ritter EM, Tesfay ST, Pimentel EA, Nagji A,
Fried GM. Certication pass rate of 100% for fundamentals
of laparoscopic surgery skills after prociency-based training.
Surg Endosc. 2008;22:188793.
52. Fried GM. FLS assessment of competency using simulated
laparoscopic tasks. J Gastrointest Surg. 2008;12:2102.
53. McCluney AL, Vassiliou MC, Kaneva PA, Cao J,
Stanbridge DD, Feldman LS, et al. FLS simulator perfor-
mance predicts intraoperative laparoscopic skill. Surg Endosc.
2007;21:19915.
54. Jain AK, Duin RPW, Mao J. Statistical pattern recognition: a
review. IEEE Trans Pattern Anal Machine Intell. 2000;
22:438.
55. Duda RO, Hart PE, Stork DG. Pattern classication. Second
ed. Willey, New York, 2001.
56. Chmarra MK, Dankelman J, van den Dobbelsteen JJ,
Jansen FW. Force feedback and basic laparoscopic skills.
Surg Endosc. 2008;22:21408.
 Objective classication 11

57. Chmarra MK, Jansen FW, Grimbergen CA, Dankelman J.
Retracting and Seeking Movements During Laparoscopic
Goal-Oriented Movements. Is the Shortest Path Length 62.
Optimal? Surg Endosc. 2008;22:9439.
58. Chmarra MK, Bakker NH, Grimbergen CA, Dankelman J.
TrEndo, a device for tracking minimally invasive surgical
instruments in training set-ups. Sensor Actuat A-Phys. 63.
2006;126:32834.
59. Satava RM. Assessing surgery skills through simulation.
Clinical Teacher 2006;3:10711. 64.
60. Lehmann KS, Ritz JP, Maass H, Cakmak HK,
Keuhnapfel UG, Germer CT, et al. A prospective randomized
study to test the transfer of basic psychomotor skills from
virtual reality to physical reality in a comparable training
setting. Ann Surg. 2005;241:4429. 65.
61. Heijnsdijk EAM, Pasdeloup A, van der Pijl AJ, Dankelman J,
Gouma DJ. The inuence of force feedback and visual
feedback in grasping tissue laparoscopically. Surg Endosc.
2004;18:9805.
Chmarra MK, Grimbergen CA, Dankelman J.
Systems for tracking minimally invasive surgical instru-
ments. Minim Invasiv Ther & Allied Technol. 2007;16:
32840.
Sokollik C, Gross J, Buess G. New model for skills assessment
and training progress in minimally invasive surgery. Surg
Endosc. 2004;18:495500.
Rosen J, Brown JD, Chang L, Sinanan M, Hannaford B.
Generalized approach for modeling minimally invasive
surgery as a stochastic process using a discrete Markov
model. IEEE Transactions on Biomedical Engineering
2006;53:399413.
Kahol K, Leyba MJ, Deka M, Deka V, Mayes S, et al. Effect of
fatigue on psychomotor and cognitive skills. Am J Surg.
2008;195:195204.
Copyright of Minimally Invasive Therapy & Allied Technologies is the property of Taylor & Francis Ltd and
its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's
express written permission. However, users may print, download, or email articles for individual use.