Surg Endosc (2004) 18: 16081612
DOI: 10.1007/s00464-003-9312-5
Springer Science+Business Media, Inc. 2004
Bimodal assessment of laparoscopic suturing skills
Construct and concurrent validity
K. Moorthy, Y. Munz, A. Dosis, F. Bello, A. Chang, A. Darzi
Department of Surgical Oncology and Technology, Imperial College, St. Marys Hospital, 10th Floor, QEQM Building, Praed Street,
W2 1NY, London
Received: 27 November 2003/Accepted: 27 May 2004/Online publication: 13 October 2004
Abstract
Background: The assessment of technical skills should
provide objective feedback and judge suitability of
progress during training. The aim of this study was to
validate two objective assessment techniques for lapa-
roscopic suturing and demonstrate a correlation be-
tween them.
Methods: Six experts, seven surgeons skilled in laparo-
scopic suturing, and 13 with no laparoscopic suturing
skills were asked to place two or three intracorporeal
sutures on a synthetic suture pad. The latter group was
given video-based instructions prior to the execution of
the sutures. Ergonomic conditions were standardized for
all subjects. The procedures were recorded on videotape
and two blinded observers rated the rst suture of each
subject on a 29-point checklist. A motion analysis sys-
tem, Imperial College Surgical Assessment Device, was
used to assess psychomotor skills.
Results: There was a signicant dierence in the time
taken (p = 0.000) and total path length (p = 0.000) per
suture across the groups. There were also a signicant
dierence in the total checklist score (p = 0.000) and its
individual categories. The was a strong correlation be-
tween the total path length and the total checklist score
(coecient, 0.78; p < 0.001).
Conclusions: A combination of the motion analysis
system and the checklist would make the process of
formative feedback during the learning of intracorporeal
suturing objective and comprehensive.
Key words: Technical skills and surgery Laparo-
scopic skills and assessment Motion analysis
Correspondence to: K. Moorthy
The high incidence of complications associated with
laparoscopic cholecystectomy during the early 1990s [4]
was probably a result of the failure to appreciate the fact
that the performance of laparoscopic surgery required
its own set of unique skills compared to open surgery. In
laparoscopic surgery, surgeons have only a two-dimen-
sional view of the operative eld, there is an absence of
tactile feedback, they have to get accustomed to the
fulcrum eect, and they also have to develop intuitive
handeye coordination and depth perception [7, 14, 18].
As a result, there is a learning curve associated with
laparoscopic procedures. It has also been found that
learning curves vary for dierent procedures. For
example, the learning curve for laparoscopic cholecys-
tectomy is approximately 2550 procedures [1, 9] and
for laparoscopic colorectal surgery approximately 50
procedures [6]. However, even the number of procedures
required to achieve competence varies among studies [6,
17]. In addition, learning curves for laparoscopic surgery
usually rely on time taken for the procedure [11]. Time is
a crude measure because it depends not only on the skill
of the surgeon but also on patient variability. Therefore,
if surgeons have to receive accreditation for procedures,
as has been suggested [16], the process of assessment
should be made more objective and reliable.
One of the main drawbacks to the current model of
psychomotor skills learning is the absence of objective
feedback [13]. Feedback is essential because deciencies
in training are dicult to correct without feedback [8].
In addition, feedback should be as objective as possible
because surgical trainees already have a certain percep-
tion of self that may be dicult to inuence. Even
though technical prociency is a small aspect of surgical
competence [20], it is considered to be of crucial
importance especially during training [3]. In a workshop
of master surgeons, the consensus was that technical
skills assessment should be an important part of a sur-
gical training program [3]. However, the assessment of
skills depends largely on the subjective evaluation of
trainers. It has been suggested that assessment by direct
 1609

observation without criteria has only a moderate level of

validity and little reliability because it is often dicult
for two trainers to agree on the skill level of a trainee
[13].
In 1971, Kopta [12] suggested the use of checklists
for the assessment of technical skills among orthopedic
trainees. Since then, checklists have been used across a
number of specialties, including laparoscopic surgery [5].
The availability of set criteria against which technical
skills can be assessed makes the assessment process more
objective, valid, and reliable [22]. It has been said that
checklists turn examiners into observers of behavior
rather than interpreters of behavior, thereby removing
the subjectivity of the evaluation process [12].
The aim of this study was to develop and establish
the construct validity of a checklist for the assessment of
laparoscopic suturing and to establish its concurrent
validity against a validated method of assessment of Fig. 1. Imperial College Surgical Assessment Device. A, signal gen-
erator; B, receivers/sensors.
technical skilldexterity analysis.

Materials and methods

Surgeons of varying levels of experience in laparoscopic surgery were

enrolled in the study. Informed consent was obtained from all par-
ticipants. The surgeons were divided into three groups. Group 1 con-
sisted of experts who had performed more than 100 endoscopic sutures
during operative procedures or had frequently practiced in pelvi-
trainers; group 2 consisted of surgeons who possessed laparoscopic
skills from having performed 1050 laparoscopic cholecystectomies
and had performed less than 50 endoscopic sutures in a pelvi-trainer;
group 3 consisted of trainees who had performed fewer than 10 lap-
aroscopic cholecystectomies but did not possess any laparoscopic
suturing skills.

Study design
All the surgeons were asked to place between one and three sutures on
a synthetic suture pad (Annexe Art, Anglesey, Wales, UK) placed
within a box trainer within a period of 15 minutes. There were pre-
determined marks on the suture pad on either side of an incision
through which the participants were instructed that the needle should
enter and exit. They were instructed to exit through the incision before
going through the opposite side. The angle of the laparoscope, the
distance of the laparoscope from the suture pad, the port positions, the
suture material (30 vicryl, USSC, Norwalk, CT, USA), and the needle
holders were all standardized. All surgeons were requested to under-
take the two-handed collapsed reef knot technique followed by a single
locking throw as taught in the endoscopic suturing course conducted
by the Royal College of Surgeons of England. The trainees from group
3 were given video-based instruction on endoscopic suturing by an Fig. 2. Check list for endoscopic suturing.
expert surgeon prior to performing the task.

Assessment of performance
Imperial college surgical assessment device
This is an electromagnetic motion tracking system in which the posi-
tional data from sensors worn on the dorsum of a surgeons hands are
converted into meaningful dexterity data using special software
(Fig. 1). The dexterity parameters are the number of movements, the
path length traveled by the hands, the speed of movements, and the
time taken for the task. Previous studies have shown that path length is
a discriminator of skill (construct validity) for simple [21] and complex
[19] laparoscopic tasks. Thus, time and path length per suture were
used as the dexterity parameters for this study.
In addition, all the tasks were videotaped, and the rst suture of
each participant was played back to two independent blinded
observers who rated the task performance using the checklist.
Checklist
The checklist consists of six categories of 29 items, which are scored as
a 1 or 0 (Fig. 2). These categories are needle loading (NL), needle
driving (ND), pulling the suture through (PS), technique of tying knots
(knot technique, KT), Knot slippage (KS), and knot quality (KQ). The
rst two categories are presented as two phases because the subjects
were requested to exit though the incision prior to driving the needle
through the opposite side. For purposes of standardization, the par-
1610

Table 1. Assessment parametersa

Time in Path length Checklist score

Group No. (sec) (cm) as % age

1 (n = 6) 128 (37.2) 1299.23 (437.7) 84.46 (9.8)

2 (n = 7) 248 (131.8) 2110.04 (708.9) 63.6 (20.05)
3 (n = 13) 667 (432.2) 10050 (7554.5) 28.8 (19.2)
a
Median values; values in parentheses, interquartile range

Fig. 4. Path length per suture in centimeters. The central black line is
the median, the data within the box are the interquartile range, and the
ends of the vertical line denotine the whole range. The circle denotes
outliers.

Fig. 3. Time taken per suture in seconds. The central black line is the
median, the data within the box are the interquartile range, and the
ends of the vertical line denote the whole range.

ticipants were requested to load the needle on the needle holder and
place the instruments on the suture pad before the Imperial College
Surgical Assessment Device (ICSAD) was started, and they were then
asked to commence the task. As result, two of the items on the
checklist relating to needle positioning before the task commenced
were eliminated from the analysis. The total checklist score is expressed
as a percentage. Comparison between the groups consisted of ana-
lyzing the total checklist score as well as the six individual categories.

Data analysis
Nonparametric tests were used for statistical analyses. Construct
validity was established by analyzing dierences across all groups
using the Kruskal-Wallis test. The Mann-Whitney U test was used to
analyze the dierences between the groups. We also performed an
intergroup analysis using the Mann-Whitney U test to evaluate any
dierences between the subjects in group 3 who possessed laparoscopic
skills and those who did not.
Spearmans rank correlation test was used to demonstrate the
correlation between the path length and the checklist score. We
determined the interrater reliability for the checklist score using
Cronbachs alpha coecient.
Results
There were six surgeons in group 1, seven in group 2,
and 13 in group 3. Table 1 gives the median and inter-
quartile ranges of all three groups for the assessment
parameters. There was a signicant dierence across the
three groups for the time taken per suture (p = 0.000)
Fig. 5. Checklist score for rst suture; % age, percentage score. The
central black line is the median, the data within the box are the in-
terquartile range, and the ends of the vertical line denote the whole
range. The circle denote outliers.
(Fig. 3), for the path length per suture (p = 0.000)
(Fig. 4), and for the total checklist score (p = 0.000)
(Fig. 5). Analysis of the individual categories of the
checklist revealed that there was a signicant dierence
across the groups for four of the six components except
PS and KS (Figs. 6 and 7).
There were signicant dierences between groups 1
and 2 for path length (p = 0.01) but not for time taken
per suture (p = 0.06). There were signicant dierences
between groups 2 and 3 for time (p = 0.001) and for
path length (p = 0.000).
There was a signicant dierence between groups 1
and 2 (p = 0.01) and between groups 2 and 3 for the
total checklist score (p = 0.001). Comparison between

Fig. 6. Cluster box plots for needle loading (NL) and needle driving
(ND). The central black line is the median, the data within the box are
the interquartile range, and the ends of the vertical line denote the
whole range. Circles denote outliers, and asterisks denote the extreme
observations.
Fig. 7. Cluster box plots for knot technique (KT) and knot quality
(KQ). The central black line is the median, the data within the box are
the interquartile range, and the ends of the vertical line denote the
whole range. Circles denote outliers, and asterisks denote the extreme
observations.
the groups for the individual checklist categories re-
vealed that although group 1 scored higher than group 2
(Figs. 6 and 7) for nearly all the checklist categories, the
dierence was only signicant for KT (p = 0.02). There
was a signicant dierence between groups 2 and 3 for
all the categories except PS and KS (Figs. 6 and 7).
Subgroup analysis of group 3 revealed that there
were no dierences between the trainees who had pre-
viously performed laparoscopic cholecystectomy and
those who had not (time, p = 0.38; path length,
p = 0.66; checklist, p = 0.92). There was also a strong
and signicant correlation between the path length and
1611
the checklist score (rho = )0.78, p = 0.000). The in-
terrater reliability for the checklist score was 0.90.
Discussion
Time to perform the task has been the only objective
measure used for the assessment of endoscopic suturing
skills in many studies [2, 10, 15]. Smith et al. [18] stated
that simply measuring the operative time does not
indicate mastery of laparoscopic manipulative skills.
They also suggested that accuracy is a more sensitive
indicator of skill than time. In our study, although there
was a signicant dierence across all the groups, time
per suture was not sensitive enough to detect a dierence
between the two groups that possessed intracorporeal
endoscopic suturing skill. ICSADs parameters help in
quantifying dexterity and in understanding expert per-
formance. In a previous study, it was found that expert
surgeons perform a task with a lower path length, which
reects their economy of motion and their accuracy [19,
21]. Similarly, in our study the path length per suture
dierentiated the skill level across all three groups of
surgeons and was sensitive enough to detect the dier-
ence between the two groups that possessed suturing
skills.
This study has demonstrated the construct validity
of a checklist developed for the assessment of endo-
scopic suturing skills. This checklist was developed after
discussions with expert surgeons. The six categories of
the checklist are similar to the categories of assessment
of suturing skills developed by Mori et al. [10]. Our
study has shown that the total checklist score is sensitive
to detect dierences across the groups and even in be-
tween the groups.
Checklists make the assessment of psychomotor
skills criteria based and thus increase the validity and
reliability of the assessment process [13]. In a previous
study comparing a checklist and a global rating scale
embedded within the Objective Structured Assessment
of Technical Skills, checklists were found to possess
poor reliability in comparison to the global rating scale
[12]. One of the reasons for the high interrater reliability
of our checklist is probably that every checklist item was
anchored by explicit criteria for that particular item to
be scored as a 1 or a 0. For example, if there was more
than one entry through the suture pad, or even one
fumble during the throws while tying the knot, the item
was scored as a 0.
The checklist evaluates task execution for mainly
psychomotor skills and bimanual coordination. How-
ever, there are some items that assess safety, such as the
needle on the needle holder being in view at all times and
the knots being squared and secured. All categories of
the checklist, except for PS and KS, were able to detect
dierences across the groups. One of the reasons for the
apparent lack of dierence for KS is that surgeons from
even group 1 (experts) did not keep the needle held on
the needle loader in view while pulling the suture
through. This may be considered to be an unsafe act due
to the likelihood of inadvertent damage to nearby
1612
structures. The experts did score higher than the inter-
mediate group on nearly all categories but the dierence
was only signicant for KT. This is one of the most
complex tasks in laparoscopic surgery and requires
bimanual dexterity and probably is achieved only with
considerable practice.
We have also demonstrated the concurrent validity
of the checklist by demonstrating the strong correlation
between the checklist score and path length. Despite
this, checklists and motion analysis systems do have
drawbacks. Checklist assessments involve either direct
observations of task performance or periods of intense
video watching. ICSAD is a quantitative measure of
skill and does not give any information about the
quality of performance. For example, a surgeon could
execute a laparoscopic knot without squaring the
throws, take fewer throws, or complete a loose knot and
yet execute the knot with fewer movements and a lower
path length. Thus, a comprehensive assessment of per-
formance should integrate quantitative measures of
dexterity using ICSAD with a qualitative assessment of
performance using checklists. The main aim of our
assessment method of combining the checklist and IC-
SAD is to make training in endoscopic suturing skills
superior with objective feedback.
Other than the obvious benets to training and
credentialing of surgeons, objective measures of lapa-
roscopic skills assessment may have extensive research
applications, such as the evaluation of the benets of
training with virtual reality systems, for studying the
eect of adverse conditions such as sleep deprivation
and shift patterns on performance, and to objectively
demonstrate the benet of training courses.
References
1. Cagir B, Rangraj M, Mauci L, Herz BL (1994) The learning
curve for laparoscopic cholecystectomy. J Laparoendosc Surg 4:
419427
2. Chung JY, Sackier JM (1998) A method of objectively evalu-
ating improvements in laparoscopic skills. Surg Endosc 12:
11111116
3. Cuschieri A, Francis N, Crosby J, Hanna GB (2001) What do
master surgeons think of surgical competence and revalidation?
Am J Surg 182: 110116
4. Deziel DJ, Millikan KW, Economou SG, Doolas A, Ko ST, Airan
MC (1993) Complications of laparoscopic cholecystectomy: a
national survey of 4,292 hospitals and an analysis of 77,604 cases.
Am J Surg 165: 914
5. Eubanks TR, Clements RH, Pohl D, Williams N, Schaad DC,
Horgan S, et al. (1999) An objective scoring system for laparo-
scopic cholecystectomy. J Am Coll Surg 189: 566574
6. Fleshman JW, Fry RD, Birnbaum EH, Kodner IJ (1996) Lapa-
roscopic-assisted and minilaparotomy approaches to colorectal
diseases are similar in early outcome. Dis Colon Rectum 39: 1522
7. Gallagher AG, Satava RM (2002) Virtual reality as a metric for
the assessment of laparoscopic psychomotor skills. Learning
curves and reliability measures. Surg Endosc 16: 17461752
8. Kopta JA (1971) An approach to the evaluation of operative skills.
Surgery 70: 297303
9. Lekawa M, Shapiro SJ, Gordon LA, Rothbart J, Hiatt JR (1995)
The laparoscopic learning curve. Surg Laparosc Endosc 5: 455
458
10. Mori T, Hatano N, Maruyama S, Atomi Y (1998) Signicance of
hands-on training in laparoscopic surgery. Surg Endosc 12:
256260
11. Peters JH, Ellison EC, Innes JT, Liss JL, Nichols KE, Lomano
JM, et al. (1991) Safety and ecacy of laparoscopic cholecystec-
tomy. A prospective analysis of 100 initial patients. Ann Surg 213:
312
12. Regehr G, MacRae H, Reznick RK, Szalay D (1998) Comparing
the psychometric properties of checklists and global rating scales
for assessing performance on an OSCE-format examination. Acad
Med 73: 993997
13. Reznick RK (1993) Teaching and testing technical skills. Am J
Surg 165: 358361
14. Rosser JC, Rosser LE, Savalgi RS (1997) Skill acquisition and
assessment for laparoscopic surgery. Arch Surg 132: 200204
15. Rosser JC Jr, Rosser LE, Savalgi RS (1998) Objective evaluation
of a laparoscopic surgical skill program for residents and senior
surgeons. Arch Surg 133: 657661
16. Royston CM, Lansdown MR, Brough WA (1994) Teaching lap-
aroscopic surgery: the need for guidelines. Br Med J 308: 1023
1025
17. Simons AJ, Anthone GJ, Ortega AE, Franklin M, Fleshman J,
Geis WP, et al. (1995) Laparoscopic-assisted colectomy learning
curve. Dis Colon Rectum 38: 600603
18. Smith CD, Farrell TM, McNatt SS, Metreveli RE (2001) Assess-
ing laparoscopic manipulative skills. Am J Surg 181: 547550
19. Smith SG, Torkington J, Brown TJ, Tander NJ, Darzi A (2002)
Motion analysis. Surg Endosc 16: 640645
20. Spencer FC, Spencer FC (1978) Teaching and measuring surgical
techniquesthe technical evaluation of competence. Bull Am Coll
Surgeons 63: 912
21. Tander N, Smith S, Mair J, Russell R, Darzi A (1999) Can a
computer measure surgical precision? Reliability, validity and
feasibility of the ICSAD. Surg Endosc 13(suppl 1): 81
22. Watts J, Feldman WB (1985) Assessment of technical skills. In:
Neufeld VR, Norman GR (eds). Assessing clinical competence.
Springer, New York, pp 259274