Spine Deformity 6 (2018) 762e770

www.spine-deformity.org

Predicting Surgical Complications in Patients Undergoing Elective

Adult Spinal Deformity Procedures Using Machine Learning
Jun S. Kim, MDa, Varun Arvind, BSa, Eric K. Oermann, MDb, Deepak Kaji, BAa,
Will Ranson, BSa, Chierika Ukogu, BAa, Awais K. Hussain, BAa, John Caridi, MDb,
Samuel K. Cho, MDa,*
a
Department of Orthopaedic Surgery, Icahn School of Medicine at Mount Sinai, New York, NY, 10029, USA
b
Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, 10029, USA
Received 30 July 2017; revised 25 February 2018; accepted 1 March 2018

Abstract
Study Design: Cross-sectional database study.
Objective: To train and validate machine learning models to identify risk factors for complications following surgery for adult spinal
deformity (ASD).
Summary of Background Data: Machine learning models such as logistic regression (LR) and artificial neural networks (ANNs) are
valuable tools for analyzing and interpreting large and complex data sets. ANNs have yet to be used for risk factor analysis in orthopedic
surgery.
Methods: The American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) database was queried for
patients who underwent surgery for ASD. This query returned 4,073 patients, which data were used to train and evaluate our models. The
predictive variables used included sex, age, ethnicity, diabetes, smoking, steroid use, coagulopathy, functional status, American Society of
Anesthesiologists (ASA) class O3, body mass index (BMI), pulmonary comorbidities, and cardiac comorbidities. The models were used to
predict cardiac complications, wound complications, venous thromboembolism (VTE), and mortality. Using ASA class as a benchmark for
prediction, area under receiver operating characteristic curves (AUC) was used to determine the accuracy of our machine learning models.
Results: The mean age of patients was 59.5 years. Forty-one percent of patients were male whereas 59.0% of patients were female. ANN
and LR outperformed ASA scoring in predicting every complication (p!.05). The ANN outperformed LR in predicting cardiac
complication, wound complication, and mortality (p!.05).
Conclusions: Machine learning algorithms outperform ASA scoring for predicting individual risk prognosis. These algorithms also
outperform LR in predicting individual risk for all complications except VTE. With the growing size of medical data, the training of
machine learning on these large data sets promises to improve risk prognostication, with the ability of continuously learning making them
excellent tools in complex clinical scenarios.
Level of Evidence: Level III.
Ó 2018 Scoliosis Research Society. All rights reserved.
Keywords: Adult spinal deformity; Machine learning; Artificial neural network; Logistic regression; Risk prediction

Author disclosures: JSK (none), VA (none), EKO (none), DK (none),
WR (none), CU (none), AKH (none), JC (none), SKC (grants from Zim-
mer, Orthopaedic Research and Education Foundation, and Stryker,
outside the submitted work).
This study was approved by the Institutional Review Board of the
Icahn School of Medicine at Mount Sinai, New York, NY.
*Corresponding author. Department of Orthopaedic Surgery, Icahn
School of Medicine at Mount Sinai, 5 East 98th Street, Box 1188, New
York, NY 10029, USA. Tel.: (212) 241-0276; fax: (646) 537-8531.
E-mail address: samuel.cho@mountsinai.org (S.K. Cho).
2212-134X/$ - see front matter Ó 2018 Scoliosis Research Society. All rights reserved.
https://doi.org/10.1016/j.jspd.2018.03.003
Introduction
The advent of digital technology, machine learning and
deep learning in particular, is increasingly making it
possible to utilize big data to more precisely risk stratify
and prognosticate how an individual patient will behave
given a disease or intervention. Machine learning has
already been used in other realms such as retail and search
engines. However, healthcare has lagged in the uptake of
newer techniques to leverage the rich information contained
in electronic health records (EHRs).
 J.S. Kim et al. / Spine Deformity 6 (2018) 762e770
The practice of evidence-based medicine has sustained
the progress seen in modern care and diagnosis. Traditional
statistical approaches have gleaned much about what is
known regarding risk factors used for prognostication. Ma-
chine learning (ML) combines these fundamental statistical
insights with modern high-performance computing to learn
patterns that can be used for recognition and prediction.
Importantly, machine learning often identifies patterns that
are not readily apparent to human intuition, thus identifying
otherwise unknown connections [1]. Multivariate logistic
regression and artificial neural networks are the two most
commonly used machine learning models employed in
medicine [2]. Artificial neural networks were first developed
to model the neural architecture of the brain. Harnessing the
structure of biology, artificial neural networks (ANNs) are
particularly well suited for modeling complex, nonlinear data
when little is known regarding the underlying distribution of
the data or colinearity among the variables [3]. Importantly,
ANNs can perform these functions without prior assump-
tions, leading to a highly adaptable system less susceptible to
anchoring biases [3]. However, similar to any machine
learning algorithm, neural networks are susceptible to
intrinsic limitations and biases of the underlying data set.
Additionally, limitations of model design such as neural
network architecture, feature selection, and optimization
functions can lead to model biases and overfitting that
decrease generalizability and prognostication value of neural
networks on external data [4]. Advancements in neural
network science and proper implementation with recognition
of these limitations are important for future integration of
machine learning in surgical practice, and the utility of ma-
chine learning in adult deformity surgery has not yet
been explored.
Adult spinal deformity (ASD) is a spinal disorder
defined as a complex spectrum of spinal diseases that
present in adulthood including adult scoliosis (progression
of childhood scoliosis), degenerative scoliosis, sagittal and
coronal imbalance, and iatrogenic deformity (with or
without spinal stenosis) [5]. Adult degenerative scoliosis is
the most common cause of ASD and is commonly seen in
elderly adults, particularly those older than 60 years, as
degeneration of intervertebral discs and facet joints exac-
erbate scoliotic curvature [6]. With the aging baby boomer
generation and overall population structure of the United
States, it is not surprising that the demand and prevalence
for ASD surgery continues to increase [7]. In the bur-
geoning era of rising healthcare costs and greater scrutiny
over surgical outcomes, there has been increasing emphasis
on understanding the risk factors and possible predictors to
optimize perioperative planning and management. Data-
driven clinical decision support tools have the potential to
lead to cost savings by leveraging the information con-
tained in large medical databases. Uptake of machine
learning approaches in the realm of spinal surgery have
lagged. However, the patient population and associated
increased rate of postoperative complications renders ASD
763
a prime target for quality improvement through the utili-
zation of machine learning.
ML algorithms have the capability of ‘‘learning’’ using
newly generated information to improve their predictive
capability. Briefly explained, these algorithms work by
utilizing a subset of the overall study data (70% in this
case) to ‘‘train’’ and create an accurate predictive model.
This established model is then validated using the
remainder of the data to determine the accuracy of the post-
training model. This study seeks to develop and validate
ML algorithms to precisely predict complications following
ASD using a national database, in order to compare ML
algorithms with logistic regression (LR) or American So-
ciety of Anesthesiologists (ASA) classification.
Methods
Patient selection and preprocessing
The National Surgical Quality Improvement Program
(NSQIP) database was used for the purpose of training and
validating ANN and LR models. Adult patients (>18 years)
undergoing adult deformity surgery were identified based
on Current Procedural Terminology (CPT) codes 22800,
22802, 22804, 22808, 22810, 22812, 22818, 22819. CPT
codes 22843, 22844, 22846, or 22847 were also included to
capture long, multilevel fusion constructs. Patients with
CPT code 22842 and 22845 were included if they had an
ICD-9 diagnosis for spinal deformity (including 737.1,
737.2, 737.3, 737.4, 737.8, 737.9). Cases with missing
preoperative data, emergency cases, patients with a wound
class of 2, 3, or 4, an open wound on their body, current
sepsis, current pneumonia, prior surgeries within 30 days,
cases requiring cardiac pulmonary resuscitation prior to
surgery, or neoplasm of spine were excluded in order to
reduce the risk of confounding variables. This resulted in
5,818 ASD patients included in our analysis.
Training and Holdout Data Sets
For development of our models, 70% of the initial data
(training set) were used for training while 30% (holdout)
was set aside for posttraining evaluation. To overcome the
low sample size for positive complication cohorts, the
adaptive synthetic sampling (ADASYN) approach for
imbalanced learning was used to artificially generate pos-
itive complication cases based on the training data set to
increase the number of positive cases in the training set to
improve class balance. In brief, ADASYN utilizes exam-
ples from the minority class that are difficult to learn and
generates synthetic new cases based on these examples to
improve model learning and generalizability [8].
Feature selection
Input features used for training include sex, age,
ethnicity (white, black, Hispanic, or other), history of
764 J.S. Kim et al. / Spine Deformity 6 (2018) 762e770

diabetes (insulin dependent and independent), history of
smoking (within one year of surgery), steroid use (<30
days prior to surgery), history of bleeding disorders, func-
tional status (independent or partially/totally dependent),
ASA >3, BMI, and presence of pulmonary (ventilator
dependent <48 hours prior to surgery or history of chronic
obstructive pulmonary disease <30 days prior to surgery)
or cardiac comorbidities (use of antihypertensive medica-
tion or history of chronic heart failure <30 days prior to
surgery). All variables were selected from the ACS-NSQIP
database and the ACS-NSQIP guide may be consulted for
further explanation of variables.
In machine learning applications, the number of training
examples required to reach a given accuracy grows expo-
nentially with the number of irrelevant features [9]. To
combat this, feature selection was performed to prevent
overfitting and improve the overall generalizability of our
models. Logistic regression analysis was performed on the
training data set, to obtain probability coefficients for each
feature. Stepwise entry and removal of features was used.
Prefiltering of features was done to minimize collinearity in
the final model by a clinician with domain knowledge. The
top six features identified as having the greatest regression
coefficient magnitudes were chosen as input variables for
the ANN and LR for mortality, venous thromboembolism
(VTE), cardiac complications, and wound complications,
respectively. Age and BMI were treated as continuous
variables, all other categorical features were handled using
one-hot encoding.
Machine learning construction and testing
Machine learning models were trained to predict
occurrence of mortality, venous thromboembolism, wound
complications, and cardiac complications. Deep ANNs
were constructed using the Neural Network toolbox in
MatLab 2016b (MathWorks, Inc., Natick, MA). L2 regu-
larization was used to combat ANN overfitting, by aug-
menting the error function used for training with the
squared magnitude of the weights used in the ANN. This
prevents overly complex models that are overfitted to a
specific data set, improving predictive generalizability.
Because of the large class imbalance even post-ADASYN,
multiple ANNs were created by partitioning the majority
class into subsets in a 1:1 ratio with the minority class,
generating ANNs trained off of each partition. Subse-
quently, patients were distributed in a 2:1:1 ratio to
generate training, validation, and testing data sets to train
each ANN in a fivefold cross-validation scheme. Data that
was not used for training (holdout data) was used for final
testing of the ANN to provide an unbiased assessment of
ANN performance. Data from the holdout set was fed into
each ANN, and final predictions were based off of indi-
vidual accuracy-weighted predictions surveyed across each

Fig. 1. (A) Schematic of study workflow. (B) Diagram of ANN model. Bar lengths represent number of patient cases. ADASYN increases the number of
positive cases to combat class imbalance. Negative cases are then partitioned in a 1:1 ratio with the positive cases to create a class-balanced data set used for
ANN training. Each partition trains an independent neural net. During evaluation, data is fed through each neural net where the responses are surveyed,
weighted by the model’s accuracy, and the net prediction is used. ADASYN, adaptive synthetic sampling; ANN, artificial neural network.
 J.S. Kim et al. / Spine Deformity 6 (2018) 762e770 765

ANN. Lastly, the patient data set was randomized and cohort, 4,073 patients (70%) were included into the training
repartitioned, and training and testing was performed again, set and 1,746 patients (30%) were used as a holdout
iteratively five times to perform statistical analysis of training set for evaluating the trained machine learning
ANN performance in a process known as ensembling. models (Fig. 1). Following our exclusion criteria, 2,376
Performance of the ANN was compared to traditional (41.0%) of patients were male, whereas 3,418 (59.0%)
logistic regression that was trained and tested on the same were female. The mean age was 59.5 years old and the
data that the ANN was evaluated on. Furthermore, these cohort exhibited low rates of complications across all out-
two machine learning models were compared to the ASA comes. Our study uses cardiac complications, VTE, wound
physical status classification system. Classification perfor- complications, and mortality as target outcomes. Specif-
mance for ANN, LR, and ASA was evaluated based on area ically, there was a 0.5% (29 patients) rate of mortality, 2%
under the receiver operating characteristic curve (AUC) (139) rate of wound complication, 2% (105) rate of VTE,
with 95% confidence intervals (CIs). and 0.7% (39) rate of cardiac complication (Table 1).
Additionally, there were low rates of overlap between
Source of funding complications except between mortality and cardiac com-
plications. Overall, 37.9% of patients who did not survive
There was no external source of funding. had preexisting cardiac complications. Unsurprisingly, age
was a highly predictive feature across all outcomes. Dia-
Results betic status and tobacco usage were also useful features,
which is consistent with their known association with poor
Data and analysis pipeline
clinical outcomes (Fig. 2) (Table 2) [10-12]. Furthermore,
A total of 5,818 patients were identified as having un- 214, 223, 217, and 212 patients were excluded from the
dergone ASD surgery between 2010 and 2014. Among this cardiac complication, VTE, wound complication, and
Table 1
Patient characteristics for patients included within the data set for model construction.
Feature Average Total Cardiac complication VTE complication Wound complication Mortality
Sex, n (%)
Male 2376 (41.0) 21 (0.9) 50 (2.1) 55 (2.3) 15 (0.6)
Female 3418 (59.0) 18 (0.5) 55 (1.6) 84 (2.5) 14 (0.4)
Age, mean 59.5
Ethnicity, n (%)
White 4703 (81.2) 26 (0.6) 89 (1.9) 111 (2.4) 22 (0.5)
Black 441 (7.6) 6 (1.4) 8 (1.8) 11 (2.5) 1 (0.2)
Hispanic 218 (3.8) 1 (0.5) 2 (0.9) 3 (1.4) 1 (0.5)
Other 432 (7.5) 6 (1.4) 6 (1.4) 14 (3.2) 5 (1.2)
Diabetes mellitus, n (%)
No 4947 (85.4) 24 (0.5) 91 (1.8) 111 (2.2) 20 (0.4)
Type 2 575 (9.9) 7 (1.2) 11 (1.9) 22 (3.8) 7 (1.2)
Type 1 272 (4.7) 8 (2.9) 3 (1.1) 6 (2.2) 2 (0.7)
Smoking history, n (%)
Smoker 1207 (20.8) 6 (0.5) 11 (0.9) 26 (2.2) 9 (0.7)
Nonsmoker 4587 (79.2) 33 (0.7) 94 (2.0) 113 (2.5) 20 (0.4)
Steroid use, n (%)
Steroid use 219 (3.8) 1 (0.5) 15 (6.8) 5 (2.3) 2 (0.9)
No steroid use 5575 (96.2) 38 (0.7) 90 (1.6) 134 (2.4) 27 (0.5)
History of bleeding disorder, n (%) 189 (0.9) 1 (0.5) 0 (0.0) 1 (0.5) 3 (1.6)
None 70 (1.2) 0 (0.0) 2 (2.9) 3 (4.3) 0 (0.0)
Functional status, n (%) 5724 (98.8) 39 (0.7) 103 (1.8) 136 (2.4) 29 (0.5)
Dependent 265 (4.6) 5 (1.9) 6 (2.3) 10 (3.8) 4 (1.5)
Independent 5529 (95.4) 34 (0.6) 99 (1.8) 129 (2.3) 25 (0.5)
ASA score, n (%)
>3 3078 (53.1) 33 (1.1) 60 (1.9) 99 (3.2) 28 (0.9)
BMI, n (%) 29.5
Comorbidities, n (%)
Pulmonary 294 (5.1) 2 (0.7) 6 (2.0) 6 (2.0) 5 (1.7)
Cardiac 3159 (54.5) 25 (0.8) 59 (1.9) 89 (2.8) 20 (0.6)
Complications, n (%)
Mortality 29 (0.5) 11 (37.9) 7 (24.1) 2 (6.9) 29 (100.0)
Wound complications 139 (2.4) 3 (2.2) 12 (8.6) 139 (100.0) 2 (1.4)
VTE 105 (1.8) 4 (3.8) 105 (100.0) 12 (11.4) 7 (6.7)
Cardiac complications 39 (0.7) 39 (100.0) 4 (10.3) 3 (7.7) 11 (28.2)
ASA, American Society of Anesthesiologists; BMI, body mass index; VTE, venous thromboembolism.
formed least effectively for all target outcomes, with an
CI: 0.76e0.80) for mortality. The ASA classifiers per-
CI: 0.56e0.58) for wound complications, and 0.787 (95%
0.547 (95% CI: 0.54e0.55) for VTE events, 0.575 (95%
0.690 (95% CI: 0.68e0.69) for cardiac complications,
consistently better than ASA as a classifier with an AUC of
predicting mortality. In contrast, the LR performed
wound complications, and 0.844 (95% CI: 0.82e0.86) for
dicting VTE, 0.606 (95% CI: 0.60e0.61) for predicting
diac complications, 0.542 (95% CI: 0.53e0.55) for pre-
an AUC of 0.768 (95% CI: 0.76e0.77) for predicting car-
obtained from the holdout data. The ANN performed with
formance of ANN with LR and ASA, AUC for ANN was
performed by LR (Fig. 4). In order to compare the per-
target except for VTE in which ANN and ASA were out-
ASA classifiers were outperformed by the ANN for every
mance of our classifiers (Fig. 3). The logistic regression and
performance. The AUC was used to measure the perfor-

ANN, LR, and ASA classification performance

data (S1e4).
respectively, to improve learning with class-imbalanced
VTE, wound complication, and mortality training sets,
were generated by ADASYN in the cardiac complication,
the training set. A total of 200, 205, 200, and 198 cases
sampling was used to generate data from minority class in
data (S1e4). As previously described, adaptive synthetic
mortality training sets, respectively, because of incomplete

predictive value, and lighter cells indicate weakly weighted features.

feature selection. Dark cells indicate highly weighted features indicating a strong
Fig. 2. Coefficient weights obtained from logistic regression analysis used for

766
ASA classification was used to benchmark ANN and LR

J.S. Kim et al. / Spine Deformity 6 (2018) 762e770

Table 2
Logistic regression results prior to feature selection.
Cardiac complications Venous thromboembolism Wound complications Mortality
Coefficient SE t Stat p value Coefficient SE t Stat p value Coefficient SE t Stat p value Coefficient SE t Stat p value
(Intercept) 4.277 0.182 23.441 .000 3.564 0.159 22.439 .000 2.920 0.103 28.442 .000 4.320 0.205 21.122 .000
Sex 0.006 0.082 1.398 .162 0.130 0.074 3.219 .001 0.016 0.072 0.907 .364 0.359 0.082 3.366 .001
Age 0.659 0.189 6.098 .000 0.067 0.127 2.475 .013 0.002 0.117 0.435 .664 0.496 0.162 3.745 .000
Black 0.261 0.110 0.291 .771 0.223 0.091 0.621 .535 0.005 0.091 0.247 .805 0.530 0.119 2.028 .043
Hispanic 0.913 0.136 0.873 .383 0.642 0.189 1.792 .073 0.636 0.170 1.838 .066 0.015 0.398 2.123 .034
Other 0.077 0.068 5.449 .000 0.068 0.076 1.191 .234 0.017 0.075 1.168 .243 0.289 0.086 0.865 .387
Diabetes mellitus type 2 0.187 0.083 1.821 .069 0.110 0.094 1.154 .249 0.011 0.065 1.994 .046 0.182 0.079 1.857 .063
Diabetes mellitus type 1 0.283 0.071 6.819 .000 0.248 0.188 2.911 .004 0.395 0.179 2.650 .008 0.224 0.087 0.293 .769
Smoke history 0.196 0.126 3.034 .002 0.475 0.120 3.926 .000 0.139 0.089 1.928 .054 0.161 0.092 2.833 .005
Steroid history 0.159 0.169 2.253 .024 0.438 0.068 5.715 .000 0.258 0.088 0.288 .774 0.031 0.103 0.284 .776
Bleeding disorder 0.015 0.042 0.198 .843 0.582 0.383 2.640 .008 0.306 0.043 0.199 .842 0.773 0.137 0.306 .760
Functional status 0.162 0.070 3.097 .002 0.017 0.094 0.212 .832 0.007 0.120 1.358 .174 0.136 0.077 2.438 .015
ASA >3 0.603 0.115 6.759 .000 0.010 0.082 1.492 .136 0.453 0.081 4.580 .000 1.337 0.165 8.534 .000
BMI 0.445 0.132 0.300 .764 0.178 0.115 1.865 .062 0.585 0.110 1.886 .059 0.382 0.125 1.148 .251
Pulmonary comorbidity 0.030 0.116 1.705 .088 0.059 0.118 1.252 .210 0.138 0.117 1.224 .221 0.099 0.082 0.038 .970
Cardiac comorbidity 0.495 0.095 3.844 .000 0.048 0.084 1.092 .275 0.142 0.083 0.919 .358 0.100 0.097 0.503 .615
ASA, American Society of Anesthesiologists; BMI, body mass index; SE, standard error.
Coefficients with p values !.05 are bolded. The p value for reach final regression model was p<.0001.
 J.S. Kim et al. / Spine Deformity 6 (2018) 762e770 767

Fig. 3. Receiver operating characteristic curves plotting sensitivity versus 1-specificity for ANN (blue), LR (green), ASA score (red), and random-chance
(black-dashed). ANN, artificial neural network; ASA, American Society of Anesthesiologists; LR, logistic regression.

Fig. 4. Heatmap of AUC values from LR, ANN, and ASA when predict-
ing cardiac complications (cardiac), VTE, wound complications (wound),
and mortality. ANN, artificial neural network; ASA, American Society of
Anesthesiologists; AUC, area under the curve; LR, logistic regression;
VTE, venous thromboembolism.

Table 3
Comparison of AUC of a logistic regression, artificial neural network, and
ASA evaluated on blinded data.
LR (95% CI) ANN (95% CI) ASA (95% CI)
Cardiac 0.690 (0.68e0.69) 0.768 (0.76e0.77) 0.469 (0.46e0.47)
VTE 0.547 (0.54e0.55) 0.542 (0.53e0.55) 0.485 (0.47e0.49)
Wound 0.575 (0.56e0.58) 0.606 (0.60e0.61) 0.508 (0.50e0.51)
Mortality 0.787 (0.76e0.80) 0.844 (0.82e0.86) 0.516 (0.49e0.54) Fig. 5. Confusion matrices of trained ANN and LR machine learners eval-
ANN, artificial neural network; ASA, American Society of Anesthe- uated on holdout (A) mortality and (B) wound complication data sets to
siologists; AUC, area under the curve; CI, confidence interval; LR, logistic demonstrate real-world performance. ANN, artificial neural network;
regression; VTE, venous thromboembolism. LR, logistic regression.
768 J.S. Kim et al. / Spine Deformity 6 (2018) 762e770
AUC of 0.469 (95% CI: 0.46e0.47) for cardiac complica-
tions, 0.485 (95% CI: 0.47e0.49) for VTE, 0.508 (95% CI:
0.50e0.51), and 0.516 (95% CI: 0.49e0.54) for mortality
(Table 3). ANN sensitivity outperformed LR sensitivity as
shown for mortality and wound complications in Figure 5.
Discussion
With the advent of large, prospective, multi-institutional
clinical registries, physicians have access to large amounts
of diverse, high-quality clinical data. This has given birth to
ideas such as ‘‘precision medicine’’ with the goal of
developing quantitative models that can be used to predict
health status, prognosticate disease processes, prevent dis-
ease, and reduce complications. Previous groups have
employed the use of ANNs and other ML models to these
data sets [13-16]. However, these studies either trained
models on extremely large databases (O1,400,000 patients)
or on complications with high occurrence rates. These ex-
amples are impractical for independent institutions or for
small-scale procedures with rare complications. Low
occurrence rates in relatively small data sets lead to large
class-imbalances that are a significant challenge in medical
machine learning [17,18]. To this end, we have trained
several supervised machine learning classifiers to predict
the probability of postoperative complications in a rela-
tively small data set (!15,000 patients) that can outper-
form traditional logistic regression and ASA classification
sensitivity with relatively low occurrence rates (!1%).
Furthermore, we have rigorously developed and tested our
models by employing the best practices in machine learning
in this study via performance of automated feature selec-
tion, L2 regularization, testing on blinded holdout data sets,
and comparison to a standard risk-scoring system, in order
to ensure a high standard necessary for implementation of
machine learning in clinical settings.
The ANN model was superior to the LR, and both were
superior to a clinical benchmark, the ASA score, with a
statistically significantly higher AUC when predicting car-
diac complications, wound complications, and mortality.
The sensitivity of the ANN was superior to LR, indicating
an ability to identify a greater portion of positive cases,
correctly identified as positive. This is an important char-
acteristic for use in clinical situations, suggesting that
ANNs are more suited for clinical prognostication than LR.
There is room, however, for significant improvement. The
ANN model had a positive likelihood ratio of 2.61 and 1.59
and a negative likelihood ratio of 0.26 and 0.58 for mor-
tality and wound complication, respectively. In contrast, the
LR model had a positive likelihood ratio approaching 0 and
0 and a negative likelihood ratio of 1 and 1 for mortality
and wound complication, respectively.
The ANN model exhibited a tendency to ‘‘overtreat,’’
and in contrast, the LR model had a tendency to ‘‘under-
treat’’ relative to that of the ANN. The LR model had a
sensitivity of 0% whereas the sensitivity for the ANN
model was 82% where the ANN correctly predicted 9 of 11
mortalities. Optimization of the hyperparameters of the
ANN algorithm, using a more granular, larger training data
set, implementing other techniques for dealing with class
imbalance (eg, undersampling), and altering the structure of
the ANN itself (eg, adding more hidden layers) are ways in
which the performance of the ANN can be improved. We
expect that with a larger, more granular data set and further
model optimization, we can increase the accuracy of our
model. In particular, the use of natural language processing
is a potential mechanism whereby the extensive and rich
data contained within medical notes can be leveraged to
further train artificial intelligence networks.
Automated feature selection with LR showed that age,
male gender, black race, bleeding disorder, high ASA
score, and BMI were strong independent risk factors for
mortality, consistent with current surgical evidence. Wound
complications were predicted by Hispanic race, BMI, dia-
betes mellitus, and high ASA score. VTE was predicted for
by Hispanic race, history of steroid use, bleeding disorder,
and smoking history. Cardiac complications were predicted
by age, Hispanic race, BMI, cardiac comorbidity, and high
ASA score. These findings are echoed heavily by domain
knowledge in prior spine literature [19,20].
A key strength of this study is the adaptability that can
be achieved by interrogating medical data with different
machine learning models. Indeed, neural network archi-
tectures alone are a diverse field of study that seek to design
optimal neural network structures to improve AI pre-
dictions [21]. In this study, a grid search was performed to
identify optimal hyperparameters. However, this was only
carried out over a certain defined domain of hyper-
parameters not considering all types of network structures
and other macro-scale parameters that may be more suited
for medical prognostication. This presents a novel oppor-
tunity to design machine learners that are adept at learning
and prognosticating based on patient data that is highly
diverse, class-imbalanced, and often limited in sample
size [22].
The ability of machine learning to identify at-risk-
patients and predict potential complications has been
clearly demonstrated here, yet the ability to suggest av-
enues of treatment based on predicted complications has
not yet been realized. Future work can take advantage of
EMR and medical literature to suggest optimal treatment
strategies based on key patient data. Such models can not
only guide physicians in the decision-making process but
can also aide health care systems in low-resource set-
tings, provide personalized care, and improve response
times during critical settings. Taken together, the op-
portunities described here can be used to strengthen
medical artificial intelligence, AI, to improve surgi-
cal outcomes.
The use of machine learning has several limitations that
deserve discussion. The performance of any classifier, is
rooted, in part, in the quality of the training data. Therefore,
 J.S. Kim et al. / Spine Deformity 6 (2018) 762e770
weaknesses in the NSQIP are represented as weaknesses in
the neural network classifier. Sources of biases such as
selection bias from the NSQIP database translate to poor
prognostication generalizability of the neural network and
are an inextricable limitation of machine learning models.
Larger national inpatient data sets such as the National
InPatient Sample (NIS) can be used in the future to address
such biases. Such data sets sampling patients with a broad
demographic spectrum can serve to elucidate patterns in the
model that are both more generalizable and predictive of
future complications and risk. A major challenge in medi-
cine is the paucity of highly granular and robust large-scale
data sets for specific operational cohorts. Large-scale da-
tabases remain scattered across institutions and are isolated
to protect patient privacy [23]. An additional source of bias
was the implementation of feature selection during model
training. Increasing number of features with small training
sizes can lead to decreased classifier performances; thus,
the use of feature selection can be used to combat this [24].
However, omission of such features because of feature se-
lection can exclude important data, leading to increased
bias within the model. Larger data sets can be employed in
the future, allowing for the inclusion of more feature
following selection. Furthermore, the NSQIP data set was
not designed with spine surgery outcomes in mind. As a
result, many features that may serve as stronger inputs were
not available. For instance, ICD diagnoses, radiologic data,
spine surgeryespecific complications, and more granular
operative variables are lacking in the NSQIP database. This
poses a limitation that is intrinsic to the data set.
The models presented in this paper have several limi-
tations that deserve attention. Logistic regression is a
generalized linear model that provides a robust and easy
way to interpret model with informative information such
as odds ratios and statistics. However, logistic regression
requires no co-linearity among independent features, which
can be difficult to assume in medical research unless an
independent study has been performed. Artificial neural
networks can be considered a special case of logistic
regression that includes hidden inner layers of logits that in
theory allow neural networks to approximate any function,
while traditional logistic regression is limited. Unlike lo-
gistic regression, following training neural networks cannot
currently be deconstructed to provide any insight into the
structure of the function being approximated. As a result,
neural networks have been considered ‘‘black boxes’’ in
machine learning. This means that although neural net-
works may provide prognosticative value, it is currently not
possible to understand how the networks are using data to
make predictions and thus not possible to provide clinical
reasoning with underlying pathophysiologic support.
Additionally, the prognosticative capability of our models
was dependent on disease prevalence. Class imbalance can
lead to bias when training machine learning algorithms due
to heavily favoring the majority class. As a result of this,
we observed a high false positive rate associated with our
769
models, and future studies can look to modifying the un-
derlying optimization function to address this issue.
However, the ability to predict clinical outcomes using a
multidimensional data set is an attractive prospect, and the
methods in this study can be extrapolated to any clinical data
set where there are class-imbalance issues between those
patients who experience complications and those that do not.
The advent of machine learning algorithms and their
implementation in a healthcare environment makes the use
of such machine learning increasingly possible. In the past,
generalized linear models such as logistic regression have
been the most commonly used classifiers for this purpose.
However, the machine learning models described here,
particularly the ANN, are similarly powerful, and in some
circumstances, far exceed logistic regression. As the ability
to obtain high-quality patient data and computing power
increases over time, it is likely that machine learning
techniques will find themselves increasingly commonplace
in the hospital setting.
References
[1] Cruz JA, Wishart DS. Applications of machine learning in cancer
prediction and prognosis. Cancer Inform 2007;2:59e77.
[2] Dreiseitl S, Ohno-Machado L. Logistic regression and artificial neu-
ral network classification models: a methodology review. J Biomed
Inform 2002;35:352e9.
[3] Deo RC. Machine learning in medicine. Circulation 2015;132:
1920e30.
[4] Crown WH. Potential application of machine learning in health out-
comes research and some statistical cautions. Value Health 2015;18:
137e40.
[5] Youssef JA, Orndorff DO, Patty CA, et al. Current status of adult spi-
nal deformity. Global Spine J 2013;3:51e62.
[6] Birknes JK, White AP, Albert TJ, et al. Adult degenerative scoliosis:
a review. Neurosurgery 2008;63:94e103.
[7] Cowan JA, Dimick JB, Wainess R, et al. Changes in the utilization of
spinal fusion in the United States. Neurosurgery 2006;59:15e20; dis-
cussion 15.
[8] He H, Bai Y, Garcia EA, Li S. ADASYN: adaptive synthetic sam-
pling approach for imbalanced learning. In: 2008 IEEE International
Joint Conference on Neural Networks (IEEE World Congress on
Computational Intelligence). IEEE 2008. p. 1322e8.
[9] Blum AL, Langley P. Selection of relevant features and examples in
machine learning. Artif Intell 1997;97:245e71.
[10] Møller AM, Pedersen T, Villebro N, Munksgaard A. Effect of smok-
ing on early complications after elective orthopaedic surgery. J Bone
Joint Surg Br 2003;85:178e81.
[11] Iorio R, Williams KM, Marcantonio AJ, et al. Diabetes mellitus, he-
moglobin A1C, and the incidence of total joint arthroplasty infection.
J Arthroplasty 2012;27:726e729.e1.
[12] Chen S, Anderson MV, Cheng WK, Wongworawat MD. Diabetes
associated with increased surgical site infections in spinal arthrodesis.
Clin Orthop Relat Res 2009;467:1670e3.
[13] Van Esbroeck A, Rubinfeld I, Hall B, Syed Z. Quantifying surgical
complexity with machine learning: looking beyond patient factors
to improve surgical models. Surgery 2014;156:1097e105.
[14] Hu Z, Simon GJ, Arsoniadis EG, et al. Automated detection of post-
operative surgical site infections using supervised methods with elec-
tronic health record data. Stud Health Technol Inform 2015;216:
706e10.
770 J.S. Kim et al. / Spine Deformity 6 (2018) 762e770
[15] Sohn S, Larson DW, Habermann EB, et al. Detection of clinically
important colorectal surgical site infection using Bayesian network.
J Surg Res 2017;209:168e73.
[16] Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of
the universal ACS NSQIP surgical risk calculator: a decision aid and
informed consent tool for patients and surgeons. J Am Coll Surg
2013;217:833e842.e1.
[17] Krell MM, Wilshusen N, Seeland A, Kim SK. Classifier transfer with
data selection strategies for online support vector machine classifica-
tion with class imbalance. J Neural Eng 2017;14:025003.
[18] Wang Q, Luo Z, Huang J, et al. A Novel Ensemble Method for Imbal-
anced Data Learning: Bagging of Extrapolation-SMOTE SVM. Com-
put Intell Neurosci 2017;2017:1827016.
[19] Somani S, Di Capua J, Kim JS, et al. Comparing NIS and NSQIP:
an independent risk factor analysis for risk stratification in anterior
cervical discectomy and fusion. Spine (Phila Pa 1976) 2017;42:
565e72.
[20] Wang TY, Martin JR, Loriaux DB, et al. Risk assessment and charac-
terization of 30-day perioperative myocardial infarction following
spine surgery: a retrospective analysis of 1346 consecutive adult pa-
tients. Spine (Phila Pa 1976) 2016;41:438e44.
[21] Tsai J-T, Chou J-H, Liu T-K. Tuning the structure and parameters of a
neural network by using hybrid Taguchi-genetic algorithm. IEEE
Trans Neural Netw 2006;17:69e80.
[22] Cios KJ, Moore GW. Uniqueness of medical data mining. Artif Intell
Med 2002;26:1e24.
[23] Weber GM, Mandl KD, Kohane IS. Finding the missing link for big
biomedical data. JAMA 2014;311:2479e80.
[24] Jain AK, Chandrasekaran B. 39 Dimensionality and sample size consid-
erations in pattern recognition practice. Handbook Stat 1982;2:835e55.