Science of the Total Environment 592 (2017) 334–344

Contents lists available at ScienceDirect

Science of the Total Environment

journal homepage: www.elsevier.com/locate/scitotenv

Expert assessment of the resilience of drinking water and sanitation

systems to climate-related hazards
Jeanne Luh ⁎, Sarah Royster, Daniel Sebastian, Edema Ojomo, Jamie Bartram
The Water Institute, Department of Environmental Sciences and Engineering, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, 135 Dauer Drive,
Chapel Hill, NC 27599, USA

H I G H L I G H T S G R A P H I C A L A B S T R A C T

• Assessed the resilience of water and

sanitation technologies to climate ex-
tremes.
• Resilience scores ranged from 1.7 to 9.9
out of a maximum resilience of 10.
• Technologies demonstrated a large
range in resilience for drought.
• Technologies demonstrated a small
range in resilience for superstorm
flooding.
• Results can be used for future adaptation
planning and vulnerability assessments.

a r t i c l e i n f o a b s t r a c t

Article history: We conducted an expert assessment to obtain expert opinions on the relative global resilience of ten drinking
Received 17 January 2017 water and five sanitation technologies to the following six climate-related hazards: drought, decreased inter-
Received in revised form 8 March 2017 annual precipitation, flood, superstorm flood, wind damage, and saline intrusion. Resilience scores ranged
Accepted 8 March 2017
from 1.7 to 9.9 out of a maximum resilience of 10, with high scores corresponding to high resilience. We find
Available online 29 March 2017
that for some climate-related hazards, such as drought, technologies demonstrated a large range in resilience, in-
Editor: D. Barcelo dicating that the choice of water and sanitation technologies is important for areas prone to drought. On the other
hand, the range of resilience scores for superstorm flooding was much smaller, particularly for sanitation technol-
Keywords: ogies, suggesting that the choice of technology is less of a determinant of functionality for superstorm flooding as
Resilience compared to other climate-related hazards. For drinking water technologies, only treated piped utility-managed
Drinking water systems that use surface water had resilience scores N6.0 for all hazards, while protected dug wells were found to
Sanitation be one of the least resilient technologies, consistently scoring b5.0 for all hazards except wind damage. In general,
Weather sanitation technologies were found to have low to medium resilience, suggesting that sanitation systems need to
Climate
be adapted to ensure functionality during and after climate-related hazards. The results of the study can be used
Hazards
to help communities decide which technologies are best suited for the climate-related challenges they face and
help in future adaptation planning.
© 2017 Elsevier B.V. All rights reserved.

1. Introduction

⁎ Corresponding author. Climate change is shifting global weather patterns in a way that is
E-mail address: jluh@email.unc.edu (J. Luh). predicted to impact both natural and anthropogenic systems such as

http://dx.doi.org/10.1016/j.scitotenv.2017.03.084
0048-9697/© 2017 Elsevier B.V. All rights reserved.
 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344
freshwater resources and sanitation systems, respectively. Projections
from the Intergovernmental Panel on Climate Change (IPCC) for the
late 21st century (2081–2100) show a probability of 90–100% for an in-
crease in the frequency, intensity, and/or amount of heavy precipitation
events over most of the mid-latitude land masses and wet tropical re-
gions, a 66–100% probability for increases in intensity and/or duration
of drought on a regional to global scale, a 90–100% probability of in-
creased incidence and/or magnitude of extreme high sea level, and a
N50–100% probability for an increase in intense tropical cyclone activity
in the Western North Pacific and North Atlantic (IPCC, 2013). In the near
future (2016–2035), IPCC projections show a 66–100% probability for
both an increase in frequency, intensity, and/or amount of heavy precip-
itation over many land areas and increased incidence and/or magnitude
of extreme high sea level (IPCC, 2013). The occurrence of these extreme
weather and climate events lead to an increase in fluvial erosion, salini-
zation of coastal aquifers, reduction in water availability, and wind dam-
age to structures in areas not accustomed to such events (IPCC, 2013,
2014a, 2008). In the case of water and sanitation systems, flood waters
can cause physical damage to water and sanitation infrastructure; fluvi-
al erosion from flooding can contaminate water supplies through the in-
troduction of debris and pollutants (Islam et al., 2007; Kistemann et al.,
2002); sea level rise and the resulting salinization of surface waters and
coastal aquifers can lead to a decrease in water quality (Hay and
Mimura, 2005); and decreased water availability from drought directly
impacts the availability of water resources, the quality (Khan et al.,
2015) of water (through increased pollutant concentration and saliniza-
tion), and leads to wastewater with higher concentrations of pollutants
that must be dealt with (IPCC, 2014a). In addition, the expected impacts
of climate change may interact with each other, for example, coastal
areas may experience both increased freshwater flooding and saline in-
trusion caused by sea-level rise, and tropical small island states may ex-
perience both tropical cyclone activity and sea level rise (IPCC, 2012).
The effects of these climate-related events can leave water and sanita-
tion systems non-functioning, exposing the population to various
health risks (e.g., waterborne illnesses due to lack of safe water (IPCC,
2014a)). These risks impact both rural and urban populations in high in-
come countries, and low and middle income countries.
To plan for and reduce the impacts of climate-related events to com-
munities, studies have been conducted to assess the vulnerability and
potential adaptability of water and sanitation systems (Sherpa et al.,
2014; Heath et al., 2012; Charles et al., 2010). For example, Sherpa
et al. (2014) examined the vulnerability of eight sanitation systems for
floods and provided guidance on systems selections, while Heath et al.
(2012) developed a vulnerability method for water and sanitation ser-
vice providers in peri-urban and informal settlements in low-income
settings. Vulnerability assessments can also be carried out to identify
the regions or populations of highest vulnerability to loss of access to
drinking water or sanitation (Luh et al., 2015; Elliott et al., 2014;
Ojomo et al., Submitted; Banerjee, 2012). As part of these assessments,
the types of drinking water and sanitation technologies used in a region
or community as well as their resilience to different climate-related
events must be known, where resilience is related to the water or sani-
tation technology's ability to absorb disturbances from climate-related
events while maintaining its same basic structure and ability to function
(Charles et al., 2010). Studies determining the resilience of drinking
water and sanitation technologies to climate-related events have re-
ported qualitative resilience ratings of ‘high’, ‘medium’, and ‘low’ for dif-
ferent water and sanitation technologies (Charles et al., 2010; Rajib
et al., 2012; Howard et al., 2010; Calow et al., 2011). As an example,
Rajib et al. (2012) evaluated the technological resilience of small drink-
ing water systems (e.g., pond sand filters, dug wells, deep and shallow
tubewells) in coastal areas of Bangladesh using the high, medium, low
classification scheme of Howard et al. (2010). These qualitative assess-
ments (Rajib et al., 2012) of resilience were based on long-term field
surveys using both user and expert opinions under projected climate
change and provide valuable insight on the potential technological
335
resilience. However, when vulnerability assessments are conducted
for the purpose of comparing, ranking, and/or identifying regions of
highest vulnerability – for example, comparison of coastal counties in
North Carolina, United States – numerical scores or indices for vulnera-
bility are often needed. As such, there is a need to have numerical values
of resilience that can be used in the calculations of vulnerability scores
or indices. In the absence of numerical resilience scores, Banerjee
(2012) and Luh et al. (2015) assigned values of 0.1, 0.4, 0.7, and 1 to cor-
respond to the qualitative assessments of high, medium, low, and no re-
silience, respectively, from Howard et al. (2010); however, these scores
assume that the difference between low and medium resilience is the
same as the difference between medium and high resilience, which
may not be true. In addition, it is unclear whether ‘low’ would have
the same meaning across all technologies or whether ‘low’ could trans-
late as 0.7 for one technology and 0.9 for a different technology. There is
therefore a need to quantify how climate-related hazards can impact
the resiliency of water and sanitation technologies. These scores of resil-
ience can then be used to more accurately assess vulnerability.
Accordingly, this study aims to provide numerical scores of resil-
ience for different types of improved water and sanitation systems to
climate-related hazards. We conducted an expert assessment, which is
a form of qualitative assessment, to obtain opinions on the resilience
of ten drinking water and five sanitation technologies to the following
six climate-related hazards: drought, decreased inter-annual precipita-
tion, flood, superstorm flood, wind damage, and saline intrusion. While
existing studies (Charles et al., 2010; Howard et al., 2010) group
precipitation-related events as ‘increase in precipitation’ or ‘decrease
in precipitation’, we differentiate between flood and superstorm flood
(as well as drought and decreased inter-annual precipitation) because
the impacts of the two events are different and thus the resiliency of a
technology to these events may also be different. The results of the ex-
pert assessment were used to obtain a single resilience score for each
pairing of climate-related hazard and water/sanitation technology. In
addition to using these resilience scores in vulnerability assessments,
determining the resiliency of different water/sanitation systems may
help communities decide which technologies are best suited for the
climate-related challenges they face and will help in future adaptation
planning.
2. Methods
2.1. Expert assessment approach
Expert elicitation and expert structured judgment are two common
systematic processes used to obtain and quantify expert judgment
about uncertain quantities when conventional scientific research is
not feasible (USEPA, 2011). While similar in purpose, the USEPA has de-
fined expert elicitation as a method focused solely on characterizing the
state of knowledge; and expert structured judgment as a method that
focuses on characterizing both the state of knowledge as well as social
values and preferences (USEPA, 2011). Both methods are often used in
risk assessment (e.g., likelihood of volcanic eruption (Klugel, 2011),
consequences of nuclear accidents (Cooke and Goossens, 2000a), in-
crease in human mortality due to air pollution (Tuomisto et al., 2005;
Roman et al., 2008)), where probabilistic distributions are typically ob-
tained, although non-probabilistic elicitations and judgments have also
been conducted (e.g., health impacts (Forsberg et al., 2012) and (re-
)emergence of infectious diseases (Cox et al., 2012) associated with cli-
mate change). For the purposes of our study – determining the resil-
ience of drinking water and sanitation technologies to climate-related
hazards relative to other technologies (i.e., the study objective is not
to determine probability of failure) – we used a modified form of
Cooke's method (also known as the classical model) for expert struc-
tured judgment to obtain relative resilience scores. Cooke's method
for expert structured judgment consists of 15 steps divided into the fol-
lowing three categories: preparation for elicitation, elicitation, and post-
336 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344
elicitation (Cooke and Goossens, 2000b; Goossens and Cooke, 2005).
We applied Cooke's method for the steps during the preparation and
elicitation processes to ensure a systematic approach for the identifica-
tion and selection of variables and experts. We then modified the ap-
proach used to aggregate expert responses during post-elicitation
processing to reflect the objective of obtaining relative resilience.
2.2. Preparation for elicitation
During the preparation process, we performed a literature review to
identify an initial panel of experts in the fields of (i) water, sanitation,
and hygiene; (ii) water engineering; and (iii) disaster management
based on their experience in the related fields. Experts identified in
this initial round were then asked to nominate peers whom they con-
sidered to be experts in the three above-mentioned fields and in the
context of our study on climate-related hazards and their impacts on
water and sanitation technologies. Successive rounds of peer nomina-
tion continued until no new names were nominated or no further re-
sponses were provided. From the pool of nominations, nominees were
selected for the interview based on the number of nominations, a re-
view of their publication list, and work experience.
The water and sanitation technology and climate-related hazard def-
initions (see section below), survey questions, seed questions to assess
expert performance, and pre-interview guide to prepare the expert for
the session (see Supplementary material) were revised through nine it-
erations and included feedback from five pilot runs. For the survey ques-
tions, we asked experts to provide relative rankings for the resilience of
different water and sanitation technologies to climate-related events,
thus allowing rankings to be comparative across technologies for the
same event. In response to feedback during the pilot sessions, we
allowed technologies to be ranked equally.
2.3. Elicitation
From the pre-elicitation process, we received 57 nominees. During
November 2015 and July 2016, we contacted the top 23 experts
(based on the abovementioned criteria) to participate in the study, of
which nine agreed to participate. According to a panel of expert elicita-
tion practitioners, a minimum of six experts is needed, with little addi-
tional benefits observed beyond 12 experts (Knol et al., 2010). Prior to
the interview, experts were provided with a pre-interview guide to pre-
pare, where the purpose, format of the survey, and definitions of tech-
nologies and climate-related hazards are described (no survey
questions or seed questions). The interview consisted of an online
Qualtrics survey with the survey administrator standing by on the
phone or Skype (without video) to clarify definitions and answer ques-
tions, while recording feedback, discussions, or any qualifying state-
ments that the expert offered during the interview. Any reasoning,
thought processes, or discussions were recorded via voice recording
and/or by written notes that corresponded to the question or group of
questions that initiated the discussion. All audio files were transcribed.
The study was exempt from further review from the Institutional Re-
view Board of the University of North Carolina at Chapel Hill (record
number 15–1874).
The Qualtrics online survey was divided into two sections, the first
on drinking water technologies and the second on sanitation technolo-
gies. Each section began with seed questions which were used to assess
the level of knowledge of the expert in the field of water and sanitation.
Seed questions are questions to which a correct answer exists, but may
not be widely known, and should require experts to estimate a value
based on the background knowledge they have. Following the seed
questions, experts were asked to rank, using scores from 0 (least resil-
ient) to 10 (most resilient), the resilience of 10 drinking water technol-
ogies (and five sanitation technologies) relative to one another for each
of the six climate hazards. When ranking the resilience, experts were re-
quested to consider these systems in their most basic state, where no
adaptive measures had been taken, and consider both the direct and in-
direct damages to water and sanitation systems.
2.4. Post-elicitation processing
Aggregation of expert responses was performed using performance-
based weighting and equal weighting, where the former was deter-
mined based on how accurate the expert's response was to the actual
answers of seed questions. The assumption is that expert performance
in the seed questions reflects their ability to make judgments in the sur-
vey questions (Cooke, 1991). To calculate performance-based
weightings, we did not follow Cooke's method which is used when
the seed questions ask for probability distributions (not relative rank-
ings). Instead, performance-based weightings were calculated using
an inverse-based approach as follows:
1
m
∑ j¼1 εi; j
wi ¼ !  100% ; ð1Þ
n 1
∑i¼1 m
∑ j¼1 εi; j
where wi is the weighting of expert i for a specific seed question, n is the
total number of experts, m is the total number of sub-questions in the
seed question, and εi,j is the absolute difference between the known an-
swer and expert i's response for sub-question j of the seed question.
Fig. 1 illustrates how Eq. (1) is used to calculate the performance-
based weight for one seed question with five components (i.e., the
seed question has five sub-questions). An inverse-based approach was
used in Eq. (1) so that higher weightings are associated with lower dif-
ferences between the known answer and expert response, and thus, de-
note higher ability of experts to make judgments that match empirical
data. The overall weighting for an expert was then calculated as the un-
weighted average from all seed questions. We note that Wang and Bier
(2013) showed how ordinal rankings (e.g., where experts are asked to
rank a list of items similar to this study) can be used to derive probabil-
ity distributions (using either probabilistic inversion or Bayesian density
estimation); this could then be used to calculate performance-based
weightings following Cooke's method. However, both probabilistic in-
version and Bayesian density estimation require that no ties be allowed
(i.e., technologies could not be ranked equally), which was not the case
for our study. For comparison purposes, in addition to performance-
based weightings, equal weightings were also used to aggregate expert
responses.
2.5. Definitions used in this study
Resilience relates to adaptive capacity (IPCC, 2014b), and so defini-
tions for both terms were provided to the experts. The definitions for re-
silience and adaptive capacity were modified based on definitions from
the IPCC (2014b) and Vision 2030 report (Charles et al., 2010). For this
study, resilience was defined as the ability of a water or sanitation sys-
tem (both hardware and software) to continue to function or rapidly re-
turn to function during and after a climate-related disturbance. This
includes all aspects of functionality without a compromise of quality,
quantity of water provided, waste containment, continuity, or reliabili-
ty. Resilience describes qualities that a system already has, and differs
from adaptive capacity which describes how much a system can be
adapted for future disturbances (Fig. 2). Adaptive capacity was defined
as the [unfulfilled potential] ability of a system to adjust to climate
change (including climate variability and extremes) to moderate poten-
tial damages, to take advantage of opportunities (associated with
changes in climate), or to cope with the consequences. Adaptive capac-
ity describes what can be (but has not been) done to increase resilience.
If adaptive measures are taken, resilience will increase while adaptive
capacity will decrease. Adaptive capacity and resilience, for the
 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344
Fig. 1. Example calculation of performance-based weightings using Eq. (1).
purposes of this study, are in the context of the occurrence of a specific
event type.
The water and sanitation technologies, as well as the six climate-
related hazards studied are listed in Table 1, with definitions modified
or adapted from IPCC (2013, 2012, 2014c), the Vision 2030 report
(Charles et al., 2010), World Health Organization (WHO) (2011), and
the WHO/UNICEF Joint Monitoring Programme (JMP) for Water
Supply and Sanitation (2016). Our list of technologies and hazards is
not comprehensive and was limited in number in order to keep the sur-
vey duration under 1 h, as suggested during the pilot tests. Specifically,
we did not include seasonal variability of precipitation, which is differ-
ent from drought and decreased inter-annual precipitation and is a crit-
ical issue in certain regions such as south Asia, where the total amount
of precipitation may be the same in consecutive years but the distribu-
tion of precipitation may differ significantly between seasons or even
months (Rajib et al., 2011). We note that for this study, decreased
inter-annual precipitation refers to a decline in mean annual precipita-
tion, rather than a decline in precipitation variation in proximate
years. In addition, our flood hazard did not differentiate between differ-
ent types of flooding, for example flash flooding versus storm surge
flooding, in order to keep the survey to a strict length. Instead,
superstorm flooding was included to represent the increase in flood in-
tensity and/or severity resulting from the combination of two types of
flooding events. As flash floods are characterized by intense rainfall in
a short period of time, the resilience scores for superstorm flooding
could potentially be used to represent flash floods as well, due to the
similarity in their destructive impacts.
Fig. 2. Relationship between resilience and adaptive capacity.
337
Similarly, our final list of technologies included ten water and five
sanitation technologies and is not a comprehensive list of all possible
technologies. For utility managed and community managed piped
drinking water systems using surface water and for piped sewerage
systems, experts were asked to rank treated and untreated systems
separately. For piped drinking water systems using groundwater, ex-
perts were only asked to rank untreated utility and community man-
aged systems. Treated utility and community managed systems using
groundwater were considered to be the same as untreated utility and
community managed systems using groundwater and were thus not in-
cluded in this study, as the extent of typical groundwater treatment
(disinfection only) is inconsistent with our definition of a treated sys-
tem, which includes a series of processes such as coagulation, floccula-
tion, sedimentation, and filtration. For treated utility and community-
managed systems using groundwater, even if the disinfection process
stops working, there is a possibility that the systems remain functioning
if the quality of groundwater is high. As such, we defined a treated
system as one comprised of several processes in place and that if
one process in the series stopped working (e.g., coagulation unit),
the system was then considered non-functional. Similarly, boreholes/
tubewells were defined in Table 1 to have a depth ranging from 3 to
300 m and we did not differentiate between shallow and deep bore-
holes. We did not consider unimproved technologies (e.g., surface
water, unprotected dug well, pit latrine without slab, etc.) in this
study.
2.6. Geographic scope of study
Experts were asked to score the resilience of water and sanitation
technologies to different climate-related hazards on a global scale. Re-
gional assessments were initially considered, and were discussed with
the pilot participants; however, we chose not to pursue this option be-
cause breaking the study into different regions would have lengthened
the survey to a duration that would have been impractical. In addition,
this would have also limited the number of respondents that would
be qualified to take each section. Instead, the survey was framed as
though each technology was installed in the same place, and that
place was hit by the same climate-related hazard, removing any ques-
tion of the likelihood of an event happening or a technology being in a
specific geographic location.
338 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344

Table 1
Definitions of climate-related hazards and water and sanitation technologies provided in the pre-interview guide and used in the survey study.

Hazard or Definition Modified from

technology

Climate-related hazards
Drought A period of abnormally dry weather that is an aberration from an established pattern of precipitation. A IPCC (2014c)
drought represents a relatively sharp decline in precipitation that is serious enough to force changes in
management and usage of water
Decreased inter-annual A long-term pattern of steadily decreasing mean annual precipitation that could, for example, lead to Based on precipitation trends
precipitation reduced recharge of water supplies over an extended period. Decreased inter-annual precipitation does not described in IPCC (2013)
require immediate changes in the management and usage of water for the resource to remain sustainable
Flood The overflowing of the normal confines of a stream or other body of water, or the accumulation of water over IPCC (2014c)
areas not normally submerged, including excessive hydrologic runoff and urban flooding. The effects of
submergence, contamination from increased runoff and/or saline water and the force of the moving water
should all be considered. Please also consider indirect damages from flooding, such as impacts to
transportation and energy that may impede regular operation and maintenance of systems
Superstorm flood The combination of flooding from heavy inland precipitation and coastal inundation. Superstorm flooding is Based on compound events
generally quick and powerful, since it is a combination of two types of flooding events. Please consider all described in IPCC (2012)
direct and indirect damages, including wind damage associated with this type of storm, saline intrusion from
coastal inundation, and the impacts to the transportation, communication, and energy sectors that may
impede regular operation and maintenance of systems
Wind damage Damage to infrastructure caused by extreme/high wind speeds due to cyclonic activity (eg. wind damage Based on impacts described in
from hurricanes, typhoons, and other wind activity). This category includes direct and indirect damage, for IPCC (2012)
example direct wind damage to water and sanitation infrastructure as well as wind damage to energy,
communications and transportation infrastructure that may impact operation and maintenance
Saline intrusion The displacement of fresh surface water or groundwater by the advancement of salt water due to sea level IPCC (2014c)
rise. For our purposes, saline intrusion due to reduced recharge rates, increased withdrawal rates or coastal
inundation is not included in this category

Drinking water technologies

Treateda Source waters in treated systems undergo (a series of) processes such as coagulation, flocculation,
sedimentation, filtration, and disinfection to remove contaminants such as microbial pathogens, particulate
matter, and chemical contaminants. When the system is working properly, the water delivered to the
consumer is potable and does not require further treatment
Untreateda For untreated systems, water does not go through any treatment process before it is delivered to the
consumer. Such water may have to be treated by the consumer to be potable
Utility managed piped water Deliver water to consumers through a network of underground, pressurized pipes. The source of this water Vision 2030 (Charles et al.,
a
systems (utilities) can be surface water or groundwater. Such entities typically have access to a sufficient body of staff, with a 2010), WHO (2011)
wide skill mix, who are in general properly trained to undertake the tasks for which they are responsible.
Small municipal systems that share these characteristics are included in this category
Community managed piped Deliver water to consumers through a network of underground, pressurized pipes. The source of this water WHO (2011)
water systemsa can be surface water or groundwater. Communities managed systems differ from utility managed systems in
that in general, they have less access to human, technical, and financial capital. Small municipal systems that
share these characteristics are included in this category
Boreholes/tubewells Have a small diameter (10–20 cm) with a varying depth (3–300 m). These wells are drilled using manual JMP (2016)
power or machinery. Boreholes/tubewells are constructed with casing, or pipes, which prevent the small
diameter hole from caving in and protects the water source from infiltration by run-off water. Water is
delivered from a tubewell or borehole through a pump, which may be powered by human, animal, wind,
electric, diesel or solar means
Protected dug wells Are shallow (3–15 m), of a large diameter (1–2 m) and are dug manually. The well is lined with stones, brick, Howard et al. (2010),
tile or other material to prevent collapse and covered with a cover of wood, stone or concrete to protect from JMP (2016)
direct contamination. Water can be withdrawn from a Dug Well using a container lowered into the well or
a pump, which may be powered by human, animal, wind, electric, diesel or solar means
Protected springs Are springs in which the eye of the spring (where the water emerges from the ground) has a protective wall or JMP (2016)
box around it. This box is called a “spring box” and it protects the spring from runoff and other sources of
contamination. The “spring box” is typically constructed of brick, masonry, or concrete and is built around the
spring so that water flows directly out of the box into a pipe or cistern, without being exposed to outside pollution
Rainwater Refers to rain that is collected or harvested from surfaces (by roof or ground catchment) and stored in a JMP (2016)
container, tank or cistern until used

Sanitation technologies
Treateda Excreta is treated at a facility so that the wastewater can return to the environment safely
Untreateda Raw excreta is output directly into the environment
Piped sewerage systemsa Collect human excreta and wastewater through underground, unpressurized pipes and bring it to facilities Vision 2030 (Charles et al.,
for treatment and disposal. Sewerage systems consist of facilities for collection, pumping, and disposing of 2010), JMP (2016)
human excreta and wastewater. This category includes conventional sewers, combined sewers (which carry
storm water in addition to wastewater), small bore sewers (which use variable pressure) and condominial
systems (which collect effluent from septic tanks). Please consider both sewerage systems with and without
a treatment facility
Septic systems Are excreta collection devices consisting of a water-tight settling tank, which is normally located JMP (2016)
underground, away from the house or toilet. The effluent of a septic tank usually seeps into the ground
through a leaching field or pit. Treatment of the excreta and wastewater occurs in the septic tank and
percolation through the soil of the leaching pit
Pit latrines Are a covered hole with or without a permeable casing where human excreta is collected and adequately Vision 2030 (Charles et al.,
separated from human contact. This category includes flush latrines and ventilated improved pit latrines 2010), JMP (2016)
Composting toilets Are a dry toilet into which carbon-rich material (vegetable wastes, straw, grass, sawdust, ash) are added to the JMP (2016)
excreta in an above ground container and special conditions are maintained to produce inoffensive compost
a
For utility managed and community managed piped drinking water systems, and for piped sewerage systems, experts were asked to rank treated and untreated water/sanitation
systems separately.
 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344 339

3. Results and discussion
3.1. Analysis of seed questions and performance-based weights
Table 2 lists the performance-based weightings for each expert and
compares these to equal weighting. For equal weighting, shown as
Weight A in Table 2, a value of 0.11 is assigned to all nine experts
(i.e., weighting = 1/9). Using Eq. (1), performance-based Weight B is
calculated using all five seed questions and performance-based Weight
C is calculated using only three of the seed questions (excluding seed
questions 1 and 5). We tested to see whether differences existed be-
tween Weight B and Weight C because analysis of the expert responses
indicated that the wording of seed questions 1 and 5 may not have been
clear, which would have affected the expert responses and therefore the
measure of error (εi,j). Specifically, in seed question 1, experts were
asked to estimate the percent use for different primary drinking water
sources and some experts indicated that this question could be
interpreted in different ways. For example, one expert indicated that
water from boreholes can then be subsequently piped or bottled – is
the water source considered to be borehole or piped system/bottled
water? – and that these nuances were not captured in the seed ques-
tions. Additionally, analysis of the survey responses indicated that
some experts misinterpreted the question and provided estimates of
percent use for all drinking water sources, not just primary sources.
For seed question 5, experts were asked to rank eight countries based
on total renewable water resources on a scale of 1 (most scarce) to 8
(most abundant), and the analysis of the survey responses showed
that it was unclear to the experts that equal ranking (ties) was not per-
mitted. One expert also indicated that the total renewable water re-
sources was highly dependent on population and thus a larger
population would have less water available, while another expert ar-
gued that it was difficult to generalize water resources by country as
there is a lot of variation within a single country (e.g., California in com-
parison to Indiana in the United States).
From Table 2, we see that depending on the expert, the difference
between Weights B and C can range from 0% to 33%, although the major-
ity had a difference b 25%. When experts were ranked based on their
performance-based weight, their relative rank showed that the change
in rank between Weights B and C was no more than two positions,
with the top three experts remaining the same (although the order
shifted between the first and second ranked expert). Due to the differ-
ences observed and the concerns raised by the experts for seed ques-
tions 1 and 5, for the remainder of the manuscript, we use only three
seed questions to calculate the performance-based weighted average
scores for resilience of water and sanitation technologies to different
climate-related hazards.
As the selection of appropriate seed questions is known to be chal-
lenging, and performance-based weightings are affected by the choice
of seed questions, some studies (Cooke and Goossens, 2008; Soares
et al., 2011; Fischer et al., 2013) question the use of performance-
based weightings to aggregate expert responses. We calculated average
resilience scores using equal weighting (Weight A) (see Supplementary
material) and show that despite differences in Weights A and C, the re-
silience scores calculated from Weights A and C were generally within
one point of each other. The similar scores between aggregated re-
sponses calculated using performance-based and equal weightings is
consistent with other studies (Clemen and Winkler, 1999; Clemen,
2008; Grigore et al., 2016) that found a similar lack of difference be-
tween the two methods, suggesting that further research is needed to
determine when seed questions and performance-based weightings
should be used. For the remainder of the paper, we present the aggre-
gated resilience scores calculated using performance-based weightings
with equal weighting scores provided in the Supplementary material.
3.2. Resilience scores for drinking water technologies
In this section and the following section on sanitation technologies,
we present the resilience scores by climate-related hazard, rather than
by technology, in order to be consistent with the framing of the survey
questions. Similar hazards, such as drought and decreased inter-annual
precipitation, were grouped together under one sub-section for com-
parison purposes, and the order with which technologies are discussed
follows the order listed in Table 1. Fig. 3 presents the performance-
weighted average scores (resilience scores corresponding to Fig. 3 are
provided in the Supplementary material), where high scores corre-
spond to high resilience. Anonymized individual expert scores for all
nine experts can be found in the Supplementary material.
3.2.1. Drought and decreased inter-annual precipitation
Experts assessed the resilience of drinking water technologies to
drought to be similar to that of decreased inter-annual precipitation,
where in general, scores for each of the water technologies were within
one point of each other for the two hazards. This suggests that a

Table 2
Comparison of equal weights (Weight A) and performance-based weights (Weights B and
C) calculated using Eq. (1). Relative expert rank based on performance-based weights are
also provided.

Expert Weight A Weight Ba Weight Cb

(equal weighting) (Eq. (1), 5 seed (Eq. (1), 3 seed
questions) questions)

Score Rank Score Rank

1 0.11 0.08 7 0.08 5

2 0.11 0.12 3 0.15 3
3 0.11 0.08 8 0.07 7
4 0.11 0.15 2 0.21 1
5 0.11 0.12 4 0.08 6
6 0.11 0.07 9 0.05 9
7 0.11 0.19 1 0.20 2
8 0.11 0.09 6 0.07 8
9 0.11 0.10 5 0.09 4
a
Weight B is calculated using all five seed questions.
b
Weight C is calculated using only three seed questions (seed questions 1 and 5 are Fig. 3. Resilience scores for drinking water technologies calculated using performance-
excluded due to potentially unclear wording in these seed questions). based Weight C.
340 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344
drought-resilient water technology would also be resilient to decreased
inter-annual precipitation. On average, piped drinking water systems
had the highest resilience to drought, with scores ranging from 6.3–
7.9 depending on the water source (surface versus groundwater), man-
agement type (community versus utility), and presence of treatment.
The exception was community-managed piped systems using surface
waters, which were assessed to have a lower resilience of ~4.4 regard-
less of treatment, likely because utility-managed water systems gener-
ally have greater access to human resources, greater potential to
develop additional sources of water and water storage, and more fi-
nances for routine maintenance and upgrades to the system in compar-
ison to community-managed systems. Experts noted that their
understanding is that community-managed piped systems operated
more independently and without much added support from the gov-
ernment. In addition, one expert stated that well managed utilities like-
ly either have or are in the process of developing climate resilient plans
that could increase their resilience to drought, while community-
managed systems likely do not have the resources to develop such
plans. Interestingly, community-managed piped systems using untreat-
ed groundwater did not have as low a score as community-managed
piped systems using untreated surface water, and were actually found
to have similar resilience scores as utility-managed piped systems
using groundwater and surface water. The difference in scores between
the two community-managed systems (untreated groundwater vs un-
treated surface water) may due to the fact that groundwater may not
be as impacted by drought as surface waters, and although experts did
not explicitly comment on this, the scores suggest that according to ex-
perts, source type is more important than management type in
assessing resilience.
Of the three non-piped water technologies that use groundwater,
boreholes/tubewells, which for this study we define as wells installed
with a pump (i.e., not connected to a piped system), were found to be
more resilient than protected dug wells and protected springs with a re-
silience score of 6.1 and 5.5 for drought and decreased inter-annual pre-
cipitation, respectively, although one expert pointed out that the
resilience would depend on the aquifer and depth of the borehole.
Even if a drought is so severe that it has affected the water table, only
those boreholes that are at a depth above the water table will be impact-
ed, while those at a depth below the water table will remain fully func-
tional. Deeper boreholes would typically require multi-year droughts
before substantive effects are observed. In addition, experts noted that
in practice, boreholes may not be installed correctly, with one expert
noting that private boreholes are often installed without regard to dis-
tance between each borehole, which can have a drawdown effect on ad-
jacent systems. In our study, boreholes/tubewells were defined to have
a depth ranging from 3 to 300 m and despite this range in depth, no ex-
pert asked us to differentiate between shallow and deep boreholes.
Deep boreholes, which are common in Bangladesh due to arsenic con-
tamination of shallow aquifers (Burgess et al., 2010), are less likely to
be affected by drought and decreased inter-annual precipitation (as
well as saline intrusion) due to the slow recharge rate of the deep aqui-
fer (U.S. Geological Survey (USGS), 2016). Based on the individual ex-
pert scores, particularly for saline intrusion, we see that most experts
assessed the boreholes as shallow boreholes.
Protected dug wells were found to have a fairly low resilience (~3.5)
to drought and decreased inter-annual precipitation due to its shallow
depth (3–15 m). Similar to dug wells, low resilience scores (3.9 and
2.8 for drought and decreased inter-annual precipitation, respectively)
were also obtained for protected springs, likely because both dug
wells and protected springs arise from shallow groundwater and are
therefore affected by changes in precipitation relatively rapidly.
As rainwater harvesting is directly affected by lack of rainfall, experts
rated its resilience to drought and decreased inter-annual precipitation
at 1.7 and 2.0, respectively. However, one expert did specify that the re-
silience of rainwater harvesting is dependent on the amount of storage
capacity available and while “…drought equals no water from the
sky…[the] ability to survive a drought using [stored] rain water is very
real”. We note that while seasonal variability of precipitation was not
considered in this study, its effect can clearly be seen with rainwater
harvesting systems. During the wet months and when there is an in-
crease in precipitation, these systems function well and deliver water
as needed; however, during the dry season, once the storage capacity
is depleted, rainwater harvesting systems no longer function when
there is no precipitation.
3.2.2. Floods and superstorm floods
For floods and superstorm floods, all drinking water technologies
had higher resilience scores to floods than to superstorm floods, with
the exception of piped surface water systems (all combinations of utility
and community managed, treated and untreated) which had the same
resilience for both hazards. Experts commented on the increased sever-
ity of superstorm floods as compared to floods and that a return to nor-
mal operations after superstorm flooding would also be slower and
more difficult than after flooding. Superstorm flooding may also have
a potentially longer-term impact with respect to salinization of ground-
water and would depend on the proximity of drinking water technolo-
gies to the storm or coast.
A range of resilience scores (3.6–8.4) for flooding and superstorm
flooding was obtained for piped drinking water systems, with the low-
est resilience scores corresponding to community-managed systems
using surface waters. This is consistent with the results obtained for
drought and decreased inter-annual precipitation and reflect the higher
resilience of groundwater systems over surface waters systems as well
as the impact that decreased access to technical, financial, and trained
human resources for community-managed systems has on resilience
as compared to utility-managed systems.
Experts scored boreholes/tubewells to be one of the more resilient
technologies to flooding and superstorm flooding at 8.2 and 6.3, respec-
tively. While boreholes/tubewells are often submerged during a flood
event, experts stated that boreholes/tubewells should be resilient to
heavy flooding and able to withstand the intrusion of floodwaters with-
out damage to the borehole when there is proper installation with pro-
tection at the well head. In Table 1, we define boreholes/tubewells to be
“…constructed [in such a way to protect] the water source from infiltra-
tion by run-off water” and thus experts scored the resilience of bore-
holes/tubewells assuming proper construction; however, we note that
many times this is not the case and that contaminant intrusion occurs
when there is improper construction. As boreholes are a low cost option,
one expert indicated that boreholes are advocated by several agencies
but that there are not necessarily any real standards on proper borehole
construction and no ability to enforce standards when they do exist,
which make these borehole investments quite vulnerable.
Protected dug wells were found to have a resilience of 4.8 to floods
and had the lowest resilience (2.7) to superstorm flooding. The low re-
silience of protected dug wells may be due to lack of proper construc-
tion, with one expert stating that it is unlikely that protected dug
wells are very well protected, thus allowing floodwaters to enter. Simi-
larly, another expert noted that the resilience of protected springs to
flooding and superstorm flooding events also depends on how well
the spring is protected.
Rainwater harvesting was assessed to have a resilience of 7.4 and 6.5
to flooding and superstorm flooding, respectively, because in order for
the system to stop functioning, physical damage to the storage tanks
or catchment systems, particularly ground catchments that are not
high enough to avoid floodwaters, need to occur. Interestingly, rainwa-
ter harvesting was found to have a lower resilience to floods as com-
pared to boreholes/tubewells. Upon closer inspection of the individual
scores from all experts (see Supplementary material), we see that six
out of nine experts actually scored rainwater harvesting to have higher
or equal resilience to floods than boreholes/tubewells. However, when
performance weights were taken into account, the overall aggregate
score resulted in a higher resilience value for boreholes/tubewells. The
 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344 341

difference in scores between different experts may be due to their inter- saline intrusion in boreholes close to the sea. Interestingly, the resilience
pretation of rainwater harvesting systems. In Table 1, both roof and of protected springs to saline intrusion was assessed to be 7.5, with six
ground catchment systems were included in the definition of rainwater of nine experts giving scores of 7 or higher. Although experts did not ex-
harvesting and no mention of construction materials was mentioned. plicitly comment on their reasoning for the high resilience scores for
Experts that scored rainwater systems to have a higher resilience protected springs, based on geology, springs are in effect ‘high’ and
score may have considered only roof catchment systems built with rein- therefore less likely to suffer from saline intrusion. Specifically, for grav-
forced concrete and thus are more robust to flooding, while experts that ity springs (experts were not asked to consider artesian springs), these
scored rainwater systems to have a low resilience score may have only water sources are considered ‘high’ because water is pulled down
considered ground catchment systems constructed without reinforced through the ground due to gravity until it reaches an impervious layer,
materials. before it flows horizontally until it reaches an opening and flows out
as a spring(Study.com, 2017).
3.2.3. Wind damage Rainwater harvesting was found to have the highest resilience of all
In addition to direct damages from wind, one key consideration technologies to saline intrusion at a score of 9.9. This was an expected
when assessing the resilience of technologies to wind damage was indi- result as rainwater comes from the atmosphere and is not subject to
rect damage, specifically damage to overhead powerlines and a sea level rise.
resulting loss of electricity. As such, wind damage is of particular con-
cern for technologies that require a centrally generated power supply,
3.2.5. Overview of all hazards
such as piped water systems. The range of resilience values for piped
The weighted-average resilience scores for drinking water technolo-
water systems to wind damage was 5.4 to 7.3, depending on the man-
gies ranged from 1.7 to 9.9 out of a maximum of 10, indicating that ex-
agement system, treatment, and water source type. Community-
perts used almost the entire scale range (0 to 10) to rank relative
managed systems were scored lower than utility-managed counter-
resilience. As shown in Fig. 3 (resilience scores corresponding to Fig. 3
parts, with one expert stating that while both community- and utility-
are provided in the Supplementary material), drought and saline intru-
managed systems are likely to use electricity, the former is less likely
sion showed the largest range in resilience between different drinking
to have access to a generator and therefore may be more impacted by
water technologies, where the difference between the highest and low-
wind damage. We note, however, that in some community-managed
est resilience scores was 6.2 and 7.4 for drought and saline intrusion, re-
systems, generators may be the primary energy source.
spectively. This shows that technologies have substantially different
Of the three non-piped technologies that use groundwater, bore-
resilience for drought and saline intrusion; thus, certain technologies
holes/tubewells scored lower (at 6.4) in resilience to wind damage
should be avoided for regions prone to these hazards. The smallest
than protected dug wells and protected springs which scored at 8.1
range (3.8) in resilience between technologies was found for
and 8.2, respectively. A closer inspection of the individual expert scores
superstorm flooding and wind damage, where five of the technologies
in the Supplementary material revealed that, in general, these three
had resilience scores within one point of each other for both hazards.
water technologies were scored very similarly; however, two experts
In general, of all the drinking water technologies, protected dug
gave low scores of 2 and 3 for boreholes/tubewells. These two experts
wells were found to be one of the least resilient technologies, consis-
both commented on the fact that electrically powered pumps for bore-
tently scoring b5.0 for all hazards except wind damage (they were
holes/tubewells would be vulnerable to wind damage and that in cer-
found to be highly resilient to wind damage with a score of 8.1). Only
tain areas prone to high winds, there would be a reluctance to
piped, treated, utility managed surface water systems had resilience
implement an electrically powered system as a lack of power would
scores N 6.0 for all hazards. Little difference in resilience was found for
cause them to be non-functional unless there is a water system that
(utility- and community-managed) piped water systems using surface
has been constructed to withstand higher and/or prolonged wind veloc-
waters across the different hazards (i.e., the range of resilience values
ity, and people with the knowledge and training to support operation,
for each technology was b 2.0 for all hazards). Interestingly, for all haz-
maintenance and repair of such systems. For example, a borehole with
ards except wind damage, experts scored the resilience of treated utility
a manual-powered handpump may be preferable to one that is electri-
and community managed surface water systems to be higher than their
cally powered in places where these conditions are not met.
untreated counterparts, despite the fact that treated systems have addi-
For wind damage, rainwater harvesting had the lowest resilience
tional unit processes (e.g., coagulation, flocculation, sedimentation) that
score at 4.6, as compared to all other technologies that ranged from
need to function properly in order for the system to deliver drinking
5.4–8.2. The lower scores for rainwater harvesting was likely associated
water. The resilience of utility-managed systems was also found to be
with roof catchment, rather than ground catchment systems, where
higher than their corresponding community-managed counterpart for
strong winds can damage the systems.
all combinations of hazard and piped systems (e.g., treated, utility-
managed surface water systems had higher resilience than treated,
3.2.4. Saline intrusion
community-managed surface water systems).
For saline intrusion, one expert noted that saline intrusion can im-
Based on the expert opinions, for regions that are prone to
pact a water source for a long time, possibly for multiple generations.
experiencing drought and decreased inter-annual precipitation,
By its very nature, water technologies that used groundwater were typ-
utility-managed piped water systems offer the greatest resilience to
ically assigned low scores for resilience. For example, for piped water
these two climate-related hazards. For areas where there are floods
systems, resilience scores ranged from 3.7–7.1 depending on the man-
and superstorm floods, experts scored technologies that draw on
agement scheme, source water, and presence of treatment. Specifically,
groundwater sources to be the most resilient. Most technologies had
piped water systems using surface waters were found to have resilience
similar resilience scores for wind damage, with protected dug wells
scores ranging from 5.4–7.5, while piped water systems using ground-
and protected springs scoring the highest, and in coastal areas subject
water were lower at 3.7–4.6. Utility-managed systems were always
to saline intrusion, rainwater harvesting is the most resilient drinking
scored higher than community-managed systems.
water technology.
Boreholes/tubewells and protected dug wells had the two lowest re-
silience scores to saline intrusion at 3.5 and 2.6, respectively, as the
water resource (groundwater) that feeds these systems can be directly 3.3. Resilience scores for sanitation technologies
impacted by saline intrusion. One expert noted that poor construction of
protected dug wells will result in saline intrusion, and another expert Similar to the previous section for drinking water technologies, the
stated that lack of proper installation and pump testing can lead to resilience scores for sanitation technologies are presented by climate-
342 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344
related hazard. Fig. 4 shows the performance-weighted average scores
with the individual expert scores found in the Supplementary material.
3.3.1. Drought and decreased inter-annual precipitation
Similar to the drinking water technologies, the largest range in resil-
ience of sanitation technologies occurred for drought, with composting
toilets having an average resilience score of 9.8 and piped untreated
sewer systems having an average resilience score of 3.6. This large
range in resilience is due to the fact that availability of water impacts
the functionality of some of the sanitation technologies. Treated and un-
treated piped sewer systems had the lowest resilience to drought as
there must be enough water to flush the waste and carry the solids
through the pipes. No difference was found between treated and un-
treated sewer systems as experts did not associate treatment with
water availability. One expert cited the example of the drought in the
early 1990s in Zimbabwe when flushing on the hour was implemented.
Residents were told that they could only flush their toilets on the hour,
otherwise there was not enough liquid to carry the solids. Septic tanks
also rely on the availability of water but were scored higher (5.2) than
piped sewer systems, with one expert stating that the slightly higher re-
silience of septic tanks was because there is less piped distance that the
waste must travel to the tank. Pit latrines and composting toilets were
found to be highly resilient to drought as they still function as intended
if water is scarce or unavailable.
A similar trend in resilience of sanitation technologies was found for
decreased inter-annual precipitation due to similar reasons of water
availability, with piped treated and untreated sewer systems having
the lowest resilience and pit latrines and composting toilets having
the highest resilience. For piped sewer systems and septic tanks, their
resilience to decreased inter-annual precipitation was higher than that
for drought. For sewer systems, this may be due to the fact that the de-
crease in precipitation patterns occurs over a long period of time, which
allow management more time to find a solution to the decrease in pre-
cipitation. As pit latrines and composting toilets do not depend on water
availability, experts assessed their resilience to decreased inter-annual
precipitation to be the same as that of drought.
3.3.2. Floods and superstorm floods
A comparison of resilience scores between flooding and superstorm
flooding showed that experts rated all sanitation technologies to have a
Fig. 4. Resilience scores for sanitation technologies calculated using performance-based
Weight C.
slightly higher resilience to flooding than superstorm flooding due to
the increased severity of the floodwaters for superstorms, although
the resilience scores for each technology were within two points of
each other for the two hazards (3.8–5.9 for floods and 2.8–4.7 for
superstorm floods). With respect to flooding, treated piped sewer sys-
tems had a slightly higher resilience than untreated piped sewer sys-
tems at 4.6 and 3.8, respectively. Septic tanks and pit latrines were
scored similarly at ~ 3.8. Experts provided the following rationale for
their assessment, stating that any sewer system regardless of treatment
would be exposed to flood waters entering the system as there are usu-
ally low points in topography that are more prone to flooding, pit la-
trines could overflow during a flood, and septic systems are equally as
exposed as pit latrines to flooding and become non-functional when
filled with water. For both floods and superstorm floods, composting
toilets had the highest resilience at 5.9 and 4.6, respectively.
3.3.3. Wind damage
The resilience of all sanitation technologies was found to range from
6.8 to 8.2 for wind damage, which is a much higher range of scores as
compared to the other climate-related hazards. Similar to the rationale
for drinking water technologies, damage to electricity was a key issue.
For treated and untreated piped sewer systems, loss of electricity can
lead to non-functioning systems if pumps and other electrical equip-
ment can no longer function. Untreated piped sewer systems were
scored slightly higher than their treated counterparts, at 7.5 and 7.1, re-
spectively, due to the additional unit processes that rely on electricity in
treated systems. Septic tanks scored the highest resilience at 8.2, likely
reflecting the fact that the settling tank is located underground and
away from winds. Pit latrines scored the lowest at 6.8, possibly due to
physical damage to the superstructure. Physical damage to infrastruc-
ture can lead to non-functionality, with one expert providing the exam-
ple of Cyclone Pam in Vanuatu in March 2015, where sanitation sub-
structures remained untouched but superstructures were damaged suf-
ficiently resulting in loss of functionality. In addition, one expert noted
that many of the sanitation technologies that are dependent on water
may also be dependent on electricity if they are being served by a
water supply that requires electricity. For example, wind damage
could potentially knock out the water supply connected to a septic tank.
3.3.4. Overview of all hazards
The weighted-average resilience scores for sanitation technologies
ranged from 2.8 to 9.8 out of a maximum of 10. As shown in Fig. 4 (re-
silience scores corresponding to Fig. 4 are provided in the Supplementa-
ry material), drought and decreased inter-annual precipitation showed
large ranges in resilience between different sanitation technologies, in-
dicating that the selection of technology is important for drought-prone
areas. Pit latrines and composting toilets are the most resilient technol-
ogies in areas with decreased water availability, as these two technolo-
gies do not require water. For floods and superstorm floods, most
sanitation technologies were scored similarly, except compositing toi-
lets, which were scored as the most resilient technology for areas that
experience floodwaters. Almost all sanitation technologies scored
above 7 against wind damage, although the preferred choice for regions
prone to high winds would be septic tanks.
3.4. Study limitations
The resilience scores and corresponding standard deviations pre-
sented in the Supplementary material show that for most combinations
of technologies and climate-related hazards, a large standard deviation
was found. The wide range in answers is not unexpected, as experts an-
swered the questions based on their different personal experiences and
backgrounds. Many experts provided commentary or justification for
how they ranked the resilience of different technologies and several
noted that quantifying resilience with this kind of nuance was very dif-
ficult. For example, one expert noted that the resilience of sanitation
 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344
technologies to superstorm flooding may be difficult to score because of
the indirect impacts on infrastructure. In another example that illus-
trates how personal experiences affected expert answers, several ex-
perts commented on the dependence of sewer systems and septic
tanks on water availability and scored these technologies as having
low resilience to drought. However, one expert ranked all sanitation
technologies to be highly resistant to drought and stated that in their
experience and what they have seen in the locations they worked in,
they have never known a sewer system to run dry or a septic system
to not have enough water to flush the toilet, and that while restrictions
on water availability may impact the timing of a toilet flush, water avail-
ability has not been severe enough to cause non-functionality of a san-
itation system.
The definitions of water and sanitation technologies set for the study
may also differ depending on the person. One expert stated that bore-
holes are quite different from tubewells in that a properly installed
borehole is quite resilient to heavy flooding while a tubewell is very
rarely properly protected for flooding. Another expert cautioned that
careful interpretation of the survey findings related to septic tanks is
needed because, in practice, what many call septic tanks are not actually
septic tanks but are soakage pits that may be partially lined on the sides
but that have no bottom and there may be no digestion happening in
the chamber and no leach field. We attempted to reduce possible misin-
terpretation of technologies by providing specific definitions to the ex-
perts prior to and during the elicitation process, however, it is possible
that experts may not have read through the definitions carefully. In ad-
dition, most of the technologies could have been further disaggregated,
for example boreholes can be assessed as deep or shallow boreholes and
piped sewer systems could have been differentiated between conven-
tional and modified sewer systems which have very different engineer-
ing and resilience, and may be an additional factor to explain the
variation in individual expert scores for piped sewer systems seen
from the raw survey data (see Supplementary material). Based on feed-
back from the pilot participants, we chose to focus only on piped water
systems for disaggregation in order to keep the survey to under 1 h.
Differences in expert interpretation and understanding of the cli-
mate hazards and water and sanitation technologies is highlighted
when one examines the raw survey scores from all experts (see Supple-
mentary material). For example, when asked to score the resilience of
rainwater harvesting to wind damage, five of the nine experts gave a
score of 1 or 2, but two experts gave a score of 8 or 10. This large discrep-
ancy between scores (from low-end to high-end of the scale) is likely
due to the fact that two of the experts considered rainwater harvesting
systems to be constructed with reinforced concrete while the majority
did not. Similarly, when scoring piped treated sewer systems to
drought, three experts gave a score below 2 which reflect their reason-
ing that water availability affects the functionality of the system, while
the remaining six gave a score greater than or equal to 7. These differ-
ences in interpretation of the hazard or technology could be avoided
by changing the format of the expert assessment. The online survey
used in this study did not allow the survey administrator to see the re-
sponses until the full survey was complete and submitted. Using a semi-
structured interview format where the expert directly gives the score to
the survey administrator would have allowed the administrator to im-
mediately realize if an expert was providing a vastly different score
and to follow up on the reasoning behind the score.
As previously discussed in the Results and Discussion, an additional
limitation of the study was the selection of seed questions used to calcu-
late the performance-based weights. Seed questions are known to be
difficult to select, and this was seen in two of our five seed questions,
where experts misinterpreted the question. Different seed questions
may potentially lead to different expert weightings, which would in
turn affect the overall resilience score. Experts were also asked to
score the resilience of different water and sanitation technologies at a
global scale. However, resilience may be dependent on the geographic
region. Certain geologic (e.g., clay area vs sandy area) and geographic
343
regions of the world are more susceptible to specific climate-related
hazards and may have adapted accordingly, causing the ‘basic state’ of
a technology to differ depending on the region. For example, one expert
stated that in some areas prone to flooding, there is an increase in fre-
quency of boreholes being built up on concrete pads to reduce the risk
of infiltration by floodwaters. As such, in this region, ‘basic state’ may
be defined differently than this study and for boreholes, may include
concrete pads.
The results of the study provide numerical resilience scores that
allow technologies to be scored on a scale of 10 and thus captures
more differences in relative resilience as compared to a ‘low’, ‘medium’,
and ‘high’ classification scheme. However, as stated above, expert opin-
ion is subject to different personal experiences and interpretations by
experts. This study should be complemented by long-term field surveys
that consider both the resilience of a system to current climate-related
hazards and also to future climate projections, such as the types of
field work in Rajib et al. (2012) and Heath et al. (2012).
4. Conclusions
The resilience of ten drinking water and five sanitation technologies
to six climate-related hazards was assessed through an expert elicita-
tion. Resilience scores ranged from 1.7 to 9.9 out of a maximum resil-
ience of 10. Technologies for some climate-related hazards, such as
drought, demonstrated a large range in resilience, indicating that the
choice of water and sanitation technologies is important for areas
prone to drought. On the other hand, the range of resilience scores for
superstorm flooding was much smaller, particularly for sanitation tech-
nologies, suggesting that the choice of technology may not be as impor-
tant when considering the technologies in their ‘basic state’.
With the exception of two technologies (protected dug wells and
untreated community-managed piped systems using surface water),
all other drinking water technologies ranked in the top three for resil-
ience to at least one climate-related hazard. This indicates that most
technologies have specific climate regions for which they would be suit-
able and that when choosing the drinking water technology for an area,
climate projections should be taken into consideration. For regions that
use technologies with a low resilience to a specific climate-related haz-
ard, adaptation measures should be considered to increase resilience.
In general, the sanitation technologies assessed in this study showed
low to medium resilience to all climate-related hazards in their ‘basic
state’, indicating that there is potential to increase the resilience of san-
itation technologies through adaptation measures. No sanitation tech-
nology scored higher than 5.0 across all hazards, suggesting that the
resilience of sanitation technologies is highly dependent on the
climate-related hazard and thus a technology that is resilient to one
hazard may not be resilient for another hazard. As such, careful selection
and consideration of additional adaptation measures should occur prior
to the selection of a sanitation technology.
Acknowledgements
We thank the experts who participated in the elicitation and upon
whose responses this article is based. We gratefully acknowledge Dr.
Cathy Zimmer from the Odum Institute at the University of North Caro-
lina, Chapel Hill for her assistance and discussions on the calculation of
performance-based weightings. This study was made possible by the
generous support of the Wells Fargo Foundation (Clean Technology
and Innovation Grant 1450). Daniel Sebastian and Sarah Royster were
supported by the Cooperative Agreement Number 1UE1EH001126 from
The Centers for Disease Control and Prevention. Its contents are solely
the responsibility of the authors and do not necessarily represent the of-
ficial views of the Centers for Disease Control and Prevention. The con-
tents of the paper also do not necessarily represent the official views of
the Wells Fargo Foundation or the University of North Carolina at Chap-
el Hill.
344 J. Luh et al. / Science of the Total Environment 592 (2017) 334–344
Appendix A. Supplementary data
Supporting data associated with this article can be found in the on-
line version and include the interview guide, seed and survey questions,
average resilience scores for drinking water and sanitation technologies
calculated using performance-based weighting Scheme C and equal
weighting, and anonymized individual expert resilience scores for
drinking water and sanitation technologies.
Supplementary data to this article can be found online at http://dx.
doi.org/10.1016/j.scitotenv.2017.03.084.
References
Banerjee, O., 2012. Evaluating Country-Level Population Vulnerabilities to Water Access
due to Climate Related Hazards Using High Spatial Resolution Methods. (BS Thesis).
University of North Carolina, Chapel Hill, Chapel Hill, NC.
Burgess, W.G., Hoque, M.A., Michael, H.A., Voss, C.I., Breit, G.N., Ahmed, K.M., 2010. Vul-
nerability of deep groundwater in the Bengal Aquifer System to contamination by ar-
senic. Nat. Geosci. 3:83–87. http://dx.doi.org/10.1038/ngeo750.
Calow, R., Bonsor, H., Jones, L., O'Meally, S., MacDonald, A., Kaur, N., 2011. Climate Change,
Water Resources and WASH: a Scoping Study. Overseas Development Institute,
London, UK (ISSN17592917).
Charles, K., Pond, K., Pedley, S., Hossain, R., Jacot-Guillarmod, F., 2010. Vision 2030
the Resilience of Water Supply and Sanitation in the Face of Climate Change:
Technology Projection Study. (Available at). http://www.who.int/entity/water_
sanitation_health/publications/vision_2030_technology_projection_report.pdf?
ua=1.
Clemen, R.T., 2008. Comment on Cooke's classical method. Reliab. Eng. Syst. Saf. 93 (5),
760–765.
Clemen, R.T., Winkler, R.L., 1999. Combining probability distributions from experts in risk
analysis. Risk Anal. 19 (2):187–203. http://dx.doi.org/10.1111/j.1539-6924.1999.
tb00399.x.
Cooke, R.M., 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science.
Oxford University Press, USA, New York.
Cooke, R.M., Goossens, L.H.J., 2000a. Procedures guide for structured expert judgment in
accident consequence modelling. Radiat. Prot. Dosim. 90 (3):303–309 (Available at).
http://www.ewi.tudelft.nl/fileadmin/Faculteit/EWI/Over_de_faculteit/Afdelingen/
Applied_Mathematics/Risico_en_Beslissings_Analyse/Papers/RPDprcgd.pdf.
Cooke, R.M., Goossens, L.H.J., 2000b. Procedures Guide for Structured Expert Judgement.
European Commission. Report EUR 18820.
Cooke, R.M., Goossens, L.L., 2008. TU Delft expert judgment data base. Reliab. Eng. Syst.
Saf. 93 (5):657–674. http://dx.doi.org/10.1016/j.ress.2007.03.005.
Cox, R., Revie, C.W., Sanchez, J., 2012. The use of expert opinion to assess the risk of emer-
gence or re-emergence of infectious diseases in Canada associated with climate
change. PLoS One 7 (7), e41590. http://dx.doi.org/10.1371/journal.pone.0041590.
Elliott, M., Banerjee, O., Christenson, E., Holcomb, D., Hamrick, L., Bartram, J., 2014. An
analysis of the vulnerability of global drinking water access to climate-related haz-
ards. American Geophysical Union, Fall Meeting 2014, Abstract #H52E-01.
Fischer, K., Lewandowski, D., Janssen, M., 2013. Estimating unknown parameters in
haemophilia using expert judgement elicitation. Haemophilia 19 (5):e282–e288.
http://dx.doi.org/10.1111/hae.12166.
Forsberg, B., Braback, L., Keune, H., Kobernus, M., von Krauss, M.K., Yang, A., Bartonova, A.,
2012. An expert assessment on climate change and health – with a European focus on
lungs and allergies. Environ. Health 11 (Suppl. 1):S4 (Available at). http://www.
ehjournal.net/content.
Goossens, L.H.J., Cooke, R.M., 2005. Expert judgement – Calibration and Combination.
Workshop on expert judgment. Aix En Provence, France. (Available at). http://
www.ewi.tudelft.nl/fileadmin/Faculteit/EWI/Over_de_faculteit/Afdelingen/Applied_
Mathematics/Risico_en_Beslissings_Analyse/Papers/WEJ_Paper_1.pdf.
Grigore, B., Peters, J., Hyde, C., Stein, K., 2016. A comparison of two methods for expert
elicitation in health technology assessments. BMC Med. Res. Methodol. 16:85.
http://dx.doi.org/10.1186/s12874-016-0186-3.
Hay, J.E., Mimura, N., 2005. Sea-level rise: implications for water resources management.
Mitig. Adapt. Strateg. Glob. Chang. 10 (4):717–737. http://dx.doi.org/10.1007/
s11027-005-7305-5.
Heath, T.T., Parker, A.H., Weatherhead, E.K., 2012. Testing a rapid climate change adapta-
tion assessment for water and sanitation providers in informal settlements in three
cities in sub-Saharan Africa. Environ. Urban. 24 (2):619–637. http://dx.doi.org/10.
1177/0956247812453540.
Howard, G., Katrina, C., Pond, K., Brookshaw, A., Hossain, R., Bartram, J., 2010. Securing
2020 vision for 2030: climate change and ensuring resilience in water and sanitation
services. J. Water Clim. Chang. 1 (1):2–16. http://dx.doi.org/10.2166/wcc.2010.105.
Intergovernmental Panel on Climate Change (IPCC), 2008. Climate Change and Water.
Technical Paper of the Intergovernmental Panel on Climate Change. IPCC Secretariat,
Geneva (210 pp. Available at). https://www.ipcc.ch/pdf/technical-papers/climate-
change-water-en.pdf.
Intergovernmental Panel on Climate Change (IPCC), 2012. Managing the Risks of Extreme
Events and Disasters to Advance Climate Change Adaptation. A Special Report of
Working Groups I and II of the Intergovernmental Panel on Climate Change. (Avail-
able at). https://www.ipcc.ch/pdf/special-reports/srex/SREX_Full_Report.pdf.
Intergovernmental Panel on Climate Change (IPCC), 2013. Climate Change 2013: the
Physical Science Basis. Working Group I Contribution to the Fifth Assessment Report
of the Intergovernmental Panel on Climate Change. (Available at). http://www.ipcc.
ch/report/ar5/wg1/.
Intergovernmental Panel on Climate Change (IPCC), 2014a. Climate Change 2014: Im-
pacts, Adaptation, and Vulnerability. Part A: Global and Sectoral Aspects. Working
Group II Contribution to the Fifth Assessment Report of the Intergovernmental
Panel on Climate Change. (Available at). http://www.ipcc.ch/report/ar5/wg2/.
Intergovernmental Panel on Climate Change (IPCC), 2014b. Annex II: Glossary In: Climate
Change 2014: Synthesis Report. Contribution of Working Groups I, II and III to the Fifth
Assessment Report of the Intergovernmental Panel on Climate Change. (Available at).
https://www.ipcc.ch/pdf/assessment-report/ar5/syr/AR5_SYR_FINAL_Glossary.pdf.
Intergovernmental Panel on Climate Change (IPCC), 2014c. Climate Change 2014: Im-
pacts, Adaptation, and Vulnerability. Part B: Regional Aspects. Working Group II Con-
tribution to the Fifth Assessment Report of the Intergovernmental Panel on Climate
Change. (Available at). http://www.ipcc.ch/report/ar5/wg2/.
Islam, M.S., Brooks, A., Kabir, M.S., Jahid, I.K., Islam, M.S., Goswami, D., Nair, G.B., Larson, C.,
Yukiko, W., Luby, S., 2007. Faecal contamination of drinking water sources of Dhaka
city during the 2004 flood in Bangladesh and use of disinfectants for water treatment.
J. Appl. Microbiol. 103 (1):80–87. http://dx.doi.org/10.1111/j.1365-2672.2006.03234.x.
Khan, S.J., Deere, D., Leusch, F.D.L., Humpage, A., Jenkins, M., Cunliffe, D., 2015. Extreme
weather events: should drinking water quality management systems adapt to chang-
ing risk profiles? Water Res. 85 (15):124–136. http://dx.doi.org/10.1016/j.watres.
2015.08.018.
Kistemann, T., Claben, T., Koch, C., Dangendorf, F., Fischeder, R., Gebel, J., Vacata, V., Exner,
M., 2002. Microbial load of drinking water reservoir tributaries during extreme rain-
fall and runoff. Appl. Environ. Microbiol. 68 (5):2188–2197. http://dx.doi.org/10.
1128/AEM.68.5.2188-2197.2002.
Klugel, J.U., 2011. Uncertainty analysis and expert judgment in seismic hazard analysis.
Pure Appl. Geophys. 168 (1):27–53. http://dx.doi.org/10.1007/s00024-010-0155-4.
Knol, A.B., Slottje, P., van der Sluijs, J.P., Lebret, E., 2010. The use of expert elicitation in en-
vironmental health impact assessment: a seven step process. Environ. Health 9:19.
http://dx.doi.org/10.1186/1476-069X-9-19.
Luh, J., Christenson, E.C., Toregozhina, A., Holcomb, D.A., Witsil, T., Hamrick, L.R., Ojomo, E.,
Bartram, J., 2015. Vulnerability assessment for loss of access to drinking water due to
extreme weather events. Clim. Chang. 133 (4):665–679. http://dx.doi.org/10.1007/
s1058.
Ojomo, E., Luh, J., Toregozhina, A., Bartram, J., 2017. Mapping the Vulnerability of
Drinking-Water Systems to the Compound Effects of Storms and Heavy Rainfall in
Coastal Vietnam and the Philippines (Submitted).
Rajib, M.A., Rahman, M.M., Islam, A.K.M.S., McBean, E.A., 2011. Analyzing the Future
Monthly Precipitation Pattern in Bangladesh from Multi-Model Projections Using
both GCM and RCM. World Environmental and Water Resources Congress:
pp. 3843–3851 http://dx.doi.org/10.1061/41173(414)402.
Rajib, M.A., Rahman, M.M., McBean, E.A., 2012. Evaluating technological resilience of
small drinking water systems under the projected changes of climate. J. Water
Clim. Chang. 3 (2):110–124. http://dx.doi.org/10.2166/wcc.2012.019.
Roman, H.A., Walker, K.D., Walsh, T.L., Conner, L., Richmond, H.M., Hubbell, B.J., Kinney,
P.L., 2008. Expert judgment assessment of the mortality impact of changes in ambient
fine particulate matter in the U.S. Environ. Sci. Technol. 42 (7):2268–2274. http://dx.
doi.org/10.1021/es0713882.
Sherpa, A.M., Koottatep, T., Zurbruegg, C., Cissé, G., 2014. Vulnerability and adaptability of
sanitation systems to climate change. J. Water Clim. Chang. 5 (4):487–495. http://dx.
doi.org/10.2166/wcc.2014.003.
Soares, M.O., Bojke, L., Dumville, J., Iglesias, C., Cullum, N., Claxton, K., 2011. Methods to
elicit experts' beliefs over uncertain quantities: application to a cost effectiveness
transition model of negative pressure wound therapy for severe pressure ulceration.
Stat. Med. 30 (19):2363–2380. http://dx.doi.org/10.1002/sim.4288.
Study.com, 2017. Springs: Definition, Formation & Types.(Available at). http://study.com/
academy/lesson/springs-definition-formation-types.html (accessed on 04 March
2017).
Tuomisto, J.T., Wilson, A., Cooke, R.M., Tainio, M., Evans, J.S., 2005. Mortality in Kuwait due
to PM from oil fires after the Gulf war: combining expert elicitation assessments. Ep-
idemiology 16 (5):S74–S75 (Available at). http://journals.lww.com/epidem/Fulltext/
2005/09000/Mortality_in_Kuwait_Due_to_Pm_From_Oil_Fires_After.183.aspx.
U.S. Geological Survey (USGS), 2016. Infiltration – the Water Cycle.(Available at). https://
water.usgs.gov/edu/watercycleinfiltration.html (accessed on 04 March 2017).
USEPA, 2011. Expert Elicitation Task Force White Paper. (Prepared for the). Science and
Technology Policy Council, Washington DC, p. 20460.
Wang, C., Bier, V.M., 2013. Expert elicitation of adversary preferences using ordinal judg-
ments. Oper. Res. 61 (2):372–385. http://dx.doi.org/10.1287/opre.2013.1159.
WHO/UNICEF Joint Monitoring Programme (JMP) for Water Supply and Sanitation, 2016l.
Improved and Unimproved Water Sources and Sanitation Facilities.(Available at).
https://www.wssinfo.org/definitions-methods/watsan-categories/ (accessed on 04
March 2017).
World Health Organization (WHO), 2011. Guidelines for Drinking-Water Quality. fourth
ed. Geneva, Switzerland, WHO Press.