Transitioning the Corbalis Water

Treatment Plant’s Control System

Mid-Atlantic Applications in Automation

Conference
March 24, 2010
Standards
Certification
Education & Training Eric Silverman, P.E. – CDM
Publishing
Conferences & Exhibits Josh Gelman, P.E. – CDM
Presenters

• Eric Silverman, P.E.

– Cambridge, MA
– Northeast Region Automation/I&C Team Leader
– Lead I&C Design Engineer
– Construction Services

• Josh Gelman, P.E.

– Falls Church, VA
– Mid-Atlantic Automation/I&C Team Leader
– Lead Applications Engineer
– Construction Services

2
Today’s Agenda

• Design Overview
• Client Goals and Challenges
• Application Engineering
Process
• Construction
• System Transition
• Success Factors
• Questions
Design Overview – Project Timeline

• In order to meet planned water demand projections,

Fairfax Water (VA) executed a planned expansion of the
J. Corbalis Water Treatment Facility
End of 2009 -
Overall
Construction
Completed

July 2008
January 2005 Corbalis Rated at 225 MGD
Construction Begins
June 2004
Final Design Complete
March 2003
Engineering Design Report

October 2002
Expansion Capacity Study
(Plant Rated at 150 MGD)
Design Overview – Control System
Scope of Work

OWS/LOP
-Replace existing -Add new controls
control system to support the new
hardware and “Process Train”
software along with other
plant upgrades
- Install new fiber
Controller
Controller
optic network

-Integrate existingField
control logic
Field
associated withInstruments
existing process
Instruments
equipment into new control system
SCADA System
 Overview - Corbalis SCADA Components

HMI Operator Historian

Servers Workstations Servers

Fiber Ring Network w/

Managed Ethernet
Switches (MES)

Local Programmable
Operator Panels Logic Controllers Operator
Workstations
Overview – SCADA Network Redundancy

2 Hr. Back-up
No Data from
UPS time for SCADA
DYNAC Equipment and
Chem. Bldg. @ OWS
Instruments.
LOP Used this OWS
to Control
Process LOP
LOP
6-Pair Fiber
Locally
PLC
PLC
MES
MES
Chem. Bldg.
X
UPS
X Corbalis DYNAC

SCADA “Ring” MES

MES
OWS

MES
MES
No Data from
Chem. Bldg. @
HES – Hirschman Ethernet Switch Facility Support Center this OWS
LOP – Local Operator Panel
OWS – Operator Workstation
PLC - Programmable Logic Controller
UPS – Uninterruptible Power Supply
Design Challenges

• Develop an understanding of
the existing system
• Avoid Technology
Obsolescence
• Ensure a “smooth” system
transition with minimal
downtime
• Develop drawings and
documents that clearly indicate
scope of work
Design Challenges

• Develop an understanding
of the existing system
- Consistent communication with
Fairfax Water staff
- Reviewing documentation
- “Walking” the system
- Taking time “on site” – no shortcuts
here.
Design Challenges

• Avoiding Technology
Obsolescence
– $200k Allowance
– Contractor restricted on
releasing computers for order
Design Challenges

• Ensure a smooth transition with minimal

downtime
– Turn the existing DCS panels into termination boxes and run
systems in parallel
– Require contractor to submit Maintenance of Plant Operation
Plans (MOPO) on a building-by-building basis
Design Challenges
System Transition

• Require contractor to submit Maintenance of

Operation Plans (MOPO) on a building-by-building
basis

Step-by-step procedures

Each I/O point identified in plan

Outage durations
Contingency plans
Critical Design Documents

• Control System Architecture (CSA)

• Piping (or Process) and Instrumentation Diagram
(P&ID)
• Control Strategies
Control System Architecture (CSA)

PCSS Panel Communication

Enclosures
Local
Workstations

Comm.
Cables

Vendor Panels

UPS
 Piping (or Process) &
Instrumentation Diagram (P&ID)

Existing vs.
New Instr.

New vs. Old

Signals

Vendor
Where signals Control Panels
are wired
Control Loop #s and
Description
Strategies

PLC
Logic

LOP/OWS
Functions

Alarms and
Monitoring
Applications Engineering (AE)

• Major Components:
– HMI System

– Local Operator Panels

– PLC Systems

– Data Historian

– Plant Reports

– Data Network
AE Project Goals

• Keep the plant running

• Maintain open lines of communication
• All components are tested prior to system start-up
• Jointly develop a thorough plan to follow
• Develop new system that meets user’s needs
• Staff is fully trained on system after project
completion
• Develop as-builts and documentation for the entire
system
Fairfax Water Project Challenges

• Moving Water
• Running on Multiple Computer Systems
• Interfacing to Different Generations of
Equipment
• Integration of “Vendor
Systems”
Application Engineering Process-
Collaboration During Development

• Work Plan provided a

“Roadmap for Success”
• Series of submittals
• System development
staged at local office
• Monthly development
meetings and workshops
• Informal training provided
improved understanding
Application Engineering Process:
Keys to Success

• Follow development procedures and standards

• Develop HMI graphics to convey process and
associated automation
• Provide PLC programming with sufficient
documentation
• Configure back-up touch screens to match HMI
workstations
• Be sensitive to Fairfax Water security
concerns
• Contribute as extension of
client staff
• Accessibility to all parties.
Construction - Phased Implementation

• Developed systems to match Contractor’s

schedule
• Each phase was treated as a mini-project
• All components tested and ready in advance of
commissioning
• Applied lessons learned immediately.
 Sequence of Construction

• Coordinated to match operational and contractual

constraints
• Factory test all components prior to field delivery.
• Install and test new fiber network.
• Install new components.
• Logically disconnect and re-terminate I/O from
existing system to new system
• Fully test systems as the complete process becomes
available
• Re-commission existing equipment
CDM and FW Challenges
During Construction

• Major changes to the construction

schedule impacted programming
• Undesirable process sequencing
• Network interruptions early
in “ring” evolution
• Difficulties in reviewing submittals by
Fairfax Water staff
• Adapting to contractor personnel
• PCSS responsibilities transitioned 8
months into construction
Success Factors

• Frequent Communications
– Daily informal meetings
– Weekly formal meetings

• Teamwork
• Operator awareness of I/O
transition
• Developer’s on-site office
Success Factors

• No major shutdowns or
disturbances
• New system has been
embraced by operations
• Accessibility to developer
minimized need for formal
training
• Engineer’s Work Plan has
evolved into programming
standards
Standards
Certification
Education & Training
Publishing
Conferences & Exhibits
 Transitioning the Corbalis Water Treatment Plant’s
Process Control System
Joshua Gelman, P.E. Eric Silverman, P.E.
Mid Atlantic Automation/I&C Team Northeast Automation/I&C Design
Leader Team Leader
CDM CDM
3130 Fairview Park Drive, Suite 400 50 Hampshire Street
Falls Church, VA 22042 Cambridge, MA 0213
KEYWORDS
Automation & Control, Engineering Design Report, Redundancy, Transition Plan, Final
Design, Construction, Applications Engineering, Start-up Challenges
ABSTRACT
Transitioning the control system for an operating water treatment plant is always a source of concern
for utility operations and managers. When you are the largest water treatment plant in Virginia and
providing water to 1.5 million customers, your concerns are even greater. Shutting off the water supply
to a community of this size is not an option!
Fairfax Water’s Corbalis Water Treatment Plant (WTP) in Virginia has recently completed an
expansion project that increased the plant’s capacity from 150 million gallons per day (mgd) to 225
mgd. In addition to the process upgrades, the existing distributed control system (DCS) from the early
1990s was completely removed and replaced with a modern, state-of-the-art process control system
(PCS). This paper will focus on the approach and procedures taken to transition the plant’s existing
DCS system to the new PCS without significant process downtime during the upgrade.
Early in the construction phase, it became evident that the project would succeed only through a
collaborative approach between the engineer and owner. The contractor established an aggressive
transition schedule that needed support to meet the project deadline. Based on contractor-developed
maintenance of plant operations (MOPO) plans, the engineer then developed a programming schedule
to support the contractor’s phased approach to the transition. Working closely with the owner, a
collaborative approach for implementation was developed, which allowed for open communication and
participation between the engineer’s development team and Fairfax Water staff. Other benefits of this
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
collaborative approach included early buy-in from plant staff, improved system understanding, and
informal training on the PCS system throughout the testing and start-up phase.
The transition sequence began by taking a single DCS process area off line to transition the existing
system over to the new PCS. During this transition, the operations group was required to control the
entire plant using two separate systems until all systems were successfully switched over to the new
PCS. To help mitigate concerns by operations, the engineer conducted informal training sessions with
operations staff to prepare them for the new system. Those sessions proved to be successful, since
operations embraced the new system early, and took ownership immediately after each process area
was transitioned to the new PCS.
This paper will focus on system transition—the initial planning and design, the construction process,
and implementation of the new PCS. It will also highlight the transition process for a live plant when
switching from one control system to another, without plant disruption or system downtime.
INTRODUCTION
This paper will discuss the project lifecycle from preliminary design through start-up and
commissioning of the Fairfax Water Corbalis Water Treatment Plant (WTP), Stage III Upgrade located
in Virginia. The emphasis of the paper will be on the project delivery methods that were implemented
to ensure a successful transition from the existing legacy Distributed Control System (DCS) to the new
Programmable Logic Controller (PLC) system and Human Machine Interface (HMI) without
significant process downtime while simultaneously meeting project goals. In addition to transitioning
the existing control system, new process areas were also incorporated into the new control system
which expanded the plant’s capacity from 150 million gallons per day (mgd) to 225 mgd. At project
completion, the new control system consisted of over 6,000 input/output (I/O) signals in the PLC
system and over 15,000 software tags at the HMI.
Although the focus of this paper is on the collaborative relationship between Fairfax Water and their
consultant, Camp Dresser & McKee Inc. (CDM), there were several other parties that were involved
with the automation and instrumentation and controls portion of the project. The following table
highlights these parties and their role on the project.
Table 1 – Roles and Responsibilities for Control System Design and Integration
Party Role on Project
Control System Design, Construction Manager, and control
CDM
system Applications Engineering System Supplier (AESS)1
General Contractor – Installation of control system and
Pizzagalli Construction Company
instrumentation
Process Control System Supplier (PCSS) – Control system and
CIM/Transdyn, Inc.
instrumentation supplier and related start-up services
1
The AESS is responsible for implementing the control system design into a seamless functional
system by developing custom programs, graphics, and reports.
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
PROJECT GOALS
Fairfax Water and CDM established several goals for the project which were as follows:
• Maintain plant operations during construction and system transition.
• Maintain open lines of communication with the Engineer and AESS.
• Establish rigorous testing protocols to ensure all components are working prior to system start-up
• Collaborate with the AES and Engineer to jointly develop a start-up plan
• Develop a control system and software that meets user’s needs now and into the future.
• Provide sufficient training so that staff is fully trained and comfortable with the new control
system.
• Develop detailed documentation for the system
Throughout the project, Fairfax Water and CDM worked together to ensure that these goals were met
by implementing appropriate protocols and design criteria and then by strictly adhering to them.
Establishing these goals at the beginning of the project was critical to the overall project success.
These goals will be referenced throughout the paper.
CHALLENGES OVERCOME
Even though the entire start-up was considered a resounding success, the automation team had their
fair share of challenges to overcome throughout the project. Some of the major challenges were as
follows:
ESTABLISHING A PLAN
Determining a plan for transitioning the system was something that required immediate attention and
could not wait until project construction. Starting with an Engineering Design Report, the groundwork
began to develop guidelines to ensure that the contractor would follow a logical construction path and
also have as much information for them as so they could develop the plan.
ADAPTING TO A CHANGING CONSTRUCTION SCHEDULE
In late 2007 the contractor presented a new construction schedule, changing the sequence of transition
for several major processes to be brought online. This change was especially challenging for the AESS
who had to stop working on the current set of PLC systems and start working on PLC’s that had been
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
originally scheduled later in the overall development schedule. This placed tremendous pressure of the
development team adapt to the new schedule and not only develop the PLC logic and create and
animate the associated graphics, but to also provide a thorough factory test prior to field testing and
start-up. For the next three months, the AESS worked furiously to complete the task in order to be
ready for the March start-up, which they completed on time.
INSTABILITY OF A DEVELOPING AND TRANSITIONING NETWORK
The fiber optic ring went through a period of instability that was very problematic due to the phased
start-up of new process areas as well as the transition of existing systems to the new control system. In
the beginning, the network would occasionally drop out, especially when someone was working within
one of the Fiber Optic Communication (FCE) cabinets. This was troubling since the PCSS performed
network testing that included continuity checks, signal propagation, and self-healing tests. Early
diagnostic efforts by the PCSS reported that several fiber terminations within the cabinets needed to be
re-terminated. After several months of continual problems, the manufacturer’s representative for the
Managed Ethernet Switch came to the plant to investigate and troubleshoot the network. It turned out
that the self healing functions were correctly set-up in software; however, the fiber connections within
the switch were terminated in the wrong locations on the module, thus causing occasional drops within
the network. After the modifications were made, the fiber optic network did not fail.
TIMELINE
This project was implemented in three stages: engineering design report, final design and
construction/applications engineering with the timeline as follows:
• Engineering Design Report – September 2002 through March 2003
• Final Design – April 2003 through June 2004
• Construction and Applications Engineering- January 2005 through December 2009
We will now look at each of these project phases in more detail.
ENGINEERING DESIGN REPORT (EDR)
The EDR phase is where the hardware, software and design protocols were established for the new
control system. Having several workshops early-on with all of the stakeholders helped build support
throughout the organization as well as document the key design elements that would later be
implemented during design and programming. The EDR and workshops addressed the following
items:
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
HARDWARE AND SOFTWARE STANDARDIZATION
The Corbalis Water Treatment plant is one of several facilities in the Fairfax Water distribution system
so it was critical to determine which components of the control system needed to be standardized.
Standardizing on key components would help minimize training as well as decrease spare part
requirements as these components were already “on the shelf”. Fairfax Water decided to standardize
on the PLCs, HMI, network switches, computers, and some instruments.
REDUNDANCY
This plant serves a very large population so minimizing disruptions and downtime was critical design
requirement. As a result, there were many layers of redundancy placed in the system in order to ensure
that this control system would remain operational. Areas identified were PLCs, HMIs, back-up power,
computers, and network communications.
For PLCs, each process area would be designed with redundant PLC processors such that in the event
of one processor failure, there would be a “bumpless” failover to the other processor without any
downtime. For HMIs, there would be local Operator Interface Terminals wired directly to the PLCs so
that in the event of a network failure, an operator could run that process area locally from the OIT as
opposed to the Operator Workstations located in the control room. Back-up power would be provided
in each process area with an Uninterruptible Power Supplies (UPS) to provide a 2-hour back-up time to
critical instrumentation, PLCs, and HMI systems in the event of a plant power failure to allow for an
orderly shutdown. Redundant data servers and collectors would maintain data integrity in the event of
a hard drive failure. Finally, the network switches would be managed switches configured in a ring
such that a break in the fiber optic “ring” would reroute the data traffic away from the break to
maintain network communications.
GENERAL PROCESS TRANSITION PLAN
A baseline transition plan was discussed and documented for each process area that would later be the
basis for a sequence of construction as part of the design documents.
CONTROL PANEL TRANSITION
Every existing DCS control panel was identified with a preliminary transition plan for it which was
either to demolish it or turn it into a junction box which would then have its associated input/output
(I/O) signals extended to the new PLC panels.
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
CONTROL PANEL REQUIREMENTS
All of the construction requirements for the new PLC control panels were identified including items
such as surge protection, signal wiring, terminal block styles, and more.
FINAL DESIGN
Once the EDR was completed, the project transitioned into the final design phase. With the EDR in
place, many of the project concepts were already finalized which allowed CDM
Automation/Instrumentation and Controls (I&C) Engineers to spend more time investigating the
details of the existing DCS system and incorporate that information into the design documents. The
following items were challenges that were overcome during the design process.
DEVELOP AN UNDERSTANDING OF THE EXISTING SYSTEM
In order to transition the existing software programs from the old DCS logic to the new PLCs, a
thorough understanding of the existing programs was required. This task involved many days on site
collecting programs and documentation for each process area and then considerable time to understand
each line of code. The code had minimal documentation making it even more challenging to interpret.
Ultimately, the code was deciphered and then translated into “control strategies” which are written
descriptions of the logic and includes items such as initial process setpoints and variables to be
monitored and controlled at the HMI.
AVOID TECHNOLOGY OBSOLESCENCE
During design there were many discussions related to protecting against technology obsolescence as
we knew that the construction period would be over several years and any computers installed early on
in the project would be towards the end of their lifecycle when construction was completed. In order
to protect against obsolescence, a $200k allowance was written into the contract that allowed Fairfax
Water to use this money to upgrade their computers or to use for other items. This money was
ultimately used to remodel the existing control room in addition to replacing computers which was an
added bonus to Fairfax Water.
Another way we helped to avoid technology obsolescence was to specify that the PCSS could not
release the order for computers until instructed by the Engineer. The way this worked was that the
PCSS submitted the technical data on the computers as part of their hardware submittal during
construction and were given a conditional approval based on the technical specifications. Then CDM
determined when they needed the computers for start-up and released them at a later date which
resulted in the client obtaining the latest model.
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
SMOOTH TRANSITION OF PROCESS SYSTEMS
As discussed earlier, the EDR provided an initial plan for transitioning the control system in each
process area. This was refined during final design into a sequence of construction for the contractor
and a requirement for the PCSS to submit on a Maintenance of Plant Operations (MoPO) for each
process area that would clearly define the transition steps from the old control system to the new. The
MoPO plans were also required to have information such as a point-by-point schedule and transition
plan, outage durations, contingency plans and any temporary systems that may be required to be
constructed to assist in maintaining operations during transition. The use of the MoPO plans will be
discussed in more detail later on.
DEVELOP DRAWINGS AND DOCUMENTS THAT CLEARLY INDICATE SCOPE OF
WORK
With so much existing wiring and control panels being re-used it, was pertinent to create plans that
clearly differentiated what would remain as existing and what would be required to be new. Different
fonts and line weights were used on the control system architecture drawings and Piping (or Process)
and Instrumentation Drawings. This attention to detail was critical in minimizing change orders during
construction as well as providing clear and concise documents for use by the general contractor and
PCSS.
CONSTRUCTION AND APPLICATIONS ENGINEERING
SETTING THE STAGE
In early 2005, project construction for the facility upgrades was underway and the applications
engineering (AE) effort was gearing up. The project delivery method employed was a traditional
design-bid-build approach with all hardware and software competitively bid and provided by the
general contractor. CDM was performing the software integration duties as the AESS working under a
professional services contract with the Owner. Using the engineer for software integration was new to
Fairfax Water (FW), as they traditionally elected to go with a traditional system integrator; however,
they liked the idea of maintaining continuity with their engineer CDM as they transitioned from the
design team to the applications engineering team.
The contract documents specified that the General Contractor was required to obtain the services of an
instrumentation / control system supplier specialist, referred to as the Process Control System Supplier
(PCSS). This PCSS was responsible for furnishing, installing, and successfully testing the process
control system hardware components including PLC panels, servers, operator workstations, UPS
systems, and instrumentation. One of the early deliverables for the PCSS was to procure and turnover
all new hardware and software to the AESS for full-system development and implementation. CDM
was not only responsible for the system development, but also provided construction services to
oversee the contractor responsible for the field installation aspects of the control system. Tasks
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
included shop drawing review, witness factory testing of control system panels, responding to
contractor questions, and attending monthly coordination meetings.
Prior to scheduling the AE Kickoff meeting with FW, CDM developed the AE Work Plan, which
provided details for the project execution. The purpose of the work plan was to ensure all project
participants involved with the control system integration had a clear understanding of the project
scope, internal management procedures, project team staff and responsibilities, communication
procedures, and key project milestones. In addition, CDM included detailed development standards for
the HMI, PLCs, plant reports, and plant wide Internet Protocol (IP) network addressing. Establishing
the detailed standards up front and obtaining FW’s buy-in early would allow for full-scale
development to take place immediately.
The standards portion of the work plan took some time in the beginning. CDM investigated any
previously documented control system guidelines plus any standards that had evolved from other
recently completed FW projects. CDM’s approach was to capture the existing standards plus provide
recommendations for new standards.
With the completion of the AE Work Plan, the kick-off meeting took place in August 2005. In this
meeting, key project stakeholders from FW and CDM gathered to discuss the work plan in a
comprehensive workshop. The outcome of the meeting was positive, as many of the proposed
standards were accepted. In addition, FW relayed their project goals and objectives for the effort.
Their most important objective was to keep the plant operational during the system upgrade. There
would only be limited opportunities to shut down the entire plant or even a specific process. Another
major objective was to make sure all users’ needs were met. Some of the different users of FW
included plant operations, distribution operations (different groups of operators responsible for various
parts of the plant/system), plant maintenance, engineering, and construction. Lastly, it was important
to have the entire staff trained on the new system prior to project completion.
SYSTEM DEVELOPMENT
After the kick-off meeting, FW and CDM staff met regularly to review all aspects of the program
development. It was through these workshops that FW staff provided regular input on the entire
system to ensure that stakeholder expectations would be met. CDM staged system development at its
nearby Falls Church office. FW regularly visited the office to review the progress and provide input.
This process facilitated extensive collaboration between FW staff and the development team, leading
to buy-in, improved system understanding, and informal training on SCADA system operations.
In the configuration office, CDM staged the early development system, consisting of the redundant
SCADA servers, two operator workstations, one local operator panel display, one hot-standby PLC
system with remote I/O rack, and the Historian cluster. Staging the system using the actual components
allowed for system development and testing to take place in an environment that mimicked the final
installation. Essentially, the early development system consisted of the primary components for the
system.
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
CDM’s experience with the actual system components was varied. First, CDM is well versed in
programming PLCs and the Local Operator Panels; however, experience configuring the specific top
end HMI and Historical systems was limited. The Historian was extremely challenging, as the system
was a custom solution for the top end HMI system. This was very different than the common historical
packages in the market today, as this system consisted of two redundant servers running Solaris and a
250 GB disk array for long term storage. The two servers and disk array were configured as a cluster,
making the three units act as one single computer. This configuration provided the server and data
redundancy required for the system.
The top end HMI is an integrated set of applications running on OpenVMS servers provides the
functions of traditional HMI packages. System development was typically done by the vendor, but for
this project they provided CDM external support. The software was purchased and owned by FW and
turned over to CDM for the project
To confirm that the HMI and PLC systems would work as proposed within the work plan, CDM
programmed and configured the first PLC, the Finished Water Pump Station PLC. This allowed the
I/O addressing within the PLC to be arranged as efficiently as possible. The top end HMI handled
digital I/O points in device specific tag groups rather than as separate inputs and outputs. For instance,
if a full open, full close valve had both open and close control operations, this could be easily related
as 4 tags, one for each point. However, within this HMI, the valve would be expressed as one tag with
4 I/O addresses assigned. Because of this aspect, CDM was able to refine the I/O addressing scheme
for the project before mass-scale development took place.
Being able to distribute system development to the virtual team was made possible through the use of
the PLC/HMI memory map. The memory map is a comprehensive listing of the data
intercommunications between the PLC, OWS, and LOPs systems. The memory map was based on the
approved I/O list developed by the PCSS for each PLC panel. Each item included within the list was
either required to be displayed on a HMI (OWS or LOP) screen or used by more than one PLC. For
each entry, the tag name, description, and address were included, plus other supplemental information,
if pertinent. Once developed, the memory maps could be used to generate databases used for any of the
software packages. Throughout the project, the memory map was the single most important tool used
in developing, troubleshooting, and maintaining the control system.
TRANSITIONING THE SYSTEM
The general contractor’s critical path method (CPM) schedule sequenced construction into seven main
phases. Each phase included two or more PLCs and was used to strategically group areas of the plant
into logical phases supported by operations. Therefore, CDM’s strategy was to fully develop and test
the PLCs (including the associated HMIs) based on this construction sequence. What this did was
establish each phase as a miniature project where the PLC and associated systems would be fully tested
prior to system commissioning. The benefit this provided was it ensured everything required for the
phased start-up was ready in advance and lessons learned could be applied immediately to the
subsequent phases. Having the early development system in place allowed for system testing to be
completed on the actual end system before system start-up.
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
While the AESS was completing their work, the PCSS was performing their work in parallel. The
PCSS was also following the general contractor’s schedule sequence. After the approved I/O list, the
PCSS went to work on the detailed hardware, panel, and loop drawings for each PLC area. In addition,
the new fiber communication enclosures (FCEs) were in fabrication in the PCSS’s factory shop and
ready for factory testing.
In August 2007, the new fiber optic network was ready for testing. Factory testing was successfully
completed, and the FCEs were shipped to the plant. The contractor’s plan was to incrementally build
out the fiber optic network based on the completion of new buildings; thus, the ring began small and
slowly grew until it was completely built out. In September of that year, the first FCEs would come on-
line.
At the same time, the AESS was in the process of relocating their office and system components to the
plant. Since the actual servers and equipment were used by CDM, they needed to be installed and
ready for the first phased startup. FW provided CDM a large space within the operations building to
set-up an on-site work space. This space was excellent, as it was directly below the Control Room
including the server room, housing the critical equipment. The location was perfect, as the AESS
maintained an onsite presence using the space as their on-site lab; however, the location was out of the
way of daily activities.
In the middle of October 2007, preparations were made to transition the first PLC, the Finished Water
Pump Station, over to the new system. The fiber optic network was in place, top end servers
operational, and a couple of temporary workstations were set-up in the control room for operations.
The on-site system transition team, made up of the general and electrical contractors, the PCSS, the
AESS, FW construction inspector, and FW operations supervisor, started conducting daily meetings to
coordinate daily and weekly activities. These 20 minute meetings proved to be worthwhile, as the
entire team, including plant operations, needed to be on the same page coordinating system transition.
From the instant the first signal was moved over to the new system, plant operations, water quality
laboratory staff, and plant management had to know and understand the implications. The entire
system transition was expected to take a week.
Prior to system transition for any area, the general contractor was required to develop the Maintenance
of Plant Operations (MoPO) Plan detailing how the process area was to be transitioned over to the new
system in an orderly manner. The MoPO contained step-by-step procedures, required durations, and
specific steps required to install and place into operation all instruments, panels, and control devices,
including specific procedures to demolish, remove, and decommission all existing components.
Additionally, the MoPO contained any temporary measures required to maintain operations. The
development of this plan was a complete team effort and required the approval of all parties prior to
beginning the transition of an area.
When the time came to transition the first process area, the plan was to set-up the new PLC control
system in parallel to the existing DCS Distributed Control Unit (DCU) cabinet and transition points
one-by-one over to the new system. The existing Distributed Control Unit (DCU) enclosures are
where the existing field instruments and equipment were terminated and tied into the legacy DCS
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
components. The intent was to install new termination blocks inside the DCU enclosures and convert
the enclosure into a junction box for the new system. The benefit of this approach was that all of the
existing field wires would remain untouched, thus eliminating massive amounts of rewiring. The new
PLCs utilized a prewired cable system between the PLC I/O module connectors and the terminal
blocks providing a quick connection and disconnection system at both ends. This reduced overall
installation time and system transition essentially boiled down to lifting wires off of the old system and
landing them into new terminal blocks inside the DCU cabinets. After the existing DCU’s points were
all moved to the new system, all of the legacy components inside the cabinet could be removed and the
termination blocks could be permanently installed inside.
After each point or set of points were transitioned, the PCSS and AESS would test the points from the
field to the HMI screen for proper verification. In general, there were not many problems; however, the
most common issues during point verification were related to lack of sufficient signal isolation, loop
versus panel powering for field devices, and unintentional multiple grounds. After the group of points
was tested, the AESS and plant operations staff tested the points collectively as a group to verify the
process control strategy. After verification, the system of points was placed into plant operations for
immediate use. At this stage, alarming and basic remote manual control was available at the HMI. The
AESS and plant operations would then schedule a time to test the automatic strategies as a system.
This systematic method was embraced by operations staff and they quickly learned to trust the new
system. To avoid confusion from having two systems in operation at the same time, FW process
engineers would remove the points from the old DCS system as they were transitioned and modified
the DCS HMI screens to convey that they were no longer active. This was important, as FW operated
four team shifts that rotated, thus some operators would be off for up to six days at a time not knowing
what had been tested while they were gone.
Before any points were transitioned over to the new PLC system, the AESS would conduct preliminary
operator training to introduce the new system to the operators. This training provided basic exposure
to the new screens, accessing control functions, and the alarm and trending applications. Early
exposure was important so operators could preview the process screens and ask questions before
having to use them on the actual system. In addition to the preliminary training, process specific
training was provided before any of the individual processes came online. Midway through the
project, operations mastered the new controls philosophy and standards therefore training from then on
were conducted on an as-needed basis with a final comprehensive training session performed at the
end of the project.
Over the course of the next 20 months, the rest of the plant processes were brought online with the new
control system. The existing DCS did not include much automation, with the exception of some
automated filter backwash sequences and pump control interlocks for certain plant-wide alarm
conditions. The new system automated the majority of the plant processes which included: automated
filter backwashes that could automatically schedule backwashes based on an intelligent priority queue,
dosage control for the chemical feed systems, and many other process enhancements. In addition, all of
the vendor-provided systems (i.e. small pre-packaged systems) were integrated into the overall control
system, not only for monitoring, but also for control. This eliminated all standalone systems as the
entire plant was now on one system.
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org
SUCCESS FACTORS
Although there were many challenges faced by the team, several factors led to overall success. One of
the most significant factors for success was the regular progress meetings. There were two types of
meetings which took place. First, daily informal meetings were held between the major parties towards
the conclusion of the day to discuss daily progress and to coordinate for the next day. Although it was
a challenge for everyone to attend, these meetings were necessary and vital to the project’s success.
For a short duration of the project, progress was significantly hampered because these meetings did not
occur. The meetings were quickly re-established and the project kept moving along. After that early
set-back, everyone on the project team truly understood the importance of these meetings and
attendance no longer became an issue. Second, formal weekly meetings were held to go over project-
wide issues, rather than area specific issues. These meetings followed a prepared agenda and were
always concluded with action items. The combination of daily and weekly meetings was the right
formula for the overall project success!
Another factor leading to success was the teamwork displayed on a daily basis by all of the parties
involved. Early on and throughout the project, team building activities were conducted to remind
everyone of the immense challenge that was faced. Throughout the project, when problems or
difficulties were presented, rather than finger-point, the project team worked together to resolve the
problems. In the long run, this saved valuable project schedule time as there was no exhausting letter
writing campaigns or long email chains to solve problems.
The last major factor to success was the effort put forth by the operations staff. From the first day when
construction broke ground to the last days of I/O point sign-offs, the plant operator’s normal day was
anything but normal. Despite what they had to endure, they were always extremely accommodating to
the AESS by allowing everything from temporary system shutdowns for performing system compiles
to testing brand new control strategies. This constant interaction between the AESS and operations
staff provided opportunities for the operations staff to be part of the project and to provide informal
feedback. The AESS found this interaction invaluable, as minor changes and tweaks could be
completed immediately which fine-tuned the system for FW.
CONCLUSION
At the conclusion of the project in late 2009, the project, including the new control system, was
deemed an overall success by the Owner. The AESS had just completed maintenance training which
provided FW staff with training on the PLC, HMI, and other new systems. The project suffered no
major shutdown or disruptions and the AESS’ work plan for the project evolved into the Owner
programming standards for all future automation projects. The new system was embraced by
operations, as the new automated strategies and procedures made their jobs much easier with a system
that was tailored specifically to their need.
Distributed with permission of author(s) by ISA 2010
Presented at ISA Mid-Atlantic Applications in Automation Conference 2010; http://www.isa.org