Improving Substation Reliability Improving Substation Reliability Improving Substation Reliability Improving Substation Reliability

and Availability and Availability

2009 PCIC 2009 PCIC 40 40 2009 PCIC 2009 PCIC 40 40
Authors: Authors: Authors: Authors:
Robert M. Spiewak Robert M. Spiewak Jerry Pittman Jerry Pittman Daniel Wilson Daniel Wilson
Dashiell, LLC. Valero Energy Corporation Valero Port Arthur Refining Dashiell, LLC. Valero Energy Corporation Valero Port Arthur Refining
Dominik Pieniazek Dominik Pieniazek Floyd Weisse Floyd Weisse
VI E i i LLC VI E i i LLC D hi ll LLC D hi ll LLC VI Engineering, LLC. VI Engineering, LLC. Dashiell, LLC. Dashiell, LLC.
2009 Petroleum and Chemical Industry Committee Technical Conference 2009 Petroleum and Chemical Industry Committee Technical Conference
Anaheim, California Anaheim, California
September 14 September 14--16, 2009 16, 2009
Todays Presentation Todays Presentation
Improving Substation Reliability and Availability Improving Substation Reliability and Availability
Background
Substation Documentation
Automation, Data Retrieval, Historian and Automation, Data Retrieval, Historian and
Alarm Annunciation
Monitoring and Operation Monitoring and Operation
Critical Auxiliaries
Reliability, Numbers and Statistics
Recommendations
Conclusions
Background
Background Background
Project Overview Project Overview
Large expansion at an existing refinery Large expansion at an existing refinery
Electrical Reliability Improvement Electrical Reliability Improvement
Additional on Additional on--site generation site generation gg
Multiple parties involved: Multiple parties involved:
Two Owner/Operators Two Owner/Operators Two Owner/Operators Two Owner/Operators
Three engineering companies Three engineering companies
Two construction companies Two construction companies Two construction companies Two construction companies
One third party testing company One third party testing company
One Utility One Utility One Utility One Utility
Background Background
Refinery Electrical System Refinery Electrical System
Background Background
Electrical Events After Expansion
Nov-
06
South Sub Ground on 13.8KV overhead line. Two new transformers installed with 400A
resistor in existing 1000 A ground system. Both transformers ground relays
set to trip on Pick Up.
Sep- Refinery Balanced load to generation. Trip 80MW generator, refinery imports through Sep
07
Refinery
Blackout,
Post
Hurricane
Humberto
Balanced load to generation. Trip 80MW generator, refinery imports through
230kV lines L1 & L2. Utility 230kV lines L554 & L591 in service. L554 faults on
blown debris, no reclosing. Seconds later L591 faults on lightning strike, no
reclosing. Refinery attempted to serve its load plus local community.
Generation trips on overload and blacks out.
N PP2 Li it d ti t PP2 I t d B il h t it hi f Nov-
07
PP2 Limited generation at PP2, Import mode. Boiler house operators switching for
maintenance to replace pole in tie line 205 A/B. Over current relay picked up
and tripped remaining tie line 402A/B. Training issues as refinery eliminates
tie lines and TGs.
Mar- PP6 & PP7 GTG 1 out for maintenance. Loss of a transformer along with GTG1, causes
08
g ,
overload on second transformer. DC batteries and chargers for equipment are
in PP5. Lost a single battery charger, batteries discharged, Microprocessor
relays lost power, When power restored, microprocessor relay initiated
breaker failure tripping first transformer, second transformer tripped on
overload overload.
Background Background
Electrical Events After Expansion (cont)
Jul-08 PP7 Installed temporary battery charger at PS5. Tripped overnight, repowering DC
system relays previously causing trips had been disabled, a line differential
microprocessor relay initiated an unexplained trip on repowering causing an
outage. outage.
Jul-08 230kV
private L1
Test " blue Button" trips line under a scheduled utility/refinery communication
test. Logic at both ends in microprocessor relays did not match causing trip.
Mar-
08
SG Sub TR1 & TR2 Microprocessor relay. Set to trip on pick up instead of TOC;
M SG S b TR2 t i d i ti d t h i t i t t t Mar-
08
SG Sub TR2 tripped on energization due to harmonic restraint not set
Apr-08 SG Sub Sync lights stay on all the time due to unproven wiring
Jun-
08
69kV Ties Overcurrent set points in microprocessor relays set incorrectly
Substation Documentation
Substation Documentation Substation Documentation
Requirements
Last items considered on new projects Last items considered on new projects
Must specify your requirements before Must specify your requirements before
awarding project, Tool awarding project, Tool--Kit Kit
Why Why
Downsized staff Downsized staff
Document control issues with advancing electronic Document control issues with advancing electronic
formats formats
St & t i l d i ti f d St & t i l d i ti f d Storage & retrieval during times of need Storage & retrieval during times of need
Tracking recall or advisory bulletins Tracking recall or advisory bulletins
FAT &SAT FAT &SAT FAT & SAT FAT & SAT
Substation Documentation Substation Documentation
Special Provisions
Microprocessor relays have replaced Microprocessor relays have replaced
h d i d l l i ith t l i di h d i d l l i ith t l i di hardwired relay logic without logic diagrams hardwired relay logic without logic diagrams
Complex relay schemes difficult to test Complex relay schemes difficult to test
D t t ( l tti ) D t t ( l tti ) Data storage (relay settings) Data storage (relay settings)
Electrical Operating Procedures Electrical Operating Procedures
FAT/SAT Test records is benchmark data FAT/SAT Test records is benchmark data
BOM and O&M manuals BOM and O&M manuals
Automation, Data Retrieval,
St Hi t i Storage, Historian,
and and
AlarmAnnunciation Alarm Annunciation
Automation, Data Retrieval, Alarm Annunciation
What?
Utilizing Installed IEDs
Data Mining, trending, equipment status
estimation
SystemMinimumConfigurations
Integration Best Practices Integration Best Practices
Use data to Increase productivity and
d t reduce cost
Automation, Data Retrieval, Alarm Annunciation
Typical IEDs existing within substations
Protective relays and meters
O li t h d lt On-line tap changers and voltage
regulators
Reclosers and digital fault recorders Reclosers and digital fault recorders
Bushing monitoring devices
Circuit breaker monitoring devices Circuit breaker monitoring devices
Dissolved gas in oil monitors
Partial discharge monitors Partial discharge monitors
CCTV cameras
Phone event messengers Phone event messengers
Substation access system
Automation, Data Retrieval, Alarm Annunciation
Example of monitoring class III transformer
Automation, Data Retrieval, Alarm Annunciation
Mining of Data
Data mining is the process of extracting
patterns from data.
As more data are available and gathered, g
data mining is becoming an increasingly
important tool to transform these data into p
information.
It is commonly used in a wide range of profiling y g p g
practices, such as marketing, surveillance, fraud
detection and scientific discovery,
Time for industrial application.
Automation, Data Retrieval, Alarm Annunciation
Data mining
Can effectively help determine:
required maintenance intervals,
replace interval-maintenance with risk-based
maintenance approach,
detect equipment deterioration at early stages,
or estimate other system properties.
Automation, Data Retrieval, Alarm Annunciation
Data mining
When such data is properly identified, the
facility has adequate time to
acquire replacement parts,
schedule required maintenance,
importantly avoid catastrophic failure.
Automation, Data Retrieval, Alarm Annunciation
Configuration and best practices
Some of the best practices:
Ensure system can integrate devices from
various manufacturers
Ensure system supports multiple technologies /
protocols
Utilize multifunction IEDs
Choose best IED for assigned function
Make system as flexible as possible
Automation, Data Retrieval, Alarm Annunciation
Configuration and best practices
Integrated system shall provide:
Increased system functionality
Increased system reliability/availability
(limit/eliminate outage time)
Increased system safety
Decrease a cost for automation
Decrease a cost for protection
Adequate tools at the fingertips of the engineer
Monitoring and Operation
Monitoring and Operation
Remote monitoring
For not-networked / not-connected
apparatus obtaining data about the present
state of the apparatus, IED, transformer or
IED that monitors this transformer is:
expensive,
time consuming,
could be risky.
F i t lli t (IED t k d) d t For intelligent (IEDs are networked), data
is stored in a database and repository and
t i il bl remote access is available.
Monitoring and Operation
First most manifested benefit
In not-networked system, to obtain
configuration files, SER, or DFR fromIEDs,
the engineer needed to travel to the site and
obtain files fromthe devices
In networked system, the engineer can y g
access historical data froma historian, verify
what is stored in the IED presently and how p y
historical data compares with present data
Monitoring and Operation
Examples of usage
IEDs, CTs, VTs problems detection,
System recovery, switching unexpected
loads and disturbances,
Analysis of changes required to the system
for changing operating conditions,
F l h i i f d h i For example: changing consistency of gas and how it
will influence starting and loading of the compressor
etc.,
Automatic change of protection parameters,
For example: different seasons, mode of operations
t etc.,
Isolation from utility for special conditions.
Monitoring and Operation
Remote operation
Remote control of:
circuit breakers,
switches,
lockout relays (LOR) and
other apparatus that at present requires an pp p q
operator to control it locally,
CCTV cameras,
Access doors and gates.
Monitoring and Operation
FOREMOST safety benefit
Performing switching operation remotely
eliminates arc-flash hazard exposure to the
operator / electrician by removing himfrom
the danger zone.
Note: safety procedures shall be established
first before remote operation is allowed.
Critical Auxiliaries
Critical Auxiliaries
Almost always forgotten
Substation aspects considered as auxiliary in
nature:
Control Power Supply
Time Synchronization
Enabling SER and DFR recorders
Critical Auxiliaries
Control and auxiliary power supply
Control Power Supply should be:
Maintained
Monitored
Redundant
Critical Auxiliaries
Control and auxiliary power supply
DC CONTROL POWER SUPPLY BLOCK DIAGRAM, NON-
REDUNDANT WITH MONITORING
AC
SOURCE
BATTERY
CB
TEST
SOURCE
DS1 DS2
MCB
FCB
DISTRIBUTION DISTRIBUTION
PANEL
Critical Auxiliaries
Control and auxiliary power supply
DC CONTROL POWER SUPPLY BLOCK DIAGRAM, AC
CHARGING REDUNDANT, DC BUS NON-REDUNDANT
WITH MONITORING WITH MONITORING
Critical Auxiliaries
Control and auxiliary power supply
DC CONTROL POWER SUPPLY BLOCK DIAGRAM,
2 x 100% REDUNDANT WITH MONITORING
Critical Auxiliaries
SER and DFR
SER and DFR default settings enable only
basic functionality at best case
Not utilized to collect operation and fault
information
The table in paper / Attachment C lists analog
channels and digital element recommended for
monitoring and data-collection applications
Critical Auxiliaries
Time synchronization
Cannot depend on internal clock
circuit timing drifts with usage time, temperature,
environmental instability and other factors
Th ti t dd i ifi t i f ti The time stamp adds significant information
Not synchronized information creates havoc
Time Synchronization could become a law
Some utilities require Time Synchronization Some utilities require Time Synchronization
measurements at the Inter-Tie point
Reliability, Numbers, Statistics
and
Hidd F il Hidden Failures
Reliability, Numbers,Hidden Failures
What is reliability?
Definition of reliability Definition of reliability
IEEE Std 100 EE Standard Dictionary IEEE Std 100 EE Standard Dictionary
The ability to perform a required function The ability to perform a required function under stated under stated
diti diti f t t d i d f ti f t t d i d f ti conditions conditions for a stated period of time for a stated period of time
IEEE Gold Book adds IEEE Gold Book adds
Th tit ti li bilit l ti it li bilit The quantitative reliability evaluation permit reliability
indexes to be computed from knowledge of the from knowledge of the
reliability of the constituent components of the reliability of the constituent components of the
system system
IEEE C37.100 IEEE C37.100
Reliability: a measure of the degree of certainty that the
relay, or relay system, will perform correctly will perform correctly
Reliability, Numbers,Hidden Failures
What is reliability?
What are we looking for when we evaluate What are we looking for when we evaluate
equipment / system reliability performance? equipment / system reliability performance?
How system should perform, How system should perform,
Highest reliability vs. CAPEX cost, Highest reliability vs. CAPEX cost,
Equipment replacement, Equipment replacement,
Are we considering all parameters? Are we considering all parameters?
Wide gap between design and operation Wide gap between design and operation g p g p g p g p
calculations calculations
Reliability, Numbers,Hidden Failures
How calculated?
Reliability modeling principles and data we
are using do not consider:
Varying parameters environmental parameters
Random, unpredictable chaotic disturbances
Large set samples
Other major reliability influences
Parameters used in reliability analysis are Parameters used in reliability analysis are
based on very small sampling set of data
Reliability, Numbers,Hidden Failures
Hidden Failure
Hidden failures Hidden failures
IN PROTECTION SYSTEM / UTILITY IN PROTECTION SYSTEM / UTILITY IN PROTECTION SYSTEM / UTILITY IN PROTECTION SYSTEM / UTILITY
APPROACH APPROACH
a permanent defect that will cause a relay system to a permanent defect that will cause a relay system to p y y p y y
incorrectly and inappropriately remove incorrectly and inappropriately remove a circuit element a circuit element
as a direct consequence of another switching event as a direct consequence of another switching event
IN PROCESS INDUSTY SYSTEM IN PROCESS INDUSTY SYSTEM IN PROCESS INDUSTY SYSTEM IN PROCESS INDUSTY SYSTEM
a permanent defect that will cause an EI&C system to a permanent defect that will cause an EI&C system to
incorrectly and inappropriately remove a circuit element incorrectly and inappropriately remove a circuit element incorrectly and inappropriately remove a circuit element incorrectly and inappropriately remove a circuit element
as a direct consequence of another usually abnormal as a direct consequence of another usually abnormal
event in EI&C system and/or plant process event in EI&C system and/or plant process
Reliability, Numbers,Hidden Failures
Causes of Hidden Failure Examples
Cause examples: Cause examples:
Circuit fault
Shortage of reactive power
Randomoperation of DC power system Random operation of DC power system
Inadequate testing / commissioning
Operation of E&IC systemin seldomused state Operation of E&IC system in seldom used state
Reliability, Numbers,Hidden Failures
Hidden Failure Host Examples
Hidden failure host Hidden failure host examples:
Control schemes from previous projects
Not performing FAT or SAT to minimize cost
Removing monitoring equipment Removing monitoring equipment
Selecting vendor/contractor based on capital
cost
Applying firmware without testing
Not updating firmware
Reliability, Numbers and Statistics
How applies to reliability?
RELIABILITY INDICES COMPARISON
WITH AND WITHOUT HIDDEN FAILURES CONTINGENCES
RELIABILITY
INDEX
NAME
HIDDEN
FAILURES
NOT
HIDDEN FAILURES
INCLUDED
CASE 1 CASE 2
NAME
INCLUDED
CASE 1 CASE 2
FAILURE PROB
1.1130E-03
1.1880E-03 1.6850E-03
INCREASE 107% 151%
FAILURE FREQ [ /YR]
0.4425
0.4586 0.6034
INCREASE 104% 136%
FAILURE DURATION
[HRS/YEAR]
22.03
22.68 24.68
INCREASE 103% 112%
Note: Hidden failure probabilities: Case 1 - 10e-3, Case 2 - 10e-2
Reliability, Numbers and Statistics
Reliability
Data in Gold Book is a generic
Small sample set
Sample set collected in minimal certain Sample set collected in minimal certain
conditions
Calculated indices Calculated indices do not represent reality Calculated indices Calculated indices do not represent reality
Paper is focusing on improving VAILABILITY
f th EI&C t t li bilit i di of the EI&C system, not reliability indices
Summary
Summary
Formulate practices, standards, and data
documentation applicable to your
organization
Include in RFQ requirements special needs
that are not included in industry guides, y g
specification, practices
Evaluate howthe project should be Evaluate how the project should be
executed
Assure contractual cooperationwith utility Assure contractual cooperation with utility
for post mortem analysis
Summary
Include provisions in POs allowing for third
party to provide testing services without
altering contractual responsibilities and
obligations
Incorporate procedures for updating and p p p g
storing advisory bulletins and recall
information for all apparatus and IEDs pp
Apply automation for communication with
IEDs and file management IEDs and file management
Summary
Implement procedures and software for
analysis of the collected data
Apply reliable engineering principles for AC ppy g g p p
and/or DC control power sources design
Test periodically power systemalarms and Test periodically power system alarms and
monitoring
Start collecting power systemdata apply Start collecting power system data apply
time synchronization
Conclusions
Conclusions
Reliability and availability of the EI&C system
in process plants:
Difficult to estimate
Not considers project factors
Sensitive to input parameters and environmental
conditions
Based on very small sample set of data
Conclusions
The reliability and availability of EI&C system
can be improved by focusing on time-tested:
Good engineering practices,
Appropriate and correct documentation,
Apparatus/equipment testing, commissioning
and monitoring,
Applications of available technologies to
performed monitoring tasks with minimum
human supervision,
Perform tasks routinely to eliminate hidden
failures in the system.
Conclusions
Data collected during monitoring of the EI&C
system and equipment could
Foresee approaching equipment failure, trigger
an alarm i.e. safe time and money!
Aid at verifying, adjusting, calibrating power
system models for power studies
Use data to your advantage!!! Use data to your advantage!!!
Do not forget about Test Switches
TEST SWITCH IS YOUR FRIEND !!! TEST SWITCH IS YOUR FRIEND !!!
Questions?
Thank you