Remote Control of VMT Final Report Review Contract: 507.12.01 PWSRCAC Board Meeting January 23-24, 2014 Hisey and Associates, LLC
Overview ∗ Scope of Contract ∗ Definition of Key Terms ∗ Review Process ∗ Applicable Regulations and Standards ∗ Alyeska documents ∗ Employee interviews ∗ Other publicly available documents ∗ Summary of Findings ∗ Recommendations
Scope of Contract 1. Identify assets at the VMT that are subject to remote control and those subject to local control. Include an analysis of the interface between the two classes of assets. 2. Analyze command and control design, capability, visibility and effectiveness in managing normal and abnormal operations at the VMT. 3. Review the training requirements and procedures for OCC operators relative to VMT- specific activities. Evaluate conformance to operating procedures and requirements. 4. Assess how the four key processes at the VMT (Tank Storage, Vessel Loading, Power Generation, and Ballast Water Treatment) are controlled at the VMT and/or the OCC and how operations and system status are coordinated and communicated between the VMT and OCC controllers. 5. Assess the extent to which moving the OCC from the VMT to the Anchorage location changed the limited control capabilities the OCC has for four VMT processes identified in #4 above. 6. Provide Alyeska with recommendations as appropriate for improving the control of assets at the Valdez Marine Terminal during both routine and abnormal operations. 7. Provide reasonable amount of consultation to third parties regarding results, findings, and recommendations associated with work performed under this agreement.
Definition of Key Terms ∗ OCC – Operations Control Center (Anchorage) ∗ SCADA – Supervisory Control and Data Acquisition o DCS – Distributed Control System o PLC – Programmable Logic Controller o HMI – Human Machine Interface o UCOS – User Configurable Open System (TAPS control system software) ∗ Controller – Person who operates system from console ∗ Alarm Management – proper design, analysis, rationalization, implementation, monitoring and improvement of control system alarms
Operations Control Center (Anchorage)
Typical System Review Process • Review applicable regulations, standards and industry best-practice • Review company policies, procedures, requirements, documents and records • Interview knowledgeable employees responsible for implementing procedures, etc. • Validate with documented evidence • Assess company practices and employee understanding against all requirements • Document findings and recommendations
Modified Review Process ∗ Alyeska provided many of their standards, policies and procedures, but provided few documents to demonstrate conformance and very limited employee interviews ∗ Scope extended to consider publicly available Alyeska documents involving control system issues ∗ FERC Tariff Litigation ∗ State Assessment Review Board (SARB) hearings ∗ Evaluate possible VMT control system related issues or similar risks
Applicable Regulations and Standards ∗ Federal Regulations (49 CFR Part 195) DOT/PHMSA Liquid Pipelines. ∗ Same control system for Pipeline and VMT ∗ Industry Standard Practices o API RP 1165 – SCADA Displays o API RP 1167 – SCADA Alarm Management o API RP 1168 – Control Room Management o ANSI/ISA – 18.2 Alarm Management
Alyeska Documents ∗ Many documents requested were deemed “out of scope” by Alyeska and not provided ∗ All documents provided by Alyeska were identified as confidential ∗ Pipeline references were redacted by Alyeska ∗ Many documents (forms) were provided blank ∗ Largely represent policies, procedures and forms exist ∗ Do not provide evidence that policies and procedures are being followed
Employee Interviews ∗ 10-12 Interviews originally planned were subsequently denied ∗ Interviewed one automation engineer and one Valdez technician ∗ All questions were provided to Alyeska in advance of interviews ∗ Individuals were largely unfamiliar with subject matter of questions, deferring responses to others who were not made available ∗ OCC Manager responded to one of nine sets of questions posed. Responses were extremely brief. ∗ Unable to test/validate conformance to standards
Other Publicly Available Documents ∗ FERC TAPS Tariff Case - Pipeline Strategic Reconfiguration (prudence challenge) ∗ Case also addressed issues with implementation of control system changes made since 2007 ∗ Searched public documents re: OCC and control system ∗ Assessed potential for similar issues related to VMT ∗ 2013 State (tax) Assessment Review Board hearings ∗ Project Work Request (PWR) involving critical control system communication building replacement at VMT
Summary of Findings ∗ Some control system documents do not contain sufficient detail ∗ BE-20 (Berth Operators Manual) provides appropriate detail ∗ PS-9 Crude Tank Overfill Incident findings are relevant to control of VMT ∗ VMT Communications Hub / SCADA Building remains in a “failed state” ∗ Key Performance Indicators (KPI) for Control System Alarm Management are inadequate
Summary of Findings (continued) ∗ Alyeska did not demonstrate having completed Documentation and Rationalization (D&R) of the VMT control system alarms as required by regulations and company standards ∗ Response requirements for a berth vapor control building exhaust fan alarm appear inadequate ∗ Status of Alyeska commitment to complete a Reliability Centered Maintenance Review of the Control System is unknown
Key Findings and Related Recommendations ∗ PS09 Tank Overfill Incident root causes and contributing factors included: ∗ Design less than adequate (LTA) ∗ Previous Incident Management Action Plans (MAP) and Lessons Learned LTA ∗ Situational Awareness LTA [OCC Controllers and field personnel] ∗ Safe Operating Committees (SOC) LTA ∗ Standards, Policies and Administrative Controls (Procedures) LTA ∗ To the extent these systemic issues remain, the same factors could contribute to unnecessary risk at VMT. Resolution of these risks needs to be validated by external risk management organization or regulatory agency.
Recommendation ∗ Alyeska has stated publicly they are evaluating Strategic Reconfiguration type of changes to the VMT. ∗ To the extent proposed configuration changes may include a significant increase in process instrumentation and status under the OCC, PWSRCAC should be aware of not only their design, but the specifics of the implementation process, taking into account the type of problems encountered when making the significant changes during the pipeline SR projects
Finding and Recommendation ∗ Alyeska has a Project Work Request (PWR – Z556) in process to replace a critical communication structure at VMT that houses control system hardware ∗ PWR submitted as evidence by Alyeska in 2013 state tax hearings ∗ The original 2008 PWR states the structure is in a “failed state” ∗ Alyeska now says the term “failed state” was in error and will be corrected ∗ PWSRCAC should request a review/update on the status of PWR Z556
Text images from PWR-Z556
VMT Communications Building Houses “critical communications infrastructure” at site of original Operations Control Center
Recommendation ∗ Alyeska should establish clear measureable goals or Key Performance Indicators (KPIs) to track and monitor their control system alarms.
Finding and Recommendation ∗ Alyeska did not demonstrate having completed Documentation and Rationalization (D&R) of the VMT control system alarms as required by regulations and company standards ∗ PWSRCAC should request documentation from Alyeska to establish the status of the Master Alarm Database (MADB) and D&R for VMT control system alarms
Finding and Recommendation ∗ OCC Terminal Controller training is critical to safe operation of VMT ∗ Training records for VMT Controllers were not provided for review ∗ PWSRCAC should request Alyeska provide a status of VMT Controller training
Finding and Recommendation ∗ Alyeska committed to the Joint Pipeline Office (JPO) in 2001-2002 to complete a Reliability Centered Maintenance (RCM) Review on the TAPS Control System ∗ PWSRCAC should ask if Alyeska has completed an RCM analysis of the control system, and if it was updated after the relocation of the OCC from Valdez to Anchorage
Recommend
More recommend
