Building Automation Systems (BAS) Network Access Procedure and Summary of Events on June 23, 2018 Matthew Berkheiser, DrPH, CIH, CSP Associate Vice President and Chief Safety Officer Environmental Health & Safety, Sustainability and Emergency Management
Contents 1 MD ANDERSON CANCER CENTER 2 WHAT IS THE BUILDING AUTOMATION SYSTEM (BAS)? 3 EVENT OF JUNE 23, 2018 Response Cause Process Changes
Our History Monroe Dunaway Anderson created a charitable foundation in 1936. After his death, the MD Anderson Foundation provided funding and land to build a cancer hospital to serve the citizens of Texas.
Our Beginnings Army Barracks were moved to the Baker family estate to Nurse helping a patient into one of the converted barracks Laboratory at the Baker estate, 1944 Waiting room at the Baker estate, 1944 Baker estate,1942 act as inpatient wards and an operating room on the Baker estate, 1951
Our Leadership Ernst W. Bertner, M.D. Randolph Lee Clark, M.D. Acting Director, 1942-1946 President, 1946-1978 Charles A. LeMaistre, M.D. John Mendelsohn, M.D. Ronald DePinho, M.D. Peter Pisters, M.D. President, 1978-1996 President, 1996-2011 President, 2011-2017 President, 2017
Our Mission To eliminate cancer in Texas, the nation and the world through outstanding programs that integrate patient care, research and prevention, and through education for undergraduate and graduate students, trainees, professionals, employees and the public.
Our Vision We shall be the premier cancer center in the world, based on the excellence of our people, our research-driven patient care and our science.
Our Core Values • • Caring Integrity Discovery
Patient Care • FY17 • More than 137,000 people sought the superior care that has made MD Anderson so widely respected — over 41,000 of whom were new patients. • MD Anderson’s cancer clinical trial program is one of the largest of its kind. Almost 9,400 participants were enrolled in 1,250+ clinical trials exploring innovative treatments.
Patient Care • FY17 Hospital admissions 28,793 Average number of inpatient beds 681 Outpatient clinic visits, treatments and procedures 1,441,403 Pathology/laboratory medicine procedures 12,700,333 Diagnostic imaging procedures 574,018 Surgery hours 70,460 Active clinical research protocols 1,255
Research
Research • FY17 • MD Anderson invested more than $844 million in research. • Every day, in our Texas Medical Center labs and our research facilities in Bastrop County, scientists are making discoveries that are rapidly being translated into clinical care.
Research • FY17 • Our Moon Shots Program™ has 13 transdisciplinary teams working to dramatically accelerate the conversion of scientific discoveries into clinical advances and significantly reduce deaths caused by the disease. • 150 clinical studies are being accelerated as a result of the Moon Shots Program. • MD Anderson ranks first in the number of research grants awarded by the National Cancer Institute.
Our People
Our People • More than 19,530 employees, including 1,743 faculty. • A volunteer workforce composed of 1,141 on-site, trained volunteers and 2,091 off-site, trained myCancerConnection survivor volunteers contributed 122,367 hours of service in FY17.
Locations
Houston area
Building Automation System • Building life safety and critical 619,000 points and 20,000 systems monitoring and alarms controllers • Data centers and IT room • Operating room temperature temperature, air flow controls, and air pressure power monitoring • Patient satisfaction and their • Client critical equipment room comfort monitoring • Pharmacy USP 800 compliance • Integration with patient scheduling (EPIC) • Chilled water control systems supporting equipment cooling (MRIs) • Building water systems
Systems Integration Support
Replication Event On June 23, 2018, the MDA BAS team was contacted by the Patient Care Facilities O&M team concerning the abnormal operation of some of their facility chilled water and air handling equipment. • Equipment was cycling on and off in the patient care buildings • Graphics were displaying points randomly going in and out of failure These events caused the O&M teams to place our air handling system in manual mode. We identified that the system was experiencing symptoms of data corruption, which is also commonly referred to as replication.
Replication Event -Impacts • Exterior temperatures ranged from 77 to 90 degrees F and 84% humidity • Approximately 500 patients in beds • On Saturdays we see approximately 400 out-patients for treatment • Pressure issues in some pharmacy areas caused us to stop/relocate pharmacy processes • Elevated temperature and humidity inside patient care buildings • 70 % humidity in operating suite with no cases scheduled
Response- Activate Incident Command
Response- Activate Incident Command
Response- Downtime Procedures
Cause During construction of the League City project, a BAS vendor was using their company’s proprietary application on their laptop. The laptop had been used in the past to work on both our research building databases and our patient care building databases. We believe data from our Research application was inadvertently copied to an image on his laptop and eventually copied to the Patient Care system causing a data mismatch and replication issue. Database 1 Database 2 Corrupt Database
BAS Vendor Network Access Procedure Changes • Vendor owned computer equipment (laptops, iPads, etc.) will no longer be allowed to access MD Anderson’s building controllers via direct connection or by utilizing the internal network via Ethernet or internal Wi-Fi. • Vetted users utilizing their vendor provided computer or laptop can connect to MD Anderson assets by utilizing MD Anderson’s Virtual Private Network (VPN) solution when connected on an external network.
BAS Vendor Network Access Procedure Changes • BAS vendors shall not be allowed to utilize any proprietary software tools on MD Anderson network. • Our internal BAS team has developed a script that runs every 12 hours verifying we do not have replication occurring on the system.
BAS Vendor Network Access Procedure Changes New Building Construction: • Vendor shall be responsible for providing a temporary local network to meet the construction schedule or utilize the internal network when available. • Vendor shall submit a detailed database merging plan thirty days prior to substantial completion for owner review and approval. • Once the submitted database merging plan is approved by MDA BAS, we will merge the local construction database with the institution database.
BAS Vendor Network Access Procedure Changes Existing Building Addition and Renovation: • Vendor will submit an outage request via the Facilities Outage Notification (FON) 5 business days before the system modification. • MDA BAS must approve the request prior to the new building controller being configured and added to the network. • Vendor shall not connect the building controller online without MDA BAS supervision being physically present at the site location.
Next Steps/Action Items • IT Engineering to review and approve plans. Completed • FM Sr. Staff to review and approve BAS Network Access Procedure. Completed • Incorporate recommendations from MDA InfoSecurity on additional recommendations to harden the BAS system. • Plan to turn replication back on in the patient care system. • Due to the events of June 23 caused by the replication event we plan to implement a financial penalty through the construction contract process.
Questions
Recommend
More recommend
