Evolution of Automated Testing for Enterprise Systems BNSF BNSF - PDF document

P R E S E N T A T I O N Presentation Bios F8 Friday, November 2, 2001 11:15 AM E VOLUTION OF A UTOMATED T ESTING FOR E NTERPRISE S YSTEMS Cherie Coles BNSF Railroad International Conference On Software Testing Analysis & Review October 29-November 2, 2001 San Jose, CA USA

Evolution of Automated Testing for Enterprise Systems BNSF BNSF

Automation Time Line Automated Solutions � Productive Methodology � Flexible � Repeatable � Robust � Reliable � Consistent Time

About BNSF � $8 billion annual revenues � $24 million capital � 33,500 miles of track, 29 states & 2 Canadian provinces � 45,000 employees across the western United States and Canada � 5000 locomotives � 200,000 freight cars

About BNSF - IT Architecture � Largest sysplexed IBM 9X2 mainframe network � Over 2,700 MIPS (millions of instructions per second) � Over 300 AIX and NT servers � 10,000 terminals � 15,000 PC devices � Telecom Network equivalent to largest phone companies � Over 100 Line-of-Business applications

About BNSF Security Analysts � “...the most advanced and sophisticated railroad technology…stock price up 20% from current levels.” � “…very impressed with the company’s use of technology, which allows it to do more with less…the company is now in a position to reap the benefits of the investment.” � “In short, the technology demonstrations increased our confidence in the company’s ability to continue to reduce costs and improve productivity…we believe BNI will continue to increase productivity…through leveraging it’s sophisticated technology.”

Application Interface Core Business Customer • Trains & Track • Safety • Movement • Pricing • Equipment • Logistics • Equipment • E-commerce • Crews • Maintenance • Relationship QA Business • Financial • Human Resources • Government Reporting • Labor Relations

The QA Team � 16 Team Members � Over 300 years business experience combined � One member with development background � All have business knowledge, technical aptitude, and motivation Good Tools + Great People = Efficient and Effective Product Evaluation

What We Test � Main Frame Core business systems � Web / JAVA - e-Commerce, Customer Interface � Client / Server - Customer, Logistics, Yard Activity � Infrastructure - Load, Performance

Testing Infrastructure � NT Integrated Test Lab (isolated domain) - 1998 � Mainframe-Controlled Test Environment - 2001 � Mid-tier (AIX) Server (4th quarter 2001)

QA STATS Manual Testing � Monthly Average YTD: � 350 Tests Completed � 200 Packages � 1100 Modules

QA STATS Automated Testing � 300 timekeeping test scripts run each night (equivalent to 8 staff days manually) � 25,000 voice files converted in one week (equivalent to 3 staff months manually) � Database performance test takes 4 hours set up and run (equivalent to “an army” of people banging on their keyboards manually) � Automated test data set up takes 1 hour (equivalent to 1 or more days manually)

Benefits � Increased productivity for TS and Customers � Better resource utilization - unattended after hours of testing � Early involvement, smoother implementation, decreased cost � Greater knowledge transfer

The Journey Leveraging automation Fully automated and unattended Application-driven - Object Oriented Data-driven - Dynamic SQL Data-driven - Spreadsheets Hard-coded scripts for specific use Record-and-playback Manual

Record-and-Playback: Consistent “Macros” Y2K Script 1 Y2K Script 2 Y2K Script 3 Type Car Init1 Type Car Init2 Type Car Init3 Type Car No1 Type Car No2 Type Car No3 Type Track1 Type Track2 Type Track3 Type Station1 Type Station2 Type Station3 Type Waybill1 Type Waybill2 Type Waybill3 Type Waybill1a Type Waybill2a Type Waybill3a Type Waybill1b Type Waybill2b Type Waybill3b Repeatable Application Tests

Hard-Coded Scripts Test Script 1 Test Script 2 Test Script 3 Type Car Init1 Type Car Init2 Type Car Init3 Type Car No1 Type Car No2 Type Car No3 Type Track1 Type Track2 Type Track3 Type Station1 Type Station2 Type Station3 Type Waybill1 Type Waybill2 Type Waybill3 Type Waybill1a Type Waybill2a Type Waybill3a Type Waybill1b Type Waybill2b Type Waybill3b Repeatable Scenarios

Automated Solutions Data-Driven 100 Test Cases with Required Data Elements Application-specific Test Script Works with all Test Cases 120 100 80 Enter TestData 60 Type {“RtCtrl”} Type CarInit 40 Type CarNo 20 Type Track 0 Type Station Cars Tracks Stations Waybills Type Waybill1 Type Waybill2 Type Waybill3 Application

Automated Solutions OO Testing Methodology Test Case Scenarios Robust Test Script works with all Test Case Application A Scenarios across applications 120 Enter TestData 100 Test Case Scenarios 80 Type {“RtCtrl”} 60 Application C 40 Type CarInit 20 0 Type CarNo 120 Cars Tracks Stations Waybills 100 Type Track 80 60 Type Station Test Case Scenarios 40 Type Waybill1 20 Application B 0 Type Waybill2 Cars Tracks Stations Waybills 120 Type Waybill3 100 Test Case Scenarios 80 60 Application D 40 20 0 120 Cars Tracks Stations Waybills 100 80 60 40 Applications 20 0 A, B, C, D…. Cars Tracks Stations Waybills

Automated Solutions Data Base (On the Horizon) Test Case Test Script Scenario Database: Database: Applications SQL Applications A, B, C, D... A, B, C, D... SQL Application Applications Database A, B, C, D...

Keys to Automation Success - Have a Vision! Find a business driver � Develop team partnership � Stabilize application design � Access to business knowledge � Recognize and plan for learning curve � Think out-of-the-box � Match business knowledge with technical skill � Continue skill development � Build from the diversity of the team �

Evolution of Automated Testing for Enterprise Systems To be continued... BNSF BNSF

Mike Amos Mike Amos is a QA Engineer for BNSF. Mike has held many positions on the Canadian National and BNSF Railroads. Currently Mike is Technology Services, Project Manager of Quality Assurance for Major Application Development Projects at BNSF. mike.amos@bnsf.com Pete Sidoti Pete Sidoti is a QA Engineer for BNSF. Pete has had many roles in 20 years with the BNSF from civil engineering to information technology. The last 5 years have been in the QA field. Pete received his MBA from Baker University in 1986. Pete is Project Manager of Technology Services, QA Infrastructure for BNSF. pete.sidoti@bnsf.com Cherie Coles Cherie Coles is manager for BNSF Technology Services, Quality Assurance. She has ten years experience in IT working in Infrastructure Engineering and Application Development. Prior to her career with BNSF she graduated from University of Texas at Arlington, with a degree in Business Administration. She earned her CPA while at ARCO Oil and Gas Company in the Systems Management Organization. cherie.coles@bnsf.com