Procedures for FOD Detection System Performance Assessments Radar-Based and Dual Sensor Systems Beth Woodworth Edwin Herricks University of Illinois
Introduction • I will be talking today about two performance assessment efforts. • The initial effort began in 2004 and was completed with assessment of an installed system on a runway at PVD in 2008. • The second began in 2007 and was completed in 2009.
Sensors Evaluated • Qinetiq Tarsier TM – Radar based system – Installed at PVD • Xsight FODetect TM – Hybrid system – Installed at BOS
Testing of QinetiQ Tarsier TM at PVD • 94Ghz radar • Detects FOD at ranges up to 1 km • 2 sensor units placed on towers scanned Runway 5/23 at PVD • Primary performance criterion was a standard -20dBm 2 target at a range of 1km
Testing of FODetect TM at BOS • 74GHz radar and video camera with image processing • Sensors replaced edge lights • Partial installation at BOS • Sensors covered 550 ft on Runway 15R • 10 surface detection units (SDUs) – 5 SDUs on each side of the runway • Each sensor covered approximately 75ft x 200ft • Primary performance criterion detection of 0.8 in (2 cm) object
General Assessment Protocol • Testing scheduled over approximately 12-month period to capture varied weather conditions – PVD – June 2007- March 2008 – BOS – June 2008 - March 2009 • Targets selected to challenge each of the systems and included • Different sizes • Different shapes • Different materials • Different hazard potentials
Detecting a Target • Radar – Reflectivity – energy returned to sensor – “Complex Scatterers” – most FOD has a variety of edges which have different likelihoods of being detected. • Electro-Optical – Background – Item surface conditions • Color, Flat/Glossy – Illumination – • Lighting and shadow
Overall Goals of Testing • Calibration – Standard Target Testing – Uses items of known detection characteristics – Provides information on system reliability and robustness • Performance – Group of FOD items with different hazard potentials – Repeatedly placed at known locations in defined positions • Blind – Random placement and actual and simulated FOD items – This test most represents typical detection needs at an airport – Challenges system to detect a diverse and unpredictable set of items.
General Methods • Safety – – Because assessments were conducted at operational airports, safety considerations were paramount. – Personnel were trained and briefed regularly by airport operations personnel. – Although runways were closed, safety areas were identified and drills held so that all assessment personnel were aware of their environment. – Rigorous accounting was made of all experimental FOD taken on the airport.
General Methods • FOD log – Over 1000 items placed and retrieved! – all items were written in a log before proceeding to the runway – after return and often during the testing the log was checked again the items in the box • FOD box – items put in boxes with dividers in order to be easily and quickly counted • Marking – Used for calibration and performance testing – Small, nearly invisible UV marks were placed on the runway – UV lights used to find markings – Locations were surveyed using a differential GPS accurate to millimeters in the X/Y plane – Surveyed locations were compared with locations provided for each detection by each technology
General Methods • Radar – Does NOT depend on illumination – All assessment campaigns at PVD were performed during normal 12 pm to 5 am runway closures • Electro-Optical – Did depend on illumination. – Testing at BOS was on Runway 15R which saw limited use allowing testing at different times. – Tests were performed at BOS during the day, at night, and at dawn and dusk to capture a range of lighting conditions.
Calibration – Standard Target Testing • GOAL = confirm detection performance using objects with defined detection characteristics • Targets selected based on technology – Metal cylinders 3in X 4.5in 1.25in with a nominal reflectivity of 0dBm 2 (Large) – Metal cylinders 2in X 2.5in with a nominal reflectivity of - 10dBm 2 (Medium) – Metal cylinders 1.5 in x 1.25in with a nominal reflectivity of -20dBm 2 (Small) – Metal 2in spheres – Colored PVC cylinders 1.5in X 1.25in colored grey, white and black
Calibration Targets Metal cylinders Metal cylinders Metal cylinders 1.5in x 1.25in 2in X 2.5in 3in X 4.5in 1.25in -20dBm 2 -10dBm 2 0dBm 2 (Large) (Small) (Medium) Metal spheres 2in Colored PVC cylinders 1.5in X 1.25in
Calibration- Standard Target Testing for Tarsier TM • 4 types of metal targets • 6 transects along full length of the runway • Transects 2, 3, and 4 within 1km of both radars • Transects 1, 5, and 6 used for system assessments Distance Distance from S Transec Distance Distance from S Radar 2 t from (m) from (ft) Radar 2 (m) (ft) 1 1675 5494 555 1821 2 1268 4160 220 723 3 864 2834 356 1168 4 502 1647 708 2322 5 204 668 1140 3738 6 443 1454 1547 5075
Calibration for FODetect TM • Metal cylinders 1.5in x 1.25in tested the radar portion of the sensor • 3 colors of PVC cylinders (white, grey, black) and small metal cylinder • Targets placed at varying distances from sensor in a test rectangle (approx. 150ft x 70ft) • Targets placed to test the system and a single sensor • Standard target array included 1 metal, 1 white, 1 grey, and 1 black cylinder • Groups of targets oriented differently
Performance Testing • Items not selected based on sensor type as with calibration testing • Used examples of common items found at airports • Selection of items based on frequency of occurrence and hazard potential • Calibration items also used during performance testing • Items placed in know positions at specific angles
Table of Standard FOD Items Used in Performance Assessments FOD Item a Expected Hazard Frequency of Occurrence 1. Small Piece of Concrete High Common 2. Standard Lug Nut From High Common Service Vehicle 3. Roller Bearing High Common 4. Chunk of Rubber Low Common 5. Mechanics Wrench High Common 6. Fuel Cap High Common 7. Cotter Key Moderate Common 8. Plastic Bottle/Bottle Cap Low Common 9. Strapping Material Moderate Common 10. Expansion Joint Low Common Material 11. Construction Material – Moderate Based on Construction Galvanized Nails or Activity Sheetrock Screws 12. Runway Infrastructure High Uncommon Part – Piece of Runway Light or Signage 13. Small Fasteners Moderate Common 14. Metal Strip High Uncommon 15. Fiberglass Door Moderate Common 16. Asphalt Chunk High Common This table contains the items selected by CEAT
Performance Testing Methods at PVD • 6 arrays related to standard target transects • In array 5x5 grid established, approximately 25ft between items • 23 targets total per array (no items set on runway lights) • Position of items varied between campaigns (random selection of location achieved in test planning); items in the same position in all arrays in any single campaign. • All items rotated 45 degrees after each scan to cover 8 cardinal points of the compass
Performance Testing Methods at BOS • Line established midway between the centerline and the edge line • 5 identical performance items placed at equal distances on this line across the length of test rectangle • Items rotated so detections at 0, 45, and 90 degrees relative center line recorded
Blind Testing • Used actual FOD items collected from runways Blind Testing • Range of sizes, materials, Grid at PVD colors • Over 100 items in collection • 30 items randomly selected for each campaign • Placement locations chosen at random from a grid • Items dropped or tossed so the orientation was random • Up to 10 items used at one time.
Blind Testing Grid of Locations at BOS Typical Grouping of FOD Blind Items
Inclement Weather Testing • Objective to evaluate sensor performance in variable weather conditions • Opportunistic based on long term plan, but scheduled adjusted to coincide with storm events. • Access to the runways in snow was impossible so assessments performed just after snow emergency conditions
Inclement Weather Assessment - Rain • PVD - Tarsier TM radar system • October 2007 • Calibration targets were deployed as a rain squall moved across the runway • BOS – FODetect TM hybrid system • March 2008 • One light rain event at dusk
Inclement Weather Assessment - Snow • PVD – Tarsier TM radar system – Assessments after 2 snow events • Jan. 24, 2008 – flurries but no accumulation • Feb. 12, 2008 – snow changing to sleet – Time for placement and retrieval were limited due to needs to clear the runway
Inclement Weather Assessment - Snow • BOS – FODetect TM hybrid system – Winter Storm Jan. 7-12 – • Weather Conditions: – rain, freezing rain, freezing drizzle, mist, ice pellets, fog, snow – Totaling 2.1in of wet precipitation • Testing took place on Jan 8, 9, and 13, 2009 – Testing on Jan. 29 and 30 after snow events as well
Recommend
More recommend
