Aquatic Identification and Sorting System SENIOR PROJECT PROPOSAL - PowerPoint PPT Presentation

Aquatic Identification and Sorting System SENIOR PROJECT PROPOSAL PRESENTATION JOHN SCHULZ AND REINER LINTAG ADVISOR: DR. IMTIAZ DECEMBER 4 TH , 2018

Agenda • Background • Project Objectives • System Functions • Design • Results • Parts • Division of Labor • Schedule • Literature Search • References 2 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Problem Statement • Modular fish sorting system • Invasive species • Avoid premature fish harvesting. 3 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Background TWO MAJOR COMMERCIAL USES: Governments Aquaculture • Inspired by Asian Carp in the Illinois River. • $50 billion dollars a year loss from inefficient harvesting practices [1] . • Army Corps of Engineering estimate $15 to $18 billion to be spent to stop Asian Carp [2] . • Aquaculture in 2012 was a $128 billion dollar industry [1]. • DNA of Asian Carps already found in the Great Lakes [3]. 4 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Objectives • Scalable modular system • Fishing lures used for humane practice What is it? • Ensure system safety electrically • Transportable and operate off grid operation What size is it? • AI specie classification • Aid in invasive specie detection and removal efforts. • Harvest fish based on size vs. bulk harvesting. Will a sort be possible? 5 AQUATIC IDENTIFICATION AND SORTING SYSTEM

System Functions • Distinguish between different fish/creatures (lures) • Sorting based on AI decision from camera • Off grid operation • Data collection system 6 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Hardware Design Camera FPGA: PYNQ IOT Sorting Servo Battery Powered Data Cache 7 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Prototype System Design Training Images DNN FPGA Sort DNN Camera Power Region Here System Prototype 8 AQUATIC IDENTIFICATION AND SORTING SYSTEM

System Prototype • Pump controls water flow from left to right • Open output pipe for gushing design • Modern lures swim when water passes over them. • 3 different lengths of output pipe to suite different lures’ swimming types. Figure 1 “Current Simulation” 9 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Simulation Tank Figure 2 “Tank Parts” Figure 3 “Assembled Tank” Pump Pump Output Pipe Controller 10 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Tank Build • Size: 18” x 26” x 15” • Capacity: 4 ¾ Gallons • Material: food grade polycarbonate • Crystal-clear • Sealable lid • Custom Additions • External reflection shielding • Aquarium grade led lighting strip Figure 4 “Tank Design” 11 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Lure Mount Points Horizontal Mount Angle Mount Output Pipe Top Mount 14”/17”/20” • Figure 6 “3/4” Piping” Figure 5 “Mount Points” 12 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Lure Design Attachment Point Ball Bearing Weight Chamber Treble Hooks Leveling Fin Figure 7 “Lure Design” 13 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Imaging Here Training DNN FPGA Images Sort DNN Camera Power Region System Prototype 14 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Imaging Angle • Subject shifted left or right to remove pipe reflections. Output Pipe Center • Subject imaged at ±15° horizontally and Intake Pipe vertically • Pseudo 3-D mapping of subject • Assuming subject is symmetrical if flipped +15° -15° 0° Figure 8 “Imaging Angle” 15 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Database • 20 Different fish classes Database size: • Minimum 14 minutes recorded per class 1.6 million • 4 Water clarity levels • 3 Imaging angles captured • 4 min. -15 ° Images total, created in house. • 6 min. 0° • 4 min. +15° • 3 Database duplication techniques • Gaussian noise • Flip left right • Flip up down 16 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Image Intensity • Image Intensity – the number of levels of color accuracy provided for each color • False Contouring – the appearance of lines of color that exist due to a lack of image intensity. Figure 9 “8 - Bit Color” Figure 11 “3 - Bit Color” Figure 12 “4 - Bit Color” Figure 10 “2 - Bit Color” 18 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Imaging • Video captured at 720p 60 fps • Image processing: • 12-bit color depth, 4-bits per RGB pixel • Video processing and bit slicing done in Matlab • Neural network trained on 12-bit images • Advantages • Minimize resource usage in FPGA • Clear enough • Disadvantages • Minor false contouring Figure 13 “Processing Example” 19 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Neural Network Here Training DNN FPGA Images Sort DNN Camera Power Region System Prototype 20 AQUATIC IDENTIFICATION AND SORTING SYSTEM

AI Classes A B • 2 different species in different sizes • Similar species of the same size • Species design to test dynamic range • Classes Include: • Fish: Trout, Blue Gill, Minnow, Bass Figure 14 “Test Classes of Color Difference ” • Worms • Mice C D • Frogs Figure 15 “Test Classes of Different Sizes” 21 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Neural Net Layout Figure 16 “AI Hierarchy” AlexNet structure [4][6] • • Space efficient on FPGAs Designed for quick response time • Initially tested on 4 classes • 22 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Training to FPGA Imaging GPU Export Model VHDL Code Gen Training • Building • Matlab ML • Weights • Generate Database of Toolbox Neural Net • Structure Classes code for FPGA • .onnx file in Matlab standard Figure 17 “Training Flowchart” 23 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Why FPGA? Latency! Inference Performance 35,000 • GPU 30,000 • High throughput or low latency Images/Second • Widely supported 25,000 • Power hungry 20,000 • Large Size 15,000 • FPGA 10,000 • High throughput and low latency 5,000 • Best for high frame rate inferencing 0 • 16x Performance/Watt , vs GPU GPU GPU GPU FPGA FPGA 2016 2017 2018 2018 2019 Source: Xilinx Developers Conference 2018 [4] 24 AQUATIC IDENTIFICATION AND SORTING SYSTEM

FPGA • PYNQ Development Board • Xilinx ZYNQ XC7Z020 FPGA HDMI • Logic Cells: 101,440 • DSP Blocks: 220 • Ex. Ram: 512 Mb DDR3 • Block Ram: 4.9 Mb • 100 MHz FPGA Clock • Video: HDMI In and Out • Dual-core 650 MHz Cortex-A9 • Max: 36 W Field Programmable Gate Array (FPGA) – Array of reprogrammable logic gates, used for high throughput tasks. 25 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Results 26 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Results: Simulation Tank • 3 Lure Mount Points • Magnetically mounted from top • Angle mounted to the output pipe • Horizontal mounted to the output pipe • Output Pipe • 14”, 17”, 20” lengths • Reflection Shielding • Hefty trash bags taped externally to reduce light bleed and reflections • Electronics • All electronics are external of the sealed container Figure 18 “Working Simulation Tank” 27 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Video 28 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Database Size • All 20 Classes Class: F2_FROG Class: F1_XRAP 5 ½ Hours Recorded • Current size of processed database (4 Classes): Class: F3_MICE Class: F4_XRAP 103k Images > 16k per training class 200x150 Res. Figure 19 “Processed Classes” 29 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Training Precision • Video Card: GTX 1060 6GB • Training Accuracy: FP16 • FPGA Accuracy: FP12 (“FIX_12_8”) Figure 20 “Tested FPGA Accuracy” 30 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Neural Network • Optimizer: Stochastic Gradient Descent (SGD) Method Figure 21 “Training Accuracy” 31 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Neural Network • Loss Function: Mean Square Error (MSE) Figure 22 “Training Loss” 32 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Prediction Figure 23 “Prediction Accuracy” 33 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Power Diagram Here Training DNN FPGA Images Sort DNN Camera Power Region System Prototype 34 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Power Distribution • Currently using grid power for the system • Implement a maneuverable power distribution system • Portable Power Station • Car Battery • Solar Panel Figure 24 “Inverter” Figure 25 “Solar Panel” 35 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Power System Power Consumption • Xilinx ZYNQ XC7Z020 FPGA – 36 [W] • Jebao DCP Water Pump – 22 [W] to 23 [W] • Mingdak LED Aquarium Light – 4 [W] System Total Wattage = 62 [W] to 63 [W] Calculation: Low = FPGA + Water Pump (Low) + LED High = FPGA + Water Pump (High) + LED 36 AQUATIC IDENTIFICATION AND SORTING SYSTEM

Parts Tank components • Jebao DCP – 2500 Water Pump • Cambro – Lid • Cambro – Container Figure 26 “Pump” • PVC Pipe ¾” • Mingdak – B00X84LQ5S Top Light • Hefty – Black Garbage Bags Figure 27 “Tank Light” 37 AQUATIC IDENTIFICATION AND SORTING SYSTEM