Vision-based Weed vs Crop Discrimination for Selective Weeding The - PowerPoint PPT Presentation

Vision-based Weed vs Crop Discrimination for Selective Weeding The UK Onion & Carrot Conference & Trade Exhibition Petra Bosilj University of Lincoln, UK L-CAS Notthingham, November 14 th 2017 Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 1 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal), classification (identifying the plant type) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal), classification (identifying the plant type) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal), classification (identifying the plant type) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal), classification (identifying the plant type) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal), classification (identifying the plant type) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal), classification (identifying the plant type) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal), classification (identifying the plant type) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Motivation project by: University of Lincoln, Garford Farm Machinery Ltd. (sponsored by Innovate UK and BBSRC) locate and recognize type of vegetation in the image segmentation (soil removal), classification (identifying the plant type) selective treatment (spraying) of weeds vision-guided robotic hoeing (mechanical) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 2 / 12

Outline Motivation and goals 1 Our approach 2 Data collection and preparation Segmentation Classification Conclusions 3 Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 3 / 12

Our approach Image Processing (attribute morphology): analysing and selecting contrasted objects in the image based on their characteristics: shape, texture, color, neighbourhood local processing, robust to different lighting Machine Learning techniques: determining the type of plant Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 4 / 12

Data collection two-camera system: near infra-red (NIR) + color Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 5 / 12

Data collection two-camera system: near infra-red (NIR) + color mounted on a manually operated setup Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 5 / 12

Data collection two-camera system: near infra-red (NIR) + color mounted on a manually operated setup (for now) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 5 / 12

Data collection two-camera system: near infra-red (NIR) + color mounted on a manually operated setup (for now) Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 5 / 12

Data collection - examples examples of carrot and onion fields in different stages of growth Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 6 / 12

Data collection - examples examples of carrot and onion fields in different stages of growth Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 6 / 12

Data collection - examples examples of carrot and onion fields in different stages of growth Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 6 / 12

Data collection - examples examples of carrot and onion fields in different stages of growth carrots Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 6 / 12

Data collection - examples examples of carrot and onion fields in different stages of growth carrots Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 6 / 12

Data collection - examples examples of carrot and onion fields in different stages of growth carrots Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 6 / 12

Data collection - examples examples of carrot and onion fields in different stages of growth carrots onions Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 6 / 12

Data collection - examples examples of carrot and onion fields in different stages of growth carrots onions Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 6 / 12

Data preparation NIR + color image registration: image alignment Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 7 / 12

Data preparation NIR + color image registration: image alignment Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 7 / 12

Data preparation NIR + color image registration: image alignment calculating NDVI (normalized difference vegetation index) image Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 7 / 12

Vegetation segmentation typical: threshold determined globally based on image histogram, gradients poor performance on textured background, noise removal required proposed approach: locally selected regions clean patch of onions Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 8 / 12

Vegetation segmentation typical: threshold determined globally based on image histogram, gradients poor performance on textured background, noise removal required proposed approach: locally selected regions onions with weeds Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 8 / 12

Vegetation segmentation typical: threshold determined globally based on image histogram, gradients poor performance on textured background, noise removal required proposed approach: locally selected regions GROUND TRUTH OTSU RATS MAX-TREE Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 8 / 12

Vegetation segmentation typical: threshold determined globally based on image histogram, gradients poor performance on textured background, noise removal required proposed approach: locally selected regions GROUND TRUTH OTSU RATS MAX-TREE Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 8 / 12

Vegetation segmentation typical: threshold determined globally based on image histogram, gradients poor performance on textured background, noise removal required proposed approach: locally selected regions good performance on low-content images artificial low vegetation example Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 8 / 12

Vegetation segmentation typical: threshold determined globally based on image histogram, gradients poor performance on textured background, noise removal required proposed approach: locally selected regions good performance on low-content images GROUND TRUTH OTSU RATS MAX-TREE Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 8 / 12

Grouping into objects The segmentation output is organized into a list of separate objects Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 9 / 12

Grouping into objects The segmentation output is organized into a list of separate objects Objects typically correspond to one plant or a few overlapping plants Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 9 / 12

Classification Object-based classifier applied to the list of regions Regions classified as crop, weed and mixed (for overlapping objects) Distinguishing features: shape (circularity, elongation), texture (color variance) GROUND TRUTH CLASSIFICATION OUTPUT Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 10 / 12

Classification Object-based classifier applied to the list of regions Regions classified as crop, weed and mixed (for overlapping objects) Distinguishing features: shape (circularity, elongation), texture (color variance) SVM (Support Vector Machines) and RF (random forest) GROUND TRUTH CLASSIFICATION OUTPUT Notthingham, November 14 th 2017 Petra Bosilj (UoL, L-CAS) 10 / 12