performance evaluation of autonomous weeding robots FIRA 2019 Rmi - PowerPoint PPT Presentation

performance evaluation of autonomous weeding robots FIRA 2019 Rémi Régnier (LNE), remi.regnier@lne.fr

 Goal : encourage the development of autonomous innovative solutions for intra-row weed control in field crops with wide spacing and vegetable crops in order to reduce by 50% the use of phytosanitary products , and thus contribute to the achievement of the objectives of the Ecophyto II plan. Inter-row Intra-row Crops The ROSE challenge goal 2

4 projects funded Develop solutions BIPBIP WeedElec Participants Contribute to the definition of the scientific and technological PEAD ROSEAU objectives of the challenge Leads the definition of competition objectives and ensures that they are measurable Operational organizer (trust third party) Organizes and leads the challenge Ensures fair treatment of participants Finance the challenge Funding body Statue on the objectives of the challenge Challenge participants

Launch of the first evaluation campaign 06/2019 Meeting to present the results of the dry- run Opening of the ROSE challenge 06/2019 01/01/2018 Launch of the second evaluation Challenge kick-off meeting campaign 02/28/2018 01/2020 Validation meeting of the evaluation plan - Launch of the third evaluation Launch of the dry-run campaign campaign 06/05/2018 06/2020 12/31/2021 2018 2021 Jan. May Sept. 2019 May Sept. 2020 May Sept. 2021 May Sept. 10/2018 05/2020 05/2021 Field meeting 1 - Dry-run Field meeting Field meeting 10/2019 Field meeting 05/2019 Field meeting 2 - Dry-run Dry-run campaign 06/2018 - 06/2019 First evaluation campaign 06/2019 - 01/2020 Second evaluation campaign 01/2020 - 06/2020 Third evaluation campaign 06/2020 - 07/2021 The macro planning of the challenge

Four evaluation campaigns Operational Six meetings in the experimental field organization An area of four hectares dedicated to experiments Organisation opérationnelle du challenge ROSE Operational organization of the ROSE challenge

AgroTechnoPôle site : Irstea Montoldre Plot challenge ROSE Irstea Montoldre Site Fields meetings

Detection • Detect and identify plants • Decide on the action to be taken Decision • Carry out the weeding action Action Three key steps to evaluate

Types of crops planted : Types of weeds planted: • large crop with wide spacing: maize spread out (horizontal): (row spacing 75 to 80 cm, foot spacing spread out (horizontal) : 14 cm) o Model weeds : mustard • field vegetable crops: beans (row o Natural weeds : matricaria . spacing 15 to 30 cm, foot spacing 3 to 8 with upright (vertical) : cm) o Model weeds : ray grass o Natural weeds : goosefoot . Maize Beans Mustard Matricaria Goosefoot Crops and weeds

Prototype presented by Pead in September 2019 Prototype presented by BIPBIP in September 2019 Détection Prototype presented by ROSEAU in September 2019 Prototype presented by Weedelec in September 2019 Detection evaluation

Participant Camera Light Resolution Surface d θ α β r RGB Artificial 5 Megapixels 45cm*5 40cm 0° 1 (DEL) (5 pixels/mm) 5 cm Visible + Natural 50 cm 60° 0° 0° 2 hyperspectral (Carbon Bee) RGB + Infrared Natural 1024*768 2m*1.3 1.3 m 0° 3 pixels m RGB Natural 5 Megapixels 25° 4 (night (1.5mm/pixel) excluded)  References Provision of Metrics services Error analysis and Definition of the comparative performance of the data evaluation task between hypothesis estimation and test and references environments  Hypothesis Four technologies for one evaluation

Comparaison References : manual annotations 1. Mapping 2. Calculation of the error rate Plant of Acquisition of images by interest the 4 evaluated robots Hypothesis : outputs from detection systems Objective: determine the position of weeds and/or plants of interest on the images Detection evaluation

Metric Evaluation via the EGER metric: 𝑂 𝐹𝐻𝐹𝑆 = 𝐷 𝑙 + 𝐺𝐵 𝑙 + 𝑃 𝑙 𝑙=1 𝑂 𝑂𝑆 𝑙 𝑙=1 𝐷 𝑙 : costs of confusion on the image k 𝐺𝐵 𝑙 : false alarm costs on the image k 𝑃 𝑙 : costs of forgetting on the image k 𝑂𝑆 𝑙 : number of plants detected in the reference (weeds and plants of interest) Detection evaluation

Development and use of the DIANNE software

Next steps : • January 2020: presentation of the results of the first campaign • Presentation of the results of the first campaign • Availability of the four annotated databases during 2020 (250 images with minimum annotations per technology). • New evaluation in June 2020 Possibility to use the parcels for image acquisition on request from IRSTEA Montoldre To follow the progress of the challenge : http://challenge-rose.fr/ ROSE Challenge

Thank you for your attention

Influencing factors Controllability Robustness test Measurements made Weather (rain, wind,...) No No Daily measurements by weather station Brightness No - During the image- Measurements by based detection task luxmeters when Agro- - During the field participants pass through pedoclimatic detection task Soil moisture content, No No Daily measurements by conditions temperature, useful water ground probes reserve Clay rate measurement Yes (constant) No Measurement before the first meeting Described before the Technical itinerary Yes (constant) No start of the campaigns - During the field Crop density and detection task Taking pictures before Test mode Yes distribution - During weeding each meeting tasks - When detecting on Stage of plant development No Daily image capture the image database Global influencing factors

Title Bloc-outil et Imagerie de Perception Et binage RObotics SEnsorimotor loops Robot de désherbage localisé par to weed AUtonomously Précision pour le Binage autonome des cultures en procédé électrique haute tension Intra-rang Précoce Agriculture Durable combiné avec une gestion prédictive par vision hyper-spectrale et post- évaluation par drone Project acronym PEAD ROSEAU WeedElec BIPBIP Coordinating body Laboratoire de l’Intégration Research institut Xlim SITIA (Engineering company) UMR Itap Information, Technologies, du Matériau au Système (UMR CNRS 7252, multi- Analyse environnementale, Procédés sites Limoges, Poitiers, agricoles (IMS, UMR5218 CNRS, Brive, Angoulême) university of Bordeaux, (Irstea, Montpellier SupAgro) Team REMIX Bordeaux INP) Teams COMIC and PEPS Team MOTIVE     Academic partners Bordeaux Sciences CNRS INRA (UMR Irstea Agro Agroécologie)   Université de CIRAD (AMAP, UR AIDA )   Limoges (Xlim) IRSEEM Bordeaux INP  INRIA ( ZENITH, LIRMM)  CNRS  INRA (UMR EMMAH/UAPV)  Université de Bordeaux (IMS, Labri équipe Rhoban)    AGRIAL Technical and Les Fermes Larrère CARBON BEE Les chambres régionales economic partners d’Agriculture de Pays de la   SABI AGRI Elatec Loire et de Bretagne  CTIFL Participating consortia to the ROSE challenge