Robotic Apple Harvesting in Washington State Joe Davidson b & - PowerPoint PPT Presentation

Robotic Apple Harvesting in Washington State Joe Davidson b & Abhisesh Silwal a IEEE Agricultural Robotics & Automation Webinar December 15 th , 2015 a Center for Precision and Automated Agricultural Systems (CPAAS) & b School of Mechanical and Materials Engineering, Washington State University (WSU) National Institute of Supported By: 1 Food and Agriculture

Acknowledgements This work was funded by the United States Department of Agriculture – National Institute of Food and Agriculture (USDA- NIFA) through the National Robotics Initiative (NRI). National Institute of Supported By: 2 Food and Agriculture

Presentation Overview • Motivation • Working environment • Design objectives • Hand picking analysis • System design • Preliminary field testing results • Future work • Questions National Institute of Supported By: 3 Food and Agriculture

Research Motivation Washington State fresh market apple industry in 2014 • – 2.7 million metric tons of apples valued at $1.84 billion USD 1 – Accounted for 70% of U.S. apple production The WA fresh market apple harvest requires • – Employment of 30,000 additional workers – An estimated cost of $1,100 to $2,100 USD per acre per year 2,3 Labor costs are rising and there is increasing uncertainty about the availability of • farm labor Lack of mechanical harvesting for fresh market apples is a significant problem • 1 USDA National Agricultural Statistics Service . (2014 Washington Agriculture Overview). Retrieved November 2, 2015, from http://www.nass.usda.gov/Quick_Stats/Ag_Overview/stateOverview.php?state=WASHINGTON 2 Galinato, S., & Gallardo, R. K. (2011). 2010 Estimated Cost of Producing Pears in North Central Washington (FS031E). Retrieved January 13, 2013, from http://extecon.wsu.edu/pages/Enterprise_Budgets. 3 Gallardo, R. K., Taylor, M., & Hinman, H. (2010). 2009 Cost Estimates of Establishing and Producing Gala Apples in Washington (FS005E). Retrieved January 7, 2013, from http://extecon.wsu.edu/pages/Enterprise_Budgets. National Institute of Supported By: 4 Food and Agriculture

Long-term Goal : Reduce dependence on the labor force for fresh market tree fruit harvesting National Institute of Supported By: 5 Food and Agriculture

Working Environment • Commercial apple orchard located in Prosser, WA • Highly unstructured environment • Modern cultivation systems with formal tree architectures • “Fruit Wall” concept simplifies the task National Institute of Supported By: 6 Food and Agriculture

Design Objectives • Cycle time < 6 sec • Detachment success > 90% • Fruit damage < 10% • Pick multiple apple varieties • Modular design that is cost-effective • Our Approach : An ‘undersensed’ design that executes look-and-move fruit picking, is mechanically robust to position error, and replicates the human picking process National Institute of Supported By: 7 Food and Agriculture

Initial Design Development: Manual Apple Picking • Fruit is grasped with a spherical power grasp 4 with the index finger applying pressure against the stem • No dexterous manipulation of the fruit with the fingers • To separate the apple from the branch, the hand moves the fruit in a pendulum motion 4 Cutkosky, M. R. (1989) On Grasp Choice, Grasp Models, and the Design of Hands for Manufacturing Tasks. IEEE Transactions on Robotics and Automation 5 (3): 269-279. National Institute of Supported By: 8 Food and Agriculture

‘Undersensed’ Hand Picking 5 • Are there effective methods to pick fruit that do not require fruit orientation and stem location? 5 Davidson, J., Silwal, A., Karkee, M., Mo, C., & Zhang, Q. (2015). Hand Picking Dynamic Analysis for Undersensed Robotic Apple Harvesting. Transactions of the ASABE . (Under review) National Institute of Supported By: 9 Food and Agriculture

Representative Hand Picking Data National Institute of Supported By: 10 Food and Agriculture

Mechanical Design • Custom design • 7 degrees of freedom • Modular configuration (Dynamixel Pro actuators) National Institute of Supported By: 11 Food and Agriculture

End-Effector Design • Underactuation provides shape-adaptive grasping • Passively compliant joints enhance robustness to position error & unplanned collisions 6 • Grasping is executed in an open-loop manner • Fabricated with additive manufacturing 6 Davidson, J., Silwal, A., Karkee, M., & Mo, C. (2015). Proof-of-Concept of a Robotic Apple Harvester. Robotics and Autonomous Systems . (Under review) National Institute of Supported By: 12 Food and Agriculture

Vision System 7 1 Color CCD Camera PMD Camcube 3.0 (ToF, 3D camera) Camera Rig Fusion 2 Identification Localization 3 7 Silwal, A., Gongal, A., & Karkee, M. (2014). Identification of Red Apples in Field Environment with Over-the-Row Machine Vision System. Agricultural Engineering International: Agric Eng Intl (CIGR National Institute of Supported By: Journal) , 16(4), 66-75. Food and Agriculture 13

Experimental Setup 8 8 Silwal, A., Davidson, J., Karkee, M., Mo, C., Zhang, Q., & Lewis, K. (2015). Design, Integration and Testing of a Robotic Apple Harvester. Journal of Field Robotics . (To be submitted) National Institute of Supported By: 14 Food and Agriculture

Hardware Architecture National Institute of Supported By: 15 Food and Agriculture

Video National Institute of Supported By: 16 Food and Agriculture

Vision Performance Actual vs. Recovered Image National Institute of Supported By: Food and Agriculture 17

Task Timing & Vision Accuracy Vision Accuracy: Total # of Images = 54 Total Fruit Manual Count: 193 Total Fruit Identified: 193 Identification Accuracy = 100% Total Fruit in Workspace = 150 Average Fruit per Image = 4 Average Vison Time per Image = 6.3 s Average Vision time per Apple = 1.7 s National Institute of Supported By: 18 Food and Agriculture

Picking Results • 127 of 150 fruits attempted were picked (approximately 85%) – 8/127 – No stems – 33/127 – Spur attached to fruit – 86/127 – Stems attached to fruit • Misses fall into the following five general categories 1. Poorly thinned branch (aka “fruit pendulum”) – 7 instances 2. Finger grabbed adjacent obstruction – 3 3. Position and/or calibration error – 8 4. Fruit slipped from grasp – 2 5. Previous fruit stuck in hand - 3 • No obvious evidence of bruising • Ideal fruit location is 3 – 6 in away from the trellis wire National Institute of Supported By: 19 Food and Agriculture

Picking Time • Mean picking time – 6.01 sec/per fruit – 1 st fruit in a cycle: 6.22 sec – Remaining fruits in a cycle: 5.84 sec • Each task in the picking sequence was segregated into an individual function and timed – Motion planning computation: 0.15 sec – Approach: 2.14 sec Picking time 2.5 – Grasp: 1.5 sec 2 – Removal: 1.23 sec 1.5 Time 1 – Fruit release: 1 sec 0.5 0 Motion Approach Grasp Removal Fruit Release Planning National Institute of Supported By: 20 Food and Agriculture

Future Work • Higher level decision making based on detection of trunks and trellis wires • Grasp planning based on visual input • Tactile sensor integration for detection of stem break, missed fruits, etc. National Institute of Supported By: 21 Food and Agriculture

Questions??? 22 National Institute of Supported By: Food and Agriculture