Fruit Detection and Pose Estimation for Grape Cluster–Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks
Open Access
- 22 June 2021
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Robotics and AI
Abstract
Reliable and robust fruit-detection algorithms in nonstructural environments are essential for the efficient use of harvesting robots. The pose of fruits is crucial to guide robots to approach target fruits for collision-free picking. To achieve accurate picking, this study investigates an approach to detect fruit and estimate its pose. First, the state-of-the-art mask region convolutional neural network (Mask R-CNN) is deployed to segment binocular images to output the mask image of the target fruit. Next, a grape point cloud extracted from the images was filtered and denoised to obtain an accurate grape point cloud. Finally, the accurate grape point cloud was used with the RANSAC algorithm for grape cylinder model fitting, and the axis of the cylinder model was used to estimate the pose of the grape. A dataset was acquired in a vineyard to evaluate the performance of the proposed approach in a nonstructural environment. The fruit detection results of 210 test images show that the average precision, recall, and intersection over union (IOU) are 89.53, 95.33, and 82.00%, respectively. The detection and point cloud segmentation for each grape took approximately 1.7 s. The demonstrated performance of the developed method indicates that it can be applied to grape-harvesting robots.Keywords
This publication has 32 references indexed in Scilit:
- Fully Convolutional Networks for Semantic SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Localisation of litchi in an unstructured environment using binocular stereo visionBiosystems Engineering, 2016
- Determination of mango fruit from binary image using randomized Hough transformPublished by SPIE-Intl Soc Optical Eng ,2015
- Fast R-CNNPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Detecting citrus fruits and occlusion recovery under natural illumination conditionsComputers and Electronics in Agriculture, 2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic SegmentationPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2014
- An image segmentation method for apple sorting and grading using support vector machine and Otsu’s methodComputers and Electronics in Agriculture, 2013
- Evaluation of a strawberry-harvesting robot in a field testBiosystems Engineering, 2010
- Field Test of an Autonomous Cucumber Picking RobotBiosystems Engineering, 2003
- A flexible new technique for camera calibrationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000