Revolutionizing Litchi Harvesting: A Smart Robotic System with Active Obstruction Removal

By Dr. Sampath LonkaReviewed by Susha Cheriyedath, M.Sc.Nov 14 2023

In a recent publication in the journal Agronomy, researchers unveiled an advanced iteration of a robotic prototype equipped with a cutting-edge visual system designed for proactive obstruction removal in Litchi harvesting.

Background

Litchi, a tropical string fruit indigenous to southern China, holds immense economic value, contributing over 60 percent of the global output. Manual harvesting currently dominates the litchi landscape, yet in anticipation of future agricultural labor shortages, the imperative shift towards agricultural automation is evident.

Automatic picking robots are pivotal in enhancing efficiency and reducing labor costs. These robots rely on perceptive devices such as lidar instruments, RGB (Red, Green, and Blue) cameras, binocular cameras, and haptic sensors for fruit recognition and spatial localization, enabling strategic picking paths.

Traditional methods, which rely on surface features, encounter challenges amidst random obstructions and changing lighting conditions. Deep learning models, including DensenNet-201, Xception, and YOLO (You Only Look Once) series algorithms, have emerged as potent tools for fruit recognition. However, in natural environments, external interference poses challenges. Researchers pioneered a robot active perception system, integrating the YOLO version 8 (YOLOv8) model with monocular and binocular image processing to identify litchi-picking points. Additionally, an intelligent algorithm discerns obstruction types, and a visual perception-based control method facilitates active obstruction removal, considering the feeding and picking ideal posture of the robotic arm.

Integrated Litchi-picking robotic system

The litchi-picking robotic system for obstruction removal comprises hardware and software elements. An industrial personal computer (IPC), an end-effector, a robotic arm, a collection device, a moving device, an obstacle removal unit, and a binocular vision camera are among the hardware components. The system's motion device follows a predefined trajectory and adjusts its path upon detecting a litchi target.

The robotic arm with six degrees of freedom executes subsequent operations when within the working range. The process involves the mobile device approaching a litchi tree, detecting litchi fruits with the YOLOv8-Seg model, localizing picking points using binocular stereo vision, and determining obstruction types. The end-effector's movement is controlled based on recognition results, and obstruction removal is implemented. Picking point recognition, localization, and obstruction identification are integral to the system's functionality.

Localization Method of Picking Points: The localization algorithm uses the YOLOv8-Seg model to segment litchi images, extract regions of interest (ROIs), and determine affiliation between litchi fruits and branches. Picking points on branches are identified using a pixel search algorithm. Three-dimensional coordinates of picking points and the number of litchi fruits are calculated using binocular stereo-vision technology. Obstruction identification involves recording the color features of the picking point, moving towards it, and analyzing scene changes. The proposed method achieved effective obstruction removal.

ROIs for Segmentation of Litchi Fruits and Branches: The YOLOv8-Seg model keeps its structural integrity when segmenting litchi fruits and branches. The model extracts features using convolutional kernels, generating segmentation areas based on the features. Affiliation relationships between litchi fruits and branches are determined by overlapping ROIs. Picking points on branches are recognized through binarization and an innovative pixel search algorithm. The model's segmentation performance and picking point recognition success rate were evaluated in experiments.

These experiments validate the system's effectiveness in litchi segmentation, picking point recognition, localization, obstruction identification, and removal. The proposed system showcases promising capabilities for autonomous litchi picking with obstruction handling.

Results and analysis

The YOLOv8-Seg model's loss convergence curves depict dynamic changes in the loss function for training and validation sets. The training loss rapidly descends, converging near zero after 200 iterations, indicating strong alignment with the training set. However, the validation loss exhibits fluctuations despite a swift descent, revealing susceptibility to random feature identification in unstructured environments.

Segmentation results showcase YOLOv8-Seg's precision (88.1 percent), recall (86.0 percent), F1-score (87.04), and mean average precision (78.1 percent), affirming accurate target recognition. Compact size and robust detection position make YOLOv8-Seg an ideal choice for a litchi-picking robot's processing unit. Indoor and outdoor litchi image segmentation demonstrates accurate results across diverse environments.

Picking point recognition achieves a high success rate of 91.42 percent outdoors and 83.33 percent indoors, showcasing real-time applicability in litchi-picking robots. Localization results reveal a maximum error of 7.7600 mm, a minimum error of 0.0708 mm, and an average error of 2.8511 mm, aligning with robotic arm fault-tolerant picking requirements.

Obstruction-type identification attains 100 percent accuracy, distinguishing obstructed and unobstructed situations effectively. In obstruction removal, motion traces display spatial redundancies, supporting successful obstruction removal. End-effector entry into redundancy achieves an 81.3 percent success rate, confirming the method's feasibility and effectiveness in navigating space for successful picking operations.

Conclusion

In summary, researchers introduced a litchi-picking robot system proficient in obstruction removal. They employ the YOLOv8-Seg model for litchi segmentation, an intelligent image algorithm that recognizes picking points. Binocular vision located points, and an algorithm identified obstruction types. The end-effector actively entered spaces, achieving an 81.3 percent success rate.

Results showcased high precision, a recognition rate of 88 percent, and accurate localization with errors ranging from 0.0708 to 7.7600 mm. The method attained 100 percent accuracy in identifying obstructions, confirming the robot's efficacy. Future research aims to delve into the posture of litchi cluster cutting points, further enhancing obstruction removal capabilities.