Cotton Project 2

A project of the Robotics 2020 class of the School of Information Science and Technology (SIST) of ShanghaiTech University. Course Instructor: Prof. Sören Schwertfeger.

Renyou Yan, Wangshu Zhu, Chenqi Luo

This project is focused on the 6D pose estimation of the cotton and motion planning of robot arm for automatic cotton picking. A camera is implemented on the arm to identify the cotton and estimate its pose, which is essential for grasp planning. After a correct pose estimation, the arm will be planned to move to the position of the cotton and grasp it.

A RealSense camera obtains the RGB and the depth image data. Both classic computer vision method and deep learning method are utilized to estimate the pose of the cotton. However, classic computer-vision-based method is not robust in complex environment, including various illuminations, backgrounds, occlusions, etc. Deep learning method is a great tool to let us detect the target cotton in such environment and a localization algorithm is needed to get the 3D position. A robust pose for grasping can improve both accuracy and efficiency of the grasping task. Up to now, we have build a system to get labeled simulation cotton dataset. We have also collected more than 20,000 labeled images using this system, which is essential for object detecting deep learning network, e.g. YOLO.


Fig. 1 The hardware description.  Our system is integrated with the Husky A200 as the base, the Schunk arm combined with the cotton sucking structure, and the RealSense L515 implemented on the end-effector ofthe arm

Video: