No Access

State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150080, P. R. China

Lining Sun

https://orcid.org/0000-0002-8354-2440

State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150080, P. R. China

E-mail Address: lnsun@hit.edu.cn

Search for more papers by this author

Mantian Li

https://orcid.org/0000-0003-4689-3484

State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150080, P. R. China

E-mail Address: lmt@hit.edu.cn

Search for more papers by this author

Pengfei Wang

https://orcid.org/0000-0002-3061-9897

State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150080, P. R. China

E-mail Address: pfwang@hit.edu.cn

Search for more papers by this author

Fusheng Zha

https://orcid.org/0000-0001-9695-1940

State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150080, P. R. China

E-mail Address: fushengzha@hit.edu.cn

Corresponding author.

Search for more papers by this author

Wei Guo

https://orcid.org/0000-0002-1845-2174

State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150080, P. R. China

E-mail Address: wguo01@hit.edu.cn

Corresponding author.

Search for more papers by this author

, and

Qiang Li

https://orcid.org/0000-0002-4315-0864

Neuroinformatics Group, Center for Cognitive Interaction Technology (CITEC), Bielefeld University, 33619 Bielefeld, Germany

E-mail Address: qli@techfak.uni-bielefeld.de

Search for more papers by this author

https://doi.org/10.1142/S0219843623500330Cited by:0 (Source: Crossref)

Abstract

Adaptive and cooperative control of arms and fingers for natural object reaching and grasping, without explicit 3D geometric pose information, is observed in humans. In this study, an image-based visual servoing controller, inspired by human grasping behavior, is proposed for an arm-gripper system. A large-scale dataset is constructed using Pybullet simulation, comprising paired images and arm-gripper control signals mimicking expert grasping behavior. Leveraging this dataset, a network is directly trained to derive a control policy that maps images to cooperative grasp control. Subsequently, the learned synergy grasping policy from the network is directly applied to a real robot with the same configuration. Experimental results demonstrate the effectiveness of the algorithm. Videos can be found at https://www.bilibili.com/video/BV1tg4y1b7Qe/.

Keywords:

References

1. Maskrcnn K. He, G. Gkioxari, P. Dollar and R. Girshick, Mask R-CNN, in Proc. IEEE Int. Conf. Computer Vision (Venice, 2017), pp. 2980–2988. Crossref, Google Scholar
2. J. Bohg et al., Towards grasp-oriented visual perception for humanoid robots, Int. J. Humanoid Robot. 6(3) (2009) 387–434. Link, Web of Science, Google Scholar
3. A. Provenzale, F. Cordella, L. Zollo, A. Davalli, R. Sacchetti and E. Guglielmelli, A grasp synthesis algorithm based on postural synergies for an anthropomorphic arm-hand robotic system, in Proc. IEEE RAS and EMBS Int. Conf. Biomedical Robotics and Biomechatronics (IEEE, 2014), pp. 958–963. Crossref, Google Scholar
4. J. Bohg, A. Morales, T. Asfour and D. Kragic, Data-driven grasp synthesis-a survey, IEEE Trans. Robot. 30(304) (2014) 289–309. Crossref, Google Scholar
5. S. Wang, W. Hu, L. Sun, X. Wang and Z. Li, Learning adaptive grasping from human demonstrations, IEEE/ASME Trans. Mechatron. 10 (2022) 3865–3873. Crossref, Google Scholar
6. K. Kleeberger, R. Bormann, W. Kraus and M. F. Huber, A survey on learning-based robotic grasping, Curr. Robot. Rep. 1 (2020) 239–249. Crossref, Google Scholar
7. J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley and K. Goldberg, Learning ambidextrous robot grasping policies, Sci. Robot. 4(26) (2019) eaau4984. Crossref, Web of Science, Google Scholar
8. H.-S. Fang, C. Wang, M. Gou and C. Lu, Graspnet-1billion: A large-scale benchmark for general object grasping, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (Seattle, Washington, USA, 2020), pp. 11444–11453. Crossref, Google Scholar
9. D. Morrison, P. Corke and J. Leitner, Learning robust, real-time, reactive robotic grasping, Int. J. Robot. Res. 39(2–3) (2020) 183–201. Crossref, Web of Science, Google Scholar
10. A. A. Shahid, L. Roveda, D. Piga and F. Braghin, Learning continuous control actions for robotic grasping with reinforcement learning, in Conf. Proc. — IEEE Int. Conf. Systems, Man and Cybernetics (IEEE, 2020), pp. 4066–4072. Crossref, Google Scholar
11. O. Brock et al., A framework for learning and control in intelligent humanoid robots, Int. J. Humanoid Robot. 2(3) (2005) 301–336. Link, Google Scholar
12. Y. Y. Tsai, H. Xu, Z. Ding, C. Zhang, E. Johns and B. Huang, Droid: Minimizing the reality gap using single-shot human demonstration, IEEE Robot. Autom. Lett. 6(4) (2021) 3168–3175. Crossref, Google Scholar
13. J. Rosell et al., Planning grasping motions for humanoid robots, Int. J. Humanoid Robot. 16(6) (2019) 1950041. Link, Web of Science, Google Scholar
14. N. Marturi, M. Kopicki, A. Rastegarpanah, V. Rajasekaran, M. Adjigble, R. Stolkin, A. Leonardis and Y. Bekiroglu, Dynamic grasp and trajectory planning for moving objects, Auton. Robots 43 (2019) 1241–1256. Crossref, Web of Science, Google Scholar
15. W. Hu, C. Yang, K. Yuan and Z. Li, Learning motor skills of reactive reaching and grasping of objects, in IEEE Int. Conf. Robotics and Biomimetics (IEEE, 2021), pp. 452–457. Crossref, Google Scholar
16. A. Depierre, E. Dellandrea and L. Chen, Jacquard: A large scale dataset for robotic grasp detection, in IEEE Int. Conf. Intelligent Robots and Systems (IEEE, 2018), pp. 3511–3516. Crossref, Google Scholar
17. Z. Zhang, C. Zhou, Y. Koike and J. Li, Single RGB image 6D object grasping system using pixel-wise voting network, Micromachines 13(2) (2022) 293. Crossref, Web of Science, Google Scholar
18. S. Peng, Y. Liu, Q. Huang, H. Bao and X. Zhou, PVNet: Pixel-wise voting network for 6DoF pose estimation, in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (IEEE, 2019), pp. 4561–4570. Crossref, Google Scholar
19. W. Kehl, F. Manhardt, F. Tombari, S. Ilic and N. Navab, SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again. https://wadimkehl.github.io/. Google Scholar
20. S. Ainetter and F. Fraundorfer, End-to-end trainable deep neural network for robotic grasp detection and semantic segmentation from RGB, in Proc. IEEE Int. Conf. Robotics and Automation (IEEE, 2021), pp. 13452–13458. Crossref, Google Scholar
21. V. Lepetit, F. Moreno-Noguer and P. Fua, Epnp: An accurate O(n) solution to the PnP problem, Int. J. Comput. Vis. 81(2) (2009) 55–166. Crossref, Web of Science, Google Scholar
22. D. Kragic and H. I. Christensen, Survey on visual servoing for manipulation, Comput. Vis. Act. Percept. Lab. Fiskartorpsv 15 (2002) 2002–2012. Google Scholar
23. P. I. Corke, D. Morrison, P. Corke and J. Rgen Leitner, Robust robotic manipulation view project learning to act on multi-modal data (LAMDa) view project learning robust, real-time, reactive robotic grasping, Int. J. Robot. Res. (2019). Google Scholar
24. S. Hutchinson and F. Chaumette, Visual servo control, part i: Basic approaches, IEEE Robot. Autom. Mag. 13(4) (2006) 82–90. Crossref, Web of Science, Google Scholar
25. D. Kappler, F. Meier, J. Issac, J. Mainprice, C. G. Cifuentes, M. Wuthrich, V. Berenz, S. Schaal, N. Ratliff and J. Bohg, Realtime perception meets reactive motion generation, IEEE Robot. Autom. Lett. 3(3) (2018) 1864–1871. Crossref, Google Scholar
26. I. Lenz, H. Lee and A. Saxena, Deep learning for detecting robotic grasps, Int. J. Robot. Res. 34(4–5) (2015) 705–724. Crossref, Web of Science, Google Scholar
27. S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz and D. Quillen, Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection, Int. J. Robot. Res. 37(4–5) (2018) 421–436. Crossref, Web of Science, Google Scholar
28. E. Valassakis, G. Papagiannis, N. Di Palo and E. Johns, Demonstrate once, imitate immediately (DOME): Learning visual servoing for one-shot imitation learning, in 2022 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS) (IEEE, 2022), pp. 8614–8621. Crossref, Google Scholar
29. C. Yu, Z. Cai, H. Pham and Q. C. Pham, Siamese convolutional neural network for sub-millimeter-accurate camera pose estimation and visual servoing, in IEEE Int. Conf. Intelligent Robots and Systems (IEEE, 2019), pp. 935–941. Crossref, Google Scholar
30. A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Commun. ACM 60(5) (2017) 84–90. Crossref, Google Scholar