No Access

Tactile Behaviors with the Vision-Based Tactile Sensor FingerVision

Graduate School of Information Sciences, Tohoku University, 6-6-01 Aramaki Aza Aoba, Aoba-ku, Sendai, Miyagi 980-8579, Japan

E-mail Address: info@akihikoy.net

Corresponding author.

Search for more papers by this author

and

Christopher G. Atkeson

The Robotics Institute, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh PA 15213-3890, USA

E-mail Address: cga@cs.cmu.edu

Search for more papers by this author

https://doi.org/10.1142/S0219843619400024Cited by:45 (Source: Crossref)

This article is part of the issue:

Special Issue on New Advances of Humanoid Robotics (Humanoids 2017); Guest Editors: Hsiu-Chin Lin (University of Edinburgh, UK), Michael Mistry (University of Edinburgh, UK), Ales Leonardis (University of Birmingham, UK) and Tamim Asfour (University of Karlsruhe, Germany)

Abstract

This paper introduces a vision-based tactile sensor FingerVision, and explores its usefulness in tactile behaviors. FingerVision consists of a transparent elastic skin marked with dots, and a camera that is easy to fabricate, low cost, and physically robust. Unlike other vision-based tactile sensors, the complete transparency of the FingerVision skin provides multimodal sensation. The modalities sensed by FingerVision include distributions of force and slip, and object information such as distance, location, pose, size, shape, and texture. The slip detection is very sensitive since it is obtained by computer vision directly applied to the output from the FingerVision camera. It provides high-resolution slip detection, which does not depend on the contact force, i.e., it can sense slip of a lightweight object that generates negligible contact force. The tactile behaviors explored in this paper include manipulations that utilize this feature. For example, we demonstrate that grasp adaptation with FingerVision can grasp origami, and other deformable and fragile objects such as vegetables, fruits, and raw eggs.

Keywords:

References

1. A. Yamaguchi and C. G. Atkeson , Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables, in 16th IEEE-RAS Int. Conf. Humanoid Robots (Humanoids’16), 2016. Google Scholar
2. N. F. Lepora and B. Ward-Cherrier , Superresolution with an optical tactile sensor, in 2015 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS) (2015), pp. 2686–2691. Google Scholar
3. R. Li, R. Platt, W. Yuan, A. Ten Pas, N. Roscup, M. A. Srinivasan and E. Adelson , Localization and manipulation of small parts using GelSight tactile sensing, in 2014 IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 2014, pp. 3988–3993. Google Scholar
4. A. Yamaguchi, FingerVision, http://akihikoy.net/p/fv.html (2017). Google Scholar
5. A. Yamaguchi and C. G. Atkeson , Implementing tactile behaviors using FingerVision, in 17th IEEE-RAS Int. Conf. Humanoid Robots (Humanoids’17) (2017). Google Scholar
6. A. Yamaguchi , FingerVision for tactile behaviors, manipulation, and haptic feedback teleoperation, in 4th IEEJ Int. Workshop on Sensing, Actuation, Motion Control, and Optimization (SAMCON2018), 2018. Google Scholar
7. A. Yamaguchi and C. G. Atkeson , Grasp adaptation control with finger vision: Verification with deformable and fragile objects, in 35th Annual Conf. Robotics Society of Japan (RSJ2017), 2017, pp. 1L3–01. Google Scholar
8. Barrett Technology, About BarrettHand, https://www.barrett.com/about-barrethand/ (2018) [accessed on 26 Oct 2018]. Google Scholar
9. Willow Garage, PR2, http://www.willowgarage.com/pages/pr2/overview (2015) [accessed on 26 Oct 2018]. Google Scholar
10. RightHand Robotics, “ReFlex hand,” https://www.labs.righthandrobotics.com/reflexhand (2018) [accessed on 26 Oct 2018]. Google Scholar
11. U. Martinez-Hernandez , Tactile sensors, Scholarpedia 10(4) (2015) 32398. Google Scholar
12. Z. Kappassov, J.-A. Corrales and V. Perdereau , Tactile sensing in dexterous robot hands review, Robot. Auton. Syst. 74 (2015) 195–220. Web of Science, Google Scholar
13. S. Begej , Planar and finger-shaped optical tactile sensors for robotic applications, IEEE J. Robot. Autom. 4(5) (1988) 472–484. Google Scholar
14. H. Maekawa, K. Tanie, K. Komoriya, M. Kaneko, C. Horiguchi, and T. Sugawara , Development of a finger-shaped tactile sensor and its evaluation by active touch, in Proc. 1992 IEEE Int. Conf. Robotics and Automation, 1992, Vol. 2, 1992, pp. 1327–1334. Google Scholar
15. H. Yussof, J. Wada and M. Ohka , Sensorization of robotic hand using optical three-axis tactile sensor: Evaluation with grasping and twisting motions, J. Comput. Sci. 6(8) (2010) 955–962. Google Scholar
16. T. Ikai, S. Kamiya and M. Ohka , Robot control using natural instructions via visual and tactile sensations, J. Comput. Sci. 12(5) (2016) 246–254. Google Scholar
17. K. Nagata, M. Ooki and M. Kakikura , Feature detection with an image based compliant tactile sensor, in 1999 IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 1999, pp. 838–843. Google Scholar
18. K. Kamiyama, H. Kajimoto, N. Kawakami and S. Tachi , Evaluation of a vision-based tactile sensor, in 2004 IEEE Int. Conf. Robotics and Automation, 2004, Proc. ICRA ’04, Vol. 2, 2004, pp. 1542–1547. Google Scholar
19. J. Ueda, Y. Ishida, M. Kondo, and T. Ogasawara , Development of the NAIST-Hand with vision-based tactile fingertip sensor, in Proc. 2005 IEEE Int. Conf. Robotics and Automation, 2005, pp. 2332–2337. Google Scholar
20. C. Chorley, C. Melhuish, T. Pipe and J. Rossiter , Development of a tactile sensor based on biologically inspired edge encoding, in Int. Conf. Advanced Robotics, 2009, ICAR 2009, 2009, pp. 1–6. Google Scholar
21. D. Hristu, N. Ferrier and R. W. Brockett , The performance of a deformable-membrane tactile sensor: Basic results on geometrically-defined tasks, in IEEE Int. Conf. Robotics and Automation, 2000, Proc. ICRA ’00, Vol. 1, 2000, pp. 508–513. Google Scholar
22. Y. Ito, Y. Kim and G. Obinata , Robust slippage degree estimation based on reference update of vision-based tactile sensor, IEEE Sensors J. 11(9) (2011) 2037–2047. Web of Science, Google Scholar
23. T. Assaf, C. Roke, J. Rossiter, T. Pipe and C. Melhuish , Seeing by touch: Evaluation of a soft biologically-inspired artificial fingertip in real-time active touch, Sensors 14(2) (2014) 2561. Web of Science, Google Scholar
24. T. Sakuma, F. Von Drigalski, M. Ding, J. Takamatsu, and T. Ogasawara , A universal gripper using optical sensing to acquire tactile information and membrane deformation, in 2018 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2018, pp. 1–9. Google Scholar
25. B. W. McInroe, C. L. Chen, K. Y. Goldberg, R. Bajcsy and R. S. Fearing , Towards a soft fingertip with integrated sensing and actuation, in 2018 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2018, pp. 6437–6444. Google Scholar
26. OptoForce Co., White paper: Optical force sensors — introduction to the technology, Tech. Rep., January 2015. Available: http://optoforce.com/. Google Scholar
27. M. K. Johnson and E. H. Adelson , Retrographic sensing for the measurement of surface texture and shape, in 2009 IEEE Conf. Computer Vision and Pattern Recognition, 2009, pp. 1070–1077. Google Scholar
28. W. Yuan, R. Li, M. A. Srinivasan, and E. H. Adelson , Measurement of shear and slip with a GelSight tactile sensor, in 2015 IEEE Int. Conf. Robotics and Automation (ICRA), 2015, pp. 304–311. Google Scholar
29. E. Donlon, S. Dong, M. Liu, J. Li, E. Adelson and A. Rodriguez , Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger, in IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2018. Google Scholar
30. R. Patel and N. Correll , Integrated force and distance sensing using elastomer-embedded commodity proximity sensors, in Proc. Robotics: Science and Systems, 2016. Google Scholar
31. M. Ueda, K. Iwata and H. Shingu , Tactile sensors for an industrial robot to detect a slip, in 2nd Int. Symp. Industrial Robots, 1972, pp. 63–76. Google Scholar
32. D. Dornfeld and C. Handy , Slip detection using acoustic emission signal analysis, in Proc. 1987 IEEE Int. Conf. Robotics and Automation, Vol. 4, 1987, pp. 1868–1875. Google Scholar
33. M. R. Tremblay and M. R. Cutkosky , Estimating friction using incipient slip sensing during a manipulation task, in 1993 Proc. IEEE Int. Conf. Robotics and Automation, Vol. 1, 1993, pp. 429–434. Google Scholar
34. R. D. Howe and M. R. Cutkosky , Sensing skin acceleration for slip and texture perception, in Proc. 1989 Int. Conf. Robotics and Automation, Vol. 1, 1989, pp. 145–150. Google Scholar
35. R. Fernandez, I. Payo, A. S. Vazquez and J. Becedas , Micro-vibration-based slip detection in tactile force sensors, Sensors 14(1) (2014) 709–730. Web of Science, Google Scholar
36. R. Fernandez, I. Payo, A. S. Vazquez, and J. Becedas , Slip Detection in Robotic Hands with Flexible Parts (Springer International Publishing, Cham, 2014), pp. 153–167. Google Scholar
37. A. A. S. Al-Shanoon, S. A. Ahmad, and M. K. B. Hassan , Slip detection with accelerometer and tactile sensors in a robotic hand model, IOP Conf. Ser. Mater. Sci. Eng. 99(1) (2015) 012001. Google Scholar
38. R. Fernandez, I. Payo, A. S. Vazquez and J. Becedas , Slip Detection in a Novel Tactile Force Sensor (Springer International Publishing, Cham, 2016), pp. 237–252. Google Scholar
39. C. Melchiorri , Slip detection and control using tactile and force sensors, IEEE/ASME Trans. Mechatron. 5(3) (2000) 235–243. Web of Science, Google Scholar
40. J. M. Romano, K. Hsiao, G. Niemeyer, S. Chitta and K. J. Kuchenbecker , Human-inspired robotic grasp control with tactile sensing, IEEE Trans. Robotics 27(6) (2011) 1067–1079. Web of Science, Google Scholar
41. E. G. M. Holweg, H. Hoeve, W. Jongkind, L. Marconi, C. Melchiorri and C. Bonivento , Slip detection by tactile sensors: Algorithms and experimental results, in Proc. IEEE Int. Conf. Robot. Automa. Vol. 4, 1996, pp. 3234–3239. Google Scholar
42. N. Tsujiuchi, T. Koizumi, A. Ito, H. Oshima, Y. Nojiri, Y. Tsuchiya and S. Kurogi , Slip detection with distributed-type tactile sensor, in 2004 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Vol. 1, 2004, pp. 331–336. Google Scholar
43. V. A. Ho, T. Nagatani, A. Noda and S. Hirai , What can be inferred from a tactile arrayed sensor in autonomous in-hand manipulation? in 2012 IEEE Int. Conf. Automation Science and Engineering, 2012, pp. 461–468. Google Scholar
44. T. Hasegawa and K. Honda , Detection and measurement of fingertip slip in multi-fingered precision manipulation with rolling contact, in Int. Conf. Multisensor Fusion and Integration for Intelligent Systems (MFI 2001), 2001, pp. 43–48. Google Scholar
45. D. Gunji, Y. Mizoguchi, S. Teshigawara, A. Ming, A. Namiki, M. Ishikawa and M. Shimojo , Grasping force control of multi-fingered robot hand based on slip detection using tactile sensor, in 2008 IEEE Int. Conf. Robotics and Automation, 2008, pp. 2605–2610. Google Scholar
46. J. Reinecke, A. Dietrich, F. Schmidt and M. Chalon , Experimental comparison of slip detection strategies by tactile sensing with the BioTac on the DLR hand arm system, in 2014 IEEE Int. Conf. Robotics and Automation (ICRA), 2014, pp. 2742–2748. Google Scholar
47. R. Johansson, G. Loeb, N. Wettels, D. Popovic and V. Santos, Biomimetic tactile sensor for control of grip, (2011), US Patent 7,878,075. Google Scholar
48. Z. Su, K. Hausman, Y. Chebotar, A. Molchanov, G. E. Loeb, G. S. Sukhatme and S. Schaal , Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor, in 2015 IEEE-RAS 15th Int. Conf. Humanoid Robots (Humanoids), 2015, pp. 297–303. Google Scholar
49. A. Ikeda, Y. Kurita, J. Ueda, Y. Matsumoto and T. Ogasawara , Grip force control for an elastic finger using vision-based incipient slip feedback,” in 2004 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Vol. 1, 2004, pp. 810–815. Google Scholar
50. P. K. Allen, A. T. Miller, P. Y. Oh and B. S. Leibowitz , Integration of vision, force and tactile sensing for grasping, Int. J. Intelligent Machines 4 (1999) 129–149. Google Scholar
51. K. Hsiao, S. Chitta, M. Ciocarlie and E. G. Jones , Contact-reactive grasping of objects with partial shape information, in 2010 IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 2010, pp. 1228–1235. Google Scholar
52. R. S. Johansson and J. R. Flanagan , Coding and use of tactile signals from the fingertips in object manipulation tasks, Nat. Rev. Neurosci. 10(345) (2009). Google Scholar
53. R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson and S. Levine , More than a feeling: Learning to grasp and regrasp using vision and touch, IEEE Robot. Autom. Lett. 3(4) (2018) 3300–3307. Web of Science, Google Scholar
54. F. Hogan, M. Bauza Villalonga, O. Canal Anton, E. Donlon and A. Rodriguez , Tactile regrasp: Grasp adjustments via simulated tactile transformations, in IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2018. Google Scholar
55. H. Dang and P. K. Allen , Stable grasping under pose uncertainty using tactile feedback, Autonomous Robots 36(4) (2014) 309–330. Web of Science, Google Scholar
56. M. Li, Y. Bekiroglu, D. Kragic and A. Billard , Learning of grasp adaptation through experience and tactile sensing, in 2014 IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 2014, pp. 3339–3346. Google Scholar
57. E. Hyttinen, D. Kragic and R. Detry , Estimating tactile data for adaptive grasping of novel objects, in 2017 IEEE-RAS 17th Int. Conf. Humanoid Robotics (Humanoids), 2017, pp. 643–648. Google Scholar
58. C. Chuang, M. Wang, Y. Yu, C. Mu, K. Lu and C. Lin , Flexible tactile sensor for the grasping control of robot fingers, in 2013 Int. Conf. Advanced Robotics and Intelligent Systems, 2013, pp. 141–146. Google Scholar
59. F. Veiga, H. van Hoof, J. Peters and T. Hermans , Stabilizing novel objects by learning to predict tactile slip, in 2015 IEEE/RSJ Int. Conf. Intelligent Robots and Systems, 2015, pp. 5065–5072. Google Scholar
60. M. Kaboli, K. Yao and G. Cheng , Tactile-based manipulation of deformable objects with dynamic center of mass, in 2016 IEEE-RAS 16th Int. Conf. Humanoid Robots (Humanoids), 2016, pp. 752–757. Google Scholar
61. J. Venter and A. M. Mazid , Tactile sensor based intelligent grasping system, in 2017 IEEE Int. Conf. Mechatronics (ICM), 2017, pp. 303–308. Google Scholar
62. L. Cramphorn, B. Ward-Cherrier and N. F. Lepora , Tactile manipulation with biomimetic active touch, in 2016 IEEE Int. Conf. Robotics and Automation (ICRA), 2016, pp. 123–129. Google Scholar
63. H. van Hoof, T. Hermans, G. Neumann and J. Peters , Learning robot in-hand manipulation with tactile features, in 2015 IEEE-RAS 15th Int. Conf. Humanoid Robots (Humanoids), 2015, pp. 121–127. Google Scholar
64. R. B. Hellman, C. Tekin, M. van der Schaar and V. J. Santos , Functional contour-following via haptic perception and reinforcement learning, IEEE Trans. Haptics 11(1) (2018) 61–72. Web of Science, Google Scholar
65. A. Yamaguchi and C. G. Atkeson , Neural networks and differential dynamic programming for reinforcement learning problems, in IEEE Int. Conf. Robotics and Automation (ICRA’16), 2016. Google Scholar