Soft Robotics and Artificial Intelligence
*Obviously rather a lot of thought has been going into this.
https://www.researchgate.net/publication/329243898_Deep_Reinforcement_Learning_for_Soft_Robotic_Applications_Brief_Overview_with_Impending_Challenges
Deep Reinforcement Learning for Soft Robotic Applications: Brief Overview with Impending Challenges
by Sarthak Bhagat
Abstract
The increasing trend of studying the innate softness of robotic structures and amalgamating it with the benefits of the extensive developments in the field of embodied intelligence has led to sprouting of a relatively new yet extremely rewarding sphere of technology. The fusion of current deep reinforcement algorithms with physical advantages of a soft bio-inspired structure certainly directs us to a fruitful prospect of designing completely self-sufficient agents that are capable of learning from observations collected from their environment to achieve a task they have been assigned. For soft robotics structure possessing countless degrees of freedom, it is often not easy (something not even possible) to formulate mathematical constraints necessary for training a deep reinforcement learning (DRL) agent for the task in hand, hence, we resolve to imitation learning techniques due to ease of manually performing such tasks like manipulation that could be comfortably mimicked by our agent. Deploying current imitation learning algorithms on soft robotic systems have been observed to provide satisfactory results but there are still challenges in doing so. This review article thus posits an overview of various such algorithms along with instances of them being applied to real world scenarios and yielding state-of-the-art results followed by brief descriptions on various pristine branches of DRL research that may be centers of future research in this field of interest.
(...)
References
Trimmer, B. A Confluence of Technology: Putting Biology into Robotics. Soft Robot. 2014, 1, 159–160. [CrossRef] [CrossRef]
Banerjee, H.; Tse, Z.T.H.; Ren, H. Soft Robotics with Compliance and Adaptation for Biomedical Applications and Forthcoming Challenges. Int. J. Robot. Autom. 2018, 33. [CrossRef] [CrossRef]
Trivedi, D.; Rahn, C.D.; Kier, W.M.; Walker, I.D. Soft robotics: Biological inspiration, state of the art, and future research. Appl. Bionics Biomech. 2008, 5, 99–117. [CrossRef] [CrossRef]
Banerjee, H.; Ren, H. Electromagnetically responsive soft-flexible robots and sensors for biomedical applications and impending challenges. In Electromagnetic Actuation and Sensing in Medical Robotics; Springer: Berlin, Germany, 2018; pp. 43–72.
Banerjee, H.; Aaron, O.Y.W.; Yeow, B.S.; Ren, H. Fabrication and Initial Cadaveric Trials of Bi-directional Soft Hydrogel Robotic Benders Aiming for Biocompatible Robot-Tissue Interactions. In Proceedings of the IEEE ICARM 2018, Singapore, 18–20 July 2018.
Banerjee, H.; Roy, B.; Chaudhury, K.; Srinivasan, B.; Chakraborty, S.; Ren, H. Frequency-induced morphology alterations in microconfined biological cells. Med. Biol. Eng. Comput. 2018. [CrossRef] [PubMed] [CrossRef]
Kim, S.; Laschi, C.; Trimmer, B. Soft robotics: A bioinspired evolution in robotics. Trends Biotechnol. 2013,
31, 287–294. [CrossRef] [PubMed] [CrossRef]
Ren, H.; Banerjee, H. A Preface in Electromagnetic Robotic Actuation and Sensing in Medicine.
In Electromagnetic Actuation and Sensing in Medical Robotics; Springer: Berlin, Germany, 2018; pp. 1–10.
Banerjee, H.; Shen, S.; Ren, H. Magnetically Actuated Minimally Invasive Microbots for Biomedical Applications. In Electromagnetic Actuation and Sensing in Medical Robotics; Springer: Berlin, Germany, 2018;
pp. 11–41.
Banerjee, H.; Suhail, M.; Ren, H. Hydrogel Actuators and Sensors for Biomedical Soft Robots: Brief Overview
with Impending Challenges. Biomimetics 2018, 3, 15. [CrossRef] [CrossRef] [PubMed]
Iida, F.; Laschi, C. Soft robotics: Challenges and perspectives. Proc. Comput. Sci. 2011, 7, 99–102. [CrossRef]
[CrossRef]
Schmidhuber, J. Deep learning in neural networks: An overview. Neural Netw. 2015, 61, 85–117. [CrossRef]
[CrossRef] [PubMed]
Andrychowicz, M.; Wolski, F.; Ray, A.; Schneider, J.; Fong, R.; Welinder, P.; McGrew, B.; Tobin, J.; Abbeel, O.P.;
Zaremba, W. Hindsight experience replay. In Proceedings of the Advances in Neural Information Processing
Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 5048–5058.
Deng, L. A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Trans.
Signal Inf. Process. 2014, 3. [CrossRef] [CrossRef]
Guo, Y.; Liu, Y.; Oerlemans, A.; Lao, S.; Wu, S.; Lew, M.S. Deep learning for visual understanding: A review.
Neurocomputing 2016, 187, 27–48. [CrossRef] [CrossRef]
Bagnell, J.A. An Invitation to Imitation; Technical Report; Carnegie-Mellon Univ Pittsburgh Pa Robotics Inst:
Pittsburgh, PA, USA, 2015.
Levine, S. Exploring Deep and Recurrent Architectures for Optimal Control. arXiv 2013, arXiv:1311.1761.
Lillicrap, T.P.; Hunt, J.J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; Wierstra, D. Continuous Control
with Deep Reinforcement Learning. arXiv 2015, arXiv:1509.02971.
Spielberg, S.; Gopaluni, R.B.; Loewen, P.D. Deep reinforcement learning approaches for process control.
In Proceedings of the 2017 6th International Symposium on Advanced Control of Industrial Processes
(AdCONIP), Taipei, Taiwan, 28–31 May 2017; pp. 201–206.
Khanbareh, H.; de Boom, K.; Schelen, B.; Scharff, R.B.N.; Wang, C.C.L.; van der Zwaag, S.; Groen, P. Large
area and flexible micro-porous piezoelectric materials for soft robotic skin. Sens. Actuators A Phys. 2017, 263,
554–562. [CrossRef]
Zhao, H.; O’Brien, K.; Li, S.; Shepherd, R.F. Optoelectronically innervated soft prosthetic hand via stretchable
optical waveguides. Sci. Robot. 2016, 1, eaai7529. [CrossRef]
Li, S.; Vogt, D.M.; Rus, D.; Wood, R.J. Fluid-driven origami-inspired artificial muscles. Proc. Natl. Acad.
Sci. USA 2017, 114, 13132–13137. [CrossRef]
Robotics 2019, 8, 4 30 of 36
Ho, S.; Banerjee, H.; Foo, Y.Y.; Godaba, H.; Aye, W.M.M.; Zhu, J.; Yap, C.H. Experimental characterization of a dielectric elastomer fluid pump and optimizing performance via composite materials. J. Intell. Mater. Syst. Struct. 2017, 28, 3054–3065. [CrossRef] [CrossRef]
Shepherd, R.F.; Ilievski, F.; Choi, W.; Morin, S.A.; Stokes, A.A.; Mazzeo, A.D.; Chen, X.; Wang, M.; Whitesides, G.M. Multigait soft robot. Proc. Natl. Acad. Sci. USA 2011, 108, 20400–20403. [CrossRef] [PubMed] [CrossRef]
Banerjee, H.; Pusalkar, N.; Ren, H. Single-Motor Controlled Tendon-Driven Peristaltic Soft Origami Robot.
J. Mech. Robot. 2018, 10, 064501. [CrossRef] [CrossRef]
Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction; MIT Press: Cambridge, MA, USA, 1998.
Dayan, P. Improving generalization for temporal difference learning: The successor representation.
Neural Comput. 1993, 5, 613–624. [CrossRef] [CrossRef]
Kulkarni, T.D.; Saeedi, A.; Gautam, S.; Gershman, S.J. Deep Successor Reinforcement Learning. arXiv 2016,
arXiv:1606.02396.
Barreto, A.; Dabney, W.; Munos, R.; Hunt, J.J.; Schaul, T.; van Hasselt, H.P.; Silver, D. Successor features
for transfer in reinforcement learning. In Proceedings of the Advances in Neural Information Processing
Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 4055–4065.
Zhang, J.; Springenberg, J.T.; Boedecker, J.; Burgard, W. Deep reinforcement learning with successor features
for navigation across similar environments. In Proceedings of the 2017 IEEE/RSJ International Conference
on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, 24–28 September 2017; pp. 2371–2378.
Fu, M.C.; Glover, F.W.; April, J. Simulation optimization: A review, new developments, and applications. In Proceedings of the 37th Conference on Winter Simulation, Orlando, FL, USA, 4–7 December 2005;
pp. 83–95.
Szita, I.; Lörincz, A. Learning Tetris using the noisy cross-entropy method. Neural Comput. 2006,
18, 2936–2941. [CrossRef] [CrossRef] [PubMed]
Schulman, J.; Moritz, P.; Levine, S.; Jordan, M.; Abbeel, P. High-Dimensional Continuous Control Using
Generalized Advantage Estimation. arXiv 2015, arXiv:1506.02438.
Williams, R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning.
Mach. Learn. 1992, 8, 229–256. [CrossRef] [CrossRef]
Silver, D.; Lever, G.; Heess, N.; Degris, T.; Wierstra, D.; Riedmiller, M. Deterministic policy gradient
algorithms. In Proceedings of the ICML, Beijing, China, 21–26 June 2014.
Sutton, R.S. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bull. 1991,
2, 160–163. [CrossRef] [CrossRef]
Weber, T.; Racanière, S.; Reichert, D.P.; Buesing, L.; Guez, A.; Rezende, D.J.; Badia, A.P.; Vinyals, O.; Heess, N.;
Li, Y.; et al. Imagination-Augmented Agents for Deep Reinforcement Learning. arXiv 2017, arXiv:1707.06203.
Kalweit, G.; Boedecker, J. Uncertainty-driven imagination for continuous deep reinforcement learning. In Proceedings of the Conference on Robot Learning, Mountain View, CA, USA, 13–15 November 2017;
pp. 195–206.
Banerjee, H.; Pusalkar, N.; Ren, H. Preliminary Design and Performance Test of Tendon-Driven Origami-Inspired
Soft Peristaltic Robot. In Proceedings of the 2018 IEEE International Conference on Robotics and Biomimetics
(IEEE ROBIO 2018), Kuala Lumpur, Malaysia, 12–15 December 2018.
Cianchetti, M.; Ranzani, T.; Gerboni, G.; Nanayakkara, T.; Althoefer, K.; Dasgupta, P.; Menciassi, A. Soft Robotics
Technologies to Address Shortcomings in Today’s Minimally Invasive Surgery: The STIFF-FLOP Approach.
Soft Robot. 2014, 1, 122–131. [CrossRef] [CrossRef]
Hawkes, E.W.; Blumenschein, L.H.; Greer, J.D.; Okamura, A.M. A soft robot that navigates its environment
through growth. Sci. Robot. 2017, 2, eaan3028. [CrossRef]
Atalay, O.; Atalay, A.; Gafford, J.; Walsh, C. A Highly Sensitive Capacitive-Based Soft Pressure Sensor Based
on a Conductive Fabric and a Microporous Dielectric Layer. Adv. Mater. 2017. [CrossRef] [CrossRef]
Truby, R.L.; Wehner, M.J.; Grosskopf, A.K.; Vogt, D.M.; Uzel, S.G.M.; Wood, R.J.; Lewis, J.A. Soft Somatosensitive
Actuators via Embedded 3D Printing. Adv. Mater. 2018, 30, e1706383. [CrossRef] [PubMed] [CrossRef]
Bishop-Moser, J.; Kota, S. Design and Modeling of Generalized Fiber-Reinforced Pneumatic Soft Actuators.
IEEE Trans. Robot. 2015, 31, 536–545. [CrossRef] [CrossRef]
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.;
Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature 2015,518, 529. [CrossRef] [PubMed] [CrossRef] [PubMed]
Robotics 2019, 8, 4 31 of 36
Katzschmann, R.K.; DelPreto, J.; MacCurdy, R.; Rus, D. Exploration of underwater life with an acoustically controlled soft robotic fish. Sci. Robot. 2018, 3, eaar3449. [CrossRef]
Van Hasselt, H.; Guez, A.; Silver, D. Deep Reinforcement Learning with Double Q-Learning. In Proceedings of the AAAI, Phoenix, AZ, USA, 12–17 February 2016; Volume 2, p. 5.
Wang, Z.; Schaul, T.; Hessel, M.; Van Hasselt, H.; Lanctot, M.; De Freitas, N. Dueling Network Architectures for Deep Reinforcement Learning. arXiv 2015, arXiv:1511.06581.
Gu, S.; Lillicrap, T.; Sutskever, I.; Levine, S. Continuous deep q-learning with model-based acceleration. In Proceedings of the International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016; pp. 2829–2838.
Gu, S.; Holly, E.; Lillicrap, T.; Levine, S. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017; pp. 3389–3396.
Mnih, V.; Badia, A.P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016; pp. 1928–1937.
Wang, J.X.; Kurth-Nelson, Z.; Tirumala, D.; Soyer, H.; Leibo, J.Z.; Munos, R.; Blundell, C.; Kumaran, D.; Botvinick, M. Learning to Reinforcement Learn. arXiv 2016, arXiv:1611.05763.
Wu, Y.; Mansimov, E.; Grosse, R.B.; Liao, S.; Ba, J. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 5279–5288.
Levine, S.; Koltun, V. Guided policy search. In Proceedings of the International Conference on Machine Learning, Atlanta, GA, USA, 16 June–21 June 2013; pp. 1–9.
Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; Moritz, P. Trust region policy optimization. In Proceedings of the International Conference on Machine Learning, Lille, France, 6 July–11 July 2015; pp. 1889–1897.
Kakade, S.; Langford, J. Approximately optimal approximate reinforcement learning. In Proceedings of the ICML, Sydney, Australia, 8–12 July 2002; Volume 2, pp. 267–274.
Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal Policy Optimization Algorithms.arXiv 2017, arXiv:1707.06347.
Mirowski, P.; Pascanu, R.; Viola, F.; Soyer, H.; Ballard, A.J.; Banino, A.; Denil, M.; Goroshin, R.; Sifre, L.; Kavukcuoglu, K.; et al. Learning to Navigate in Complex Environments. arXiv 2016, arXiv:1611.03673.
Riedmiller, M.; Hafner, R.; Lampe, T.; Neunert, M.; Degrave, J.; Van de Wiele, T.; Mnih, V.; Heess, N.; Springenberg, J.T. Learning by Playing-Solving Sparse Reward Tasks from Scratch. arXiv 2018, arXiv:1802.10567.
Yu, T.; Finn, C.; Xie, A.; Dasari, S.; Zhang, T.; Abbeel, P.; Levine, S. One-Shot Imitation from Observing
Humans via Domain-Adaptive Meta-Learning. arXiv 2018, arXiv:1802.01557.
Levine, S.; Finn, C.; Darrell, T.; Abbeel, P. End-to-end training of deep visuomotor policies. J. Mach. Learn. Res.
2016, 17, 1334–1373.
Jaderberg, M.; Mnih, V.; Czarnecki, W.M.; Schaul, T.; Leibo, J.Z.; Silver, D.; Kavukcuoglu, K. Reinforcement
Learning with Unsupervised Auxiliary Tasks. arXiv 2016, arXiv:1611.05397.
Schaul, T.; Quan, J.; Antonoglou, I.; Silver, D. Prioritized Experience Replay. arXiv 2015, arXiv:1511.05952.
Bengio, Y.; Louradour, J.; Collobert, R.; Weston, J. Curriculum learning. In Proceedings of the 26th Annual
International Conference on Machine Learning, Montreal, QC, Canada, 14–18 June 2009; pp. 41–48.
Zhang, J.; Tai, L.; Boedecker, J.; Burgard, W.; Liu, M. Neural SLAM. arXiv 2017, arXiv:1706.09520.
Florensa, C.; Held, D.; Wulfmeier, M.; Zhang, M.; Abbeel, P. Reverse Curriculum Generation for
Reinforcement Learning. arXiv 2017, arXiv:1707.05300.
Pathak, D.; Agrawal, P.; Efros, A.A.; Darrell, T. Curiosity-driven exploration by self-supervised
prediction. In Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia,
6–11 August 2017; Volume 2017.
Sukhbaatar, S.; Lin, Z.; Kostrikov, I.; Synnaeve, G.; Szlam, A.; Fergus, R. Intrinsic Motivation and Automatic
Curricula via Asymmetric Self-Play. arXiv 2017, arXiv:1703.05407.
Fortunato, M.; Azar, M.G.; Piot, B.; Menick, J.; Osband, I.; Graves, A.; Mnih, V.; Munos, R.; Hassabis, D.;
Pietquin, O.; et al. Noisy Networks for Exploration. arXiv 2017, arXiv:1706.10295.
Plappert, M.; Houthooft, R.; Dhariwal, P.; Sidor, S.; Chen, R.Y.; Chen, X.; Asfour, T.; Abbeel, P.;
Andrychowicz, M. Parameter Space Noise for Exploration. arXiv 2017, arXiv:1706.01905.
Robotics 2019, 8, 4 32 of 36
Rafsanjani, A.; Zhang, Y.; Liu, B.; Rubinstein, S.M.; Bertoldi, K. Kirigami skins make a simple soft actuator crawl. Sci. Robot. 2018. [CrossRef]
Zhu, Y.; Mottaghi, R.; Kolve, E.; Lim, J.J.; Gupta, A.; Fei-Fei, L.; Farhadi, A. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 30–31 May 2017; pp. 3357–3364.
He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778.
Kolve, E.; Mottaghi, R.; Gordon, D.; Zhu, Y.; Gupta, A.; Farhadi, A. AI2-THOR: An Interactive 3d
Environment for Visual AI. arXiv 2017, arXiv:1712.05474.
Tai, L.; Paolo, G.; Liu, M. Virtual-to-real deep reinforcement learning: Continuous control of mobile robots
for mapless navigation. In Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), Vancouver, BC, Canada, 24–28 September 2017; pp. 31–36.
Chen, Y.F.; Everett, M.; Liu, M.; How, J.P. Socially aware motion planning with deep reinforcement learning. In Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
Vancouver, BC, Canada, 24–28 September 2017; pp. 1343–1350.
Long, P.; Fan, T.; Liao, X.; Liu, W.; Zhang, H.; Pan, J. Towards Optimally Decentralized Multi-Robot Collision
Avoidance via Deep Reinforcement Learning. arXiv 2017, arXiv:1709.10082.
Thrun, S.; Burgard, W.; Fox, D. Probabilistic Robotics (Intelligent Robotics and Autonomous Agents);
The MIT Press: Cambridge, MA, USA, 2001.
Gupta, S.; Davidson, J.; Levine, S.; Sukthankar, R.; Malik, J. Cognitive Mapping and Planning for Visual
Navigation. arXiv 2017, arXiv:1702.03920.
Gupta, S.; Fouhey, D.; Levine, S.; Malik, J. Unifying Map and Landmark Based Representations for Visual
Navigation. arXiv 2017, arXiv:1712.08125.
Parisotto, E.; Salakhutdinov, R. Neural Map: Structured Memory for Deep Reinforcement Learning. arXiv
2017, arXiv:1702.08360.
Kümmerle, R.; Grisetti, G.; Strasdat, H.; Konolige, K.; Burgard, W. G2o: A general framework for graph
optimization. In Proceedings of the 2011 IEEE International Conference on Robotics and Automation (ICRA),
Shanghai, China, 9–13 May 2011; pp. 3607–3613.
Parisotto, E.; Chaplot, D.S.; Zhang, J.; Salakhutdinov, R. Global Pose Estimation with an Attention-Based
Recurrent Network. arXiv 2018, arXiv:1802.06857.
Schaul, T.; Horgan, D.; Gregor, K.; Silver, D. Universal value function approximators. In Proceedings of the
International Conference on Machine Learning, Lille, France, 6 July–11 July 2015; pp. 1312–1320.
Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired Image-to-Image Translation Using Cycle-Consistent
Adversarial Networks. arXiv 2017, arXiv:1703.10593.
Khan, A.; Zhang, C.; Atanasov, N.; Karydis, K.; Kumar, V.; Lee, D.D. Memory Augmented Control Networks.
arXiv 2017, arXiv:1709.05706.
Bruce, J.; Sünderhauf, N.; Mirowski, P.; Hadsell, R.; Milford, M. One-Shot Reinforcement Learning for Robot
Navigation with Interactive Replay. arXiv 2017, arXiv:1711.10137.
Chaplot, D.S.; Parisotto, E.; Salakhutdinov, R. Active Neural Localization. arXiv 2018, arXiv:1801.08214.
Savinov, N.; Dosovitskiy, A.; Koltun, V. Semi-Parametric Topological Memory for Navigation. arXiv 2018,
arXiv:1803.00653.
Heess, N.; Sriram, S.; Lemmon, J.; Merel, J.; Wayne, G.; Tassa, Y.; Erez, T.; Wang, Z.; Eslami, A.; Riedmiller, M.; et al.
Emergence of Locomotion Behaviours in Rich Environments. arXiv 2017, arXiv:1707.02286.
Calisti, M.; Giorelli, M.; Levy, G.; Mazzolai, B.; Hochner, B.; Laschi, C.; Dario, P. An octopus-bioinspired solutiontomovementandmanipulationforsoftrobots. Bioinspir.Biomim.2011,6,036002.[CrossRef][PubMed]
[CrossRef]
Martinez, R.V.; Branch, J.L.; Fish, C.R.; Jin, L.; Shepherd, R.F.; Nunes, R.M.D.; Suo, Z.; Whitesides, G.M.
Robotic tentacles with three-dimensional mobility based on flexible elastomers. Adv. Mater. 2013, 25, 205–212.
[CrossRef] [PubMed] [CrossRef]
Caldera, S. Review of Deep Learning Methods in Robotic Grasp Detection. Multimodal Technol. Interact.
2018, 2, 57. [CrossRef] [CrossRef]
Zhou, J.; Chen, S.; Wang, Z. A Soft-Robotic Gripper with Enhanced Object Adaptation and Grasping
Reliability. IEEE Robot. Autom. Lett. 2017, 2, 2287–2293. [CrossRef] [CrossRef]
Robotics 2019, 8, 4 33 of 36
Finn, C.; Tan, X.Y.; Duan, Y.; Darrell, T.; Levine, S.; Abbeel, P. Deep Spatial Autoencoders for Visuomotor Learning. arXiv 2015, arXiv:1509.06113.
Tzeng, E.; Devin, C.; Hoffman, J.; Finn, C.; Peng, X.; Levine, S.; Saenko, K.; Darrell, T. Towards Adapting Deep Visuomotor Representations from Simulated to Real Environments. arXiv 2015, arXiv:1511.07111v3.
Fu, J.; Levine, S.; Abbeel, P. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, 9–14 October 2016; pp. 4019–4026.
Kumar, V.; Todorov, E.; Levine, S. Optimal control with learned local models: Application to dexterous manipulation. In Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), New York, NY, USA, 16–20 May 2016; pp. 378–383.
Gupta, A.; Eppner, C.; Levine, S.; Abbeel, P. Learning dexterous manipulation for a soft robotic hand from human demonstrations. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Deajeon, Korea, 9–14 October 2016; pp. 3786–3793.
Popov, I.; Heess, N.; Lillicrap, T.; Hafner, R.; Barth-Maron, G.; Vecerik, M.; Lampe, T.; Tassa, Y.; Erez, T.; Riedmiller, M. Data-Efficient Deep Reinforcement Learning for Dexterous manipulation. arXiv 2017, arXiv:1704.03073.
Prituja,A.;Banerjee,H.;Ren,H.ElectromagneticallyEnhancedSoftandFlexibleBendSensor:AQuantitative Analysis with Different Cores. IEEE Sens. J. 2018, 18, 3580–3589. [CrossRef] [CrossRef]
Sun, J.Y.; Zhao, X.; Illeperuma, W.R.; Chaudhuri, O.; Oh, K.H.; Mooney, D.J.; Vlassak, J.J.; Suo, Z. Highly stretchable and tough hydrogels. Nature 2012, 489, 133–136. [CrossRef] [CrossRef]
Tzeng,E.;Hoffman,J.;Zhang,N.;Saenko,K.;Darrell,T.DeepDomainConfusion:MaximizingforDomain Invariance. arXiv 2014, arXiv:1412.3474.
Goodfellow,I.;Pouget-Abadie,J.;Mirza,M.;Xu,B.;Warde-Farley,D.;Ozair,S.;Courville,A.;Bengio,Y. Generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems (NIPS 2014), Montreal, QC, Canada, 8–13 December 2014; pp. 2672–2680.
Radford, A.; Metz, L.; Chintala, S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv 2015, arXiv:1511.06434.
Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein gan. arXiv 2017, arXiv:1701.07875.
Hoffman,J.;Tzeng,E.;Park,T.;Zhu,J.Y.;Isola,P.;Saenko,K.;Efros,A.A.;Darrell,T.Cycada:Cycle-Consistent
AdversarialDomainAdaptation. arXiv2017,arXiv:1711.03213.
Doersch, C. Tutorial on Variational Autoencoders. arXiv 2016, arXiv:1606.05908v2.
Szabó,A.;Hu,Q.;Portenier,T.;Zwicker,M.;Favaro,P.ChallengesinDisentanglingIndependentFactorsof
Variation. arXiv 2017, arXiv:1711.02245v1.
Mathieu, M.; Zhao, J.J.; Sprechmann, P.; Ramesh, A.; LeCun, Y. Disentangling factors of variation in
deep representations using adversarial training. In Proceedings of the NIPS 2016, Barcelona, Spain,
5–10 December 2016.
Bousmalis,K.;Irpan,A.;Wohlhart,P.;Bai,Y.;Kelcey,M.;Kalakrishnan,M.;Downs,L.;Ibarz,J.;Pastor,P.;
Konolige, K.; et al. Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic
Grasping. arXiv 2017, arXiv:1709.07857.
Tobin,J.;Fong,R.;Ray,A.;Schneider,J.;Zaremba,W.;Abbeel,P.Domainrandomizationfortransferring
deep neural networks from simulation to the real world. In Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, 24–28 September 2017; pp. 23–30.
Peng, X.B.; Andrychowicz, M.; Zaremba, W.; Abbeel, P. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. arXiv 2017, arXiv:1710.06537.
Rusu,A.A.;Vecerik,M.;Rothörl,T.;Heess,N.;Pascanu,R.;Hadsell,R.Sim-to-RealRobotLearningfrom Pixels with Progressive Nets. arXiv 2016, arXiv:1610.04286.
Zhang,J.;Tai,L.;Xiong,Y.;Liu,M.;Boedecker,J.;Burgard,W.VrGogglesforRobots:Real-to-SimDomain Adaptation for Visual Control. arXiv 2018, arXiv:1802.00265.
Ruder, M.; Dosovitskiy, A.; Brox, T. Artistic style transfer for videos and spherical images. Int. J. Comput. Vis.2018, 126, 1199–1219. [CrossRef] [CrossRef]
Koenig, N.P.; Howard, A. Design and use paradigms for Gazebo, an open-source multi-robot simulator.IROS. Citeseer 2004, 4, 2149–2154.
Robotics 2019, 8, 4 34 of 36
Maddern, W.; Pascoe, G.; Linegar, C.; Newman, P. 1 year, 1000 km: The Oxford RobotCar dataset. Int. J. Robot. Res. 2017, 36, 3–15. [CrossRef] [CrossRef]
Dosovitskiy,A.;Ros,G.;Codevilla,F.;Lopez,A.;Koltun,V.CARLA:AnOpenUrbanDrivingSimulator.arXiv 2017, arXiv:1711.03938.
Chen,L.C.;Papandreou,G.;Kokkinos,I.;Murphy,K.;Yuille,A.L.SemanticImageSegmentationwithDeep Convolutional Nets and Fully Connected Crfs. arXiv 2014, arXiv:1412.7062.
Yang, L.; Liang, X.; Xing, E. Unsupervised Real-to-Virtual Domain Unification for End-to-End Highway Driving. arXiv 2018, arXiv:1801.03458.
Uesugi,K.;Shimizu,K.;Akiyama,Y.;Hoshino,T.;Iwabuchi,K.;Morishima,K.Contractileperformanceand controllability of insect muscle-powered bioactuator with different stimulation strategies for soft robotics.Soft Robot. 2016, 3, 13–22. [CrossRef] [CrossRef]
Niiyama,R.;Sun,X.;Sung,C.;An,B.;Rus,D.;Kim,S.PouchMotors:PrintableSoftActuatorsIntegrated with Computational Design. Soft Robot. 2015, 2, 59–70. [CrossRef] [CrossRef]
Gul,J.Z.;Sajid,M.;Rehman,M.M.;Siddiqui,G.U.;Shah,I.;Kim,K.C.;Lee,J.W.;Choi,K.H.3Dprintingfor soft robotics—A review. Sci. Technol. Adv. Mater. 2018, 19, 243–262. [CrossRef] [PubMed] [CrossRef]
Umedachi, T.; Vikas, V.; Trimmer, B. Highly deformable 3-D printed soft robot generating inching and crawling locomotions with variable friction legs. In Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, 3–7 November 2013; pp. 4590–4595.
Mutlu,R.;Tawk,C.;Alici,G.;Sariyildiz,E.A3Dprintedmonolithicsoftgripperwithadjustablestiffness. In Proceedings of the IECON 2017—43rd Annual Conference of the IEEE Industrial Electronics Society, Beijing, China, 29 October–1 November 2017; pp. 6235–6240.
Lu,N.;HyeongKim,D.FlexibleandStretchableElectronicsPavingtheWayforSoftRobotics.SoftRobot.2014, 1, 53–62 [CrossRef] [CrossRef]
Rohmer, E.; Singh, S.P.; Freese, M. V-REP: A versatile and scalable robot simulation framework. In Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan, 3–8 November 2013; pp. 1321–1326.
Shah,S.;Dey,D.;Lovett,C.;Kapoor,A.Airsim:High-fidelityvisualandphysicalsimulationforautonomous vehicles. In Field and Service Robotics; Springer: Berlin, Germany, 2018; pp. 621–635.
Pan,X.;You,Y.;Wang,Z.;Lu,C.VirtualtoRealReinforcementLearningforAutonomousDriving.arXiv2017, arXiv:1704.03952.
Savva,M.;Chang,A.X.;Dosovitskiy,A.;Funkhouser,T.;Koltun,V.MINOS:MultimodalIndoorSimulator for Navigation in Complex Environments. arXiv 2017, arXiv:1712.03931.
Wu,Y.;Wu,Y.;Gkioxari,G.;Tian,Y.BuildingGeneralizableAgentswithaRealisticandRich3DEnvironment.arXiv 2018, arXiv:1801.02209.
Coevoet, E.; Bieze, T.M.; Largilliere, F.; Zhang, Z.; Thieffry, M.; Sanz-Lopez, M.; Carrez, B.; Marchal, D.; Goury, O.; Dequidt, J.; et al. Software toolkit for modeling, simulation, and control of soft robots. Adv. Robot.2017, 31, 1208–1224. [CrossRef] [CrossRef]
Duriez, C.; Coevoet, E.; Largilliere, F.; Bieze, T.M.; Zhang, Z.; Sanz-Lopez, M.; Carrez, B.; Marchal, D.; Goury, O.; Dequidt, J. Framework for online simulation of soft robots with optimization-based inverse model. In Proceedings of the 2016 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), San Francisco, CA, USA, 13–16 December 2016; pp. 111–118.
Olaya,J.;Pintor,N.;Avilés,O.F.;Chaparro,J.Analysisof3RPSRoboticPlatformMotioninSimScapeand MATLAB GUI Environment. Int. J. Appl. Eng. Res. 2017, 12, 1460–1468.
Coevoet,E.;Escande,A.;Duriez,C.Optimization-basedinversemodelofsoftrobotswithcontacthandling.IEEE Robot. Autom. Lett. 2017, 2, 1413–1419. [CrossRef]
Yekutieli,Y.;Sagiv-Zohar,R.;Aharonov,R.;Engel,Y.;Hochner,B.;Flash,T.Dynamicmodeloftheoctopus arm. I. Biomechanics of the octopus reaching movement. J. Neurophysiol. 2005, 94, 1443–1458. [CrossRef] [PubMed] [CrossRef]
Zatopa,A.;Walker,S.;Menguc,Y.Fullysoft3D-printedelectroactivefluidicvalveforsofthydraulicrobots.Soft Robot. 2018, 5, 258–271. [CrossRef]
Ratliff, N.D.; Bagnell, J.A.; Srinivasa, S.S. Imitation learning for locomotion and manipulation. In Proceedings of the 2007 7th IEEE-RAS International Conference on Humanoid Robots, Pittsburgh, PA, USA, 29 November–1 December 2007; pp. 392–397.
Robotics 2019, 8, 4 35 of 36
Langsfeld,J.D.;Kaipa,K.N.;Gentili,R.J.;Reggia,J.A.;Gupta,S.K.Towards Imitation Learning of Dynamic Manipulation Tasks: A Framework to Learn from Failures. Available online: https://pdfs.semanticscholar. org/5e1a/d502aeb5a800f458390ad1a13478d0fbd39b.pdf (accessed on 18 January 2019).
Ross,S.;Gordon,G.;Bagnell,D.Areductionofimitationlearningandstructuredpredictiontono-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 11–13 April 2011; pp. 627–635.
Bojarski, M.; Del Testa, D.; Dworakowski, D.; Firner, B.; Flepp, B.; Goyal, P.; Jackel, L.D.; Monfort, M.; Muller, U.; Zhang, J.; et al. End to end Learning for Self-Driving Cars. arXiv 2016, arXiv:1604.07316.
Tai, L.; Li, S.; Liu, M. A deep-network solution towards model-less obstacle avoidance. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, 9–14 October 2016; pp. 2759–2764.
Giusti, A.; Guzzi, J.; Ciresan, D.C.; He, F.L.; Rodríguez, J.P.; Fontana, F.; Faessler, M.; Forster, C.; Schmidhuber, J.; Di Caro, G.; et al. A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots. IEEE Robot. Autom. Lett. 2016, 1, 661–667. [CrossRef] [CrossRef]
Codevilla,F.;Müller,M.;Dosovitskiy,A.;López,A.;Koltun,V.End-to-EndDrivingviaConditionalImitation Learning. arXiv 2017, arXiv:1710.02410.
Duan,Y.;Andrychowicz,M.;Stadie,B.C.;Ho,J.;Schneider,J.;Sutskever,I.;Abbeel,P.;Zaremba,W.One-Shot Imitation Learning. In Proceedings of the NIPS, Long Beach, CA, USA, 4–9 December 2017.
Finn, C.; Yu, T.; Zhang, T.; Abbeel, P.; Levine, S. One-Shot Visual Imitation Learning via Meta-Learning.arXiv 2017, arXiv:1709.04905.
Finn, C.; Abbeel, P.; Levine, S. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv2017, arXiv:1703.03400.
Eitel, A.; Hauff, N.; Burgard, W. Learning to Singulate Objects Using a Push Proposal Network. arXiv 2017, arXiv:1707.08101.
Ziebart, B.D.; Maas, A.L.; Bagnell, J.A.; Dey, A.K. Maximum Entropy Inverse Reinforcement Learning. In Proceedings of the AAAI, Chicago, IL, USA, 13–17 July 2008; Volume 8, pp. 1433–1438.
Okal, B.; Arras, K.O. Learning socially normative robot navigation behaviors with bayesian inverse reinforcement learning. In Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 16–20 May 2016; pp. 2889–2895.
Pfeiffer,M.;Schwesinger,U.;Sommer,H.;Galceran,E.;Siegwart,R.Predictingactionstoactpredictably: Cooperative partial motion planning with maximum entropy models. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Deajeon, Korea, 9–14 October 2016; pp. 2096–2101.
Kretzschmar,H.;Spies,M.;Sprunk,C.;Burgard,W.Sociallycompliantmobilerobotnavigationviainverse reinforcement learning. Int. J. Robot. Res. 2016, 35, 1289–1307. [CrossRef] [CrossRef]
Wulfmeier,M.;Ondruska,P.;Posner,I.MaximumEntropyDeepInverseReinforcementLearning.arXiv2015, arXiv:1507.04888.
Ho, J.; Ermon, S. Generative adversarial imitation learning. In Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016; pp. 4565–4573.
Baram, N.; Anschel, O.; Mannor, S. Model-Based Adversarial Imitation Learning. arXiv 2016, arXiv:1612.02179.
Wang, Z.; Merel, J.S.; Reed, S.E.; de Freitas, N.; Wayne, G.; Heess, N. Robust imitation of diverse behaviors. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA,
USA, 4–9 December 2017; pp. 5320–5329.
Li,Y.;Song,J.;Ermon,S.InferringtheLatentStructureofHumanDecision-MakingfromRawVisualInputs.
arXiv 2017, arXiv:1604.07316.
Tai, L.; Zhang, J.; Liu, M.; Burgard, W. Socially-Compliant Navigation through Raw Depth Inputs with
Generative Adversarial Imitation Learning. arXiv 2017, arXiv:1710.02543.
Stadie, B.C.; Abbeel, P.; Sutskever, I. Third-Person Imitation Learning. arXiv 2017, arXiv:1703.01703.
Wehner,M.;Truby,R.L.;Fitzgerald,D.J.;Mosadegh,B.;Whitesides,G.M.;Lewis,J.A.;Wood,R.J.Anintegrated
design and fabrication strategy for entirely soft, autonomous robots. Nature 2016, 536. [CrossRef]
Katzschmann, R.K.; de Maille, A.; Dorhout, D.L.; Rus, D. Physical human interaction for an inflatable manipulator. In Proceedings of the 2011 IEEE/EMBC Annual International Conference of the Engineering in
Medicine and Biology Society, Boston, MA, USA, August 30–3 September 2011; pp. 7401–7404.
Robotics 2019, 8, 4 36 of 36
Rogóz, M.; Zeng, H.; Xuan, C.; Wiersma, D.S.; Wasylczyk, P. Light-driven soft robot mimics caterpillar locomotion in natural scale. Adv. Opt. Mater. 2016, 4.
Katzschmann,R.K.;Marchese,A.D.;Rus,D.HydraulicAutonomousSoftRoboticFishfor3DSwimming. In Proceedings of the ISER, Marrakech and Essaouira, Morocco, 15–18 June 2014.
Katzschmann,R.K.;deMaille,A.;Dorhout,D.L.;Rus,D.Cyclichydraulicactuationforsoftroboticdevices. In Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, 9–14 October 2016; pp. 3048–3055.
DelPreto,J.;Katzschmann,R.K.;MacCurdy,R.B.;Rus,D.ACompactAcousticCommunicationModulefor Remote Control Underwater. In Proceedings of the WUWNet, Washington, DC, USA, 22–24 October 2015.
Marchese, A.D.; Onal, C.D.; Rus, D. Towards a Self-contained Soft Robotic Fish: On-Board Pressure
Generation and Embedded Electro-permanent Magnet Valves. In Proceedings of the ISER, Quebec City, QC,
Canada, 17–21 June 2012.
Narang, Y.S.; Degirmenci, A.; Vlassak, J.J.; Howe, R.D. Transforming the Dynamic Response of Robotic
Structures and Systems Through Laminar Jamming. IEEE Robot. Autom. Lett. 2018, 3, 688–695. [CrossRef]
[CrossRef]
Narang,Y.S.;Vlassak,J.J.;Howe,R.D.MechanicallyVersatileSoftMachinesThroughLaminarJamming.
Adv. Funct. Mater. 2017, 28, 1707136. [CrossRef] [CrossRef]
Kim, T.; Yoon, S.J.; Park, Y.L. Soft Inflatable Sensing Modules for Safe and Interactive Robots. IEEE Robot.
Autom. Lett. 2018, 3, 3216–3223. [CrossRef] [CrossRef]
Qi, R.; Lam, T.L.; Xu, Y. Mechanical design and implementation of a soft inflatable robot arm for safe
human-robot interaction. In Proceedings of the 2014 IEEE International Conference on Robotics and
Automation (ICRA), Hong Kong, China, 31 May–7 June 2014; pp. 3490–3495.
Zeng, H.; Wani, O.M.; Wasylczyk, P.; Priimagi, A. Light-Driven, Caterpillar-Inspired Miniature Inching
Robot. Macromol. Rapid Commun. 2018, 39, 1700224. [CrossRef] [CrossRef]
Banerjee, H.; Ren, H. Optimizing double-network hydrogel for biomedical soft robots. Soft Robot. 2017,
4, 191–201. [CrossRef] [CrossRef]
Henderson,P.;Islam,R.;Bachman,P.;Pineau,J.;Precup,D.;Meger,D.DeepReinforcementLearningthat
Matters. arXiv 2017, arXiv:1709.06560.
Vecerík, M.; Hester, T.; Scholz, J.; Wang, F.; Pietquin, O.; Piot, B.; Heess, N.; Rothörl, T.; Lampe, T.;
Riedmiller, M.A. Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems
with Sparse Rewards. arXiv 2017, arXiv:1707.08817v1.
Nair,A.;McGrew,B.;Andrychowicz,M.;Zaremba,W.;Abbeel,P.OvercomingExplorationinReinforcement
Learning with Demonstrations. arXiv 2017, arXiv:1709.10089.
Gao, Y.; Lin, J.; Yu, F.; Levine, S.; Darrell, T. Reinforcement Learning from Imperfect Demonstrations.
arXiv 2018, arXiv:1802.05313.
Zhu, Y.; Wang, Z.; Merel, J.; Rusu, A.; Erez, T.; Cabi, S.; Tunyasuvunakool, S.; Kramár, J.; Hadsell, R.;
de Freitas, N.; et al. Reinforcement and Imitation Learning for Diverse Visuomotor Skills. arXiv 2018,
arXiv:1802.09564.
Nichol, A.; Schulman, J. Reptile: A Scalable Metalearning Algorithm. arXiv 2018, arXiv:1803.02999.
⃝c 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license
(http://creativecommons.org/licenses/by/4.0/).
0 notes