Article Contents
Citation: V. Moya, , . Delayed teleoperation with force feedback of a humanoid robot. International Journal of Automation and Computing, doi:  10.1007/s11633-020-1267-7
Cite as: Citation: V. Moya, , . Delayed teleoperation with force feedback of a humanoid robot. International Journal of Automation and Computing , doi:  10.1007/s11633-020-1267-7

Delayed Teleoperation with Force Feedback of a Humanoid Robot

Author Biography:
  • Viviana Moya received M. Eng. degree in electronics and control from National Polytechnic School, Ecuador in 2016. She is a Ph. D. degree candidate in control systems engineering from National University of San Juan, Argentina with a DAAD scholarship from Germany. Her research interests include teleoperation systems and automatic control. E-mail: (Corresponding author) ORCID iD: 0000-0002-6064-6925

    Emanuel Slawiñski received the Ph. D. degree in control systems engineering from National University of San Juan (UNSJ), Argentina in 2006. He is a researcher at the National Council of Scientific and Technical Investigations of Argentina (CONICET) and a professor at the UNSJ. His research interests include delayed bilateral teleoperation systems, human factors, haptic feedback, human-robot interaction, and software development. E-mail:

    Vicente Mut received the Ph. D. degree in control systems engineering from National University of San Juan, Argentina in 1987. He is a professor at the UNSJ, developing research activities and teaching at the graduate and postgraduate programs at the Automatics Institute and the Department of Electronics and Automatics, National University of San Juan, Argentina. His research interests include robotics, manufacturing systems, adaptive control, and artificial intelligence applied to automatic control. E-mail:

  • Received: 2020-03-09
  • Accepted: 2020-11-09
  • Published Online: 2021-03-01
  • Teleoperation systems allow the extension of human capabilities to remote-control devices by providing the operator with conditions similar to those at the remote site through a communication channel that sends information from one site to the other. This article aims to present an analysis of the benefits of force feedback applied to the bilateral teleoperation of a humanoid robot with time-varying delay. As a control scheme, we link adaptive inverse dynamics compensation, balance control, and P+d like controllers. Finally, a test is performed where an operator simultaneously handles the locomotion (forward velocity and turn angle) and arm of a simulated 3D humanoid robot to do a pick-and-place task using two master devices with force feedback, where indexes such as time to complete the task, coordination errors, path tracking error, and percentage of successful tests are reported for different time-delays. We conclude with the results achieved.
  • 加载中
  • [1] M. Mihelj, J. Podobnik. Haptics for Virtual Reality and Teleoperation, Dordrecht, Netherlands: Springer, pp. 161−166, 2012.
    [2] E. Slawinski, V. A. Mut, P. Fiorini, L. R. Salinas.  Quantitative absolute transparency for bilateral teleoperation of mobile robots[J]. IEEE Transactions on Systems, Man, and Cybernetics − Part A: Systems and Humans, 2012, 42(2): 430-442. doi: 10.1109/TSMCA.2011.2159588
    [3] Q. W. Deng, Q. Wei, Z. X. Li.  Analysis of absolute stability for time-delay teleoperation systems[J]. International Journal of Automation and Computing, 2007, 4(): 203-207. doi: 10.1007/s11633-007-0203-4
    [4] Z. Chen, F. H. Huang, C. N. Yang, B. Yao.  Adaptive fuzzy backstepping control for stable nonlinear bilateral teleoperation manipulators with enhanced transparency performance[J]. IEEE Transactions on Industrial Electronics, 2020, 67(1): 746-756. doi: 10.1109/TIE.2019.2898587
    [5] S. Phukan, C. Mahanta.  A position synchronization controller for co-ordinated links (COOL) dual robot arm based on integral sliding mode: Design and experimental validation[J]. International Journal of Automation and Computing, 2020, (): -. doi: 10.1007/s11633-020-1242-3
    [6] Z. Chen, F. H. Huang, W. C. Sun, J. Gu, B. Yao.  RBF-neural-network-based adaptive robust control for nonlinear bilateral teleoperation manipulators with uncertainty and time delay[J]. IEEE/ASME Transactions on Mechatronics, 2020, 25(2): 906-918. doi: 10.1109/TMECH.2019.2962081
    [7] E. R. Westervelt, J. W. Grizzle, C. Chevallereau, J. H. Choi, B. Morris. Feedback Control of Dynamic Bipedal Robot Locomotion, Boca Raton, USA: CRC Press, 2007.
    [8] A. D. Ames, K. Galloway, K. Sreenath, J. W. Grizzle.  Rapidly exponentially stabilizing control Lyapunov functions and hybrid zero dynamics[J]. IEEE Transactions on Automatic Control, 2014, 59(4): 876-891. doi: 10.1109/TAC.2014.2299335
    [9] Q. Nguyen, K. Sreenath. L1 adaptive control for bipedal robots with control Lyapunov function based quadratic programs. In Proceedings of American Control Conference, IEEE, Chicago, USA, pp. 862−867, 2015.
    [10] A. Brygo, I. Sarakoglou, N. Tsagarakis, D. G. Caldwell. Tele-manipulation with a humanoid robot under autonomous joint impedance regulation and vibrotactile balancing feedback. In Proceedings of IEEE-RAS International Conference on Humanoid Robots, IEEE, Madrid, Spain, pp. 862−867, 2014.
    [11] F. Abi-Farrajl, B. Henze, A. Werner, M. Panzirsch, C. Ott, M. A. Roa. Humanoid teleoperation using task-relevant haptic feedback. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Madrid, Spain, pp. 5010−5017, 2018.
    [12] J. Ramos, A. Wang, S. Kim. A balance feedback human machine interface for humanoid teleoperation in dynamic tasks. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Hamburg, Germany, pp. 4229−4235, 2015.
    [13] K. Hongo, M. Yoshida, Y. Nakanishi, I. Mizuuchi, M. Inaba. Development of bilateral wearable device “Kento” for control robots using muscle actuator modules. In Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication, IEEE, Toyama, Japan, pp. 897−902, 2009.
    [14] J. J. O. Barros, V. M. F. dos Santos, F. M. T. P. da Silva. Bimanual haptics for humanoid robot teleoperation using ROS and V-REP. In Proceedings of IEEE International Conference on Autonomous Robot Systems and Competitions, IEEE, Vila Real, Portugal, pp. 174−179, 2015.
    [15] J. Ramos, S. Kim.  Dynamic locomotion synchronization of bipedal robot and human operator via bilateral feedback teleoperation[J]. Science Robotics, 2019, 4(35): eaav4282-. doi: 10.1126/scirobotics.aav4282
    [16] N. Karbasizadeh, M. Zarei, A. Aflakian, M. T. Masouleh, A. Kalhor.  Experimental dynamic identification and model feed-forward control of Novint Falcon haptic device[J]. Mechatronics, 2018, 51(): 19-30. doi: 10.1016/j.mechatronics.2018.02.013
    [17] M. Arbulú, D. Kaynov, L. Cabas, C. Balaguer.  The Rh-1 full-size humanoid robot: Design, walking pattern generation and control[J]. Applied Bionics and Biomechanics, 2009, 6(): 974354-. doi: 10.1080/11762320903123575
    [18] A. De Santis, P. Pierro, B. Siciliano. The multiple virtual end-effectors approach for human-robot interaction. Advances in Robot Kinematics, J. Lennarčič, B. Roth, Eds., Dordrecht, Netherlands: Springer, pp. 133−144, 2006.
    [19] B. Henze, M. A. Roa, C. Ott.  Passivity-based whole-body balancing for torque-controlled humanoid robots in multi-contact scenarios[J]. The International Journal of Robotics Research, 2016, 35(12): 1522-1543. doi: 10.1177/0278364916653815
    [20] S. Faraji, S. Pouya, C. G. Atkeson, A. J. Ijspeert. Versatile and robust 3D walking with a simulated humanoid robot (Atlas): A model predictive control approach. In Proceedings of IEEE International Conference on Robotics and Automation, IEEE, Hong Kong, China, pp. 1943−1950, 2014.
    [21] F. J. A. Chavez, J. Kangro, S. Traversaro, F. Nori, D. Pucci.  Contact force and joint torque estimation using skin[J]. IEEE Robotics and Automation Letters, 2018, 3(4): 3900-3907. doi: 10.1109/LRA.2018.2856914
    [22] C. C. Hua, X. P. Liu.  Delay-dependent stability criteria of teleoperation systems with asymmetric time-varying delays[J]. IEEE Transactions on Robotics, 2010, 26(5): 925-932. doi: 10.1109/TRO.2010.2053736
    [23] E. Nuño, R. Ortega, N. Barabanov, L. Basañez.  A globally stable PD controller for bilateral teleoperators[J]. IEEE Transactions on Robotics, 2008, 24(3): 753-758. doi: 10.1109/TRO.2008.921565
    [24] T. Q. Yang, W. M. Zhang, X. C. Chen, Z. G. Yu, L. B. Meng, Q. Huang.  Turning gait planning method for humanoid robots[J]. Applied Sciences, 2018, 8(8): 1257-. doi: 10.3390/app8081257
    [25] A. D. Ames.  Human-inspired control of bipedal walking robots[J]. IEEE Transactions on Automatic Control, 2014, 59(5): 1115-1130. doi: 10.1109/TAC.2014.2299342
    [26] C. L. Vaughan, B. L. Davis, J. C. O′Connor. Dynamics of Human Gait, 2nd ed., Champaign, USA: Human Kinetics Press, 1992.
    [27] M. W. Spong, S. Hutchinson, M. Vidyasagar. Robot Modeling and Control, New York, USA: John Wiley & Sons, Inc., 2005.
    [28] C. L. Shih, J. W. Grizzle, C. Chevallereau.  From stable walking to steering of a 3D bipedal robot with passive point feet[J]. Robotica, 2012, 30(7): 1119-1130. doi: 10.1017/S026357471100138X
    [29] J. B. Pomet, L. Praly.  Adaptive nonlinear regulation: Estimation from the Lyapunov equation[J]. IEEE Transactions on Automatic Control, 1992, 37(6): 729-740. doi: 10.1109/9.256328
    [30] C. Y. Cao, N. Hovakimyan. L1 adaptive controller for a class of systems with unknown nonlinearities: Part I. In Proceedings of American Control Conference, IEEE, Seattle, USA, pp. 4093−4098, 2008.
    [31] E. Nuño, L. Basañez, R. Ortega.  Passivity-based control for bilateral teleoperation: A tutorial[J]. Automatica, 2011, 47(3): 485-495. doi: 10.1016/j.automatica.2011.01.004
    [32] E. Slawiñski, V. Mut, D. Santiago.  PD-like controller for delayed bilateral teleoperation of wheeled robots[J]. International Journal of Control, 2016, 89(8): 1622-1631. doi: 10.1080/00207179.2016.1144234
    [33] D. Lee, O. Martinez-Palafox, M. W. Spong. Bilateral teleoperation of a wheeled mobile robot over delayed communication network. In Proceedings of IEEE International Conference on Robotics and Automation, IEEE, Orlando, USA, pp. 3298−3303, 2006.
  • 加载中
  • [1] Ravi Kumar Mandava, Pandu Ranga Vundavilli. Near Optimal PID Controllers for the Biped Robot While Walking on Uneven Terrains . International Journal of Automation and Computing,  doi: 10.1007/s11633-018-1121-3
    [2] Saad Kashem, Hutomo Sufyan. A Novel Design of an Aquatic Walking Robot Having Webbed Feet . International Journal of Automation and Computing,  doi: 10.1007/s11633-017-1075-x
    [3] M. Akhtaruzzaman, Amir A. Shafie, Raisuddin Md. Khan. Quasi-inverse Pendulum Model of 12 DoF Bipedal Walking . International Journal of Automation and Computing,  doi: 10.1007/s11633-016-1023-1
    [4] Xiao-Jing Wu, Xue-Li Wu, Xiao-Yuan Luo. Adaptive Neural Network Dynamic Surface Control for a Class of Nonlinear Systems with Uncertain Time Delays . International Journal of Automation and Computing,  doi: 10.1007/s11633-015-0945-3
    [5] Bin Wang,  Jun-Yong Zhai,  Shu-Min Fei. Output Feedback Tracking Control for a Class of Switched Nonlinear Systems with Time-varying Delay . International Journal of Automation and Computing,  doi: 10.1007/s11633-014-0848-8
    [6] Shafiqul Islam, Xiaoping P. Liu, Abdulmotaleb El Saddik, Lakmal Seneviratne, Jorge Dias. Control Schemes for Passive Teleoperation Systems over Wide Area Communication Networks with Time Varying Delay . International Journal of Automation and Computing,  doi: 10.1007/s11633-014-0771-z
    [7] Pan Wang,  Wei-Wei Sun. Adaptive H Control for Nonlinear Hamiltonian Systems with Time Delay and Parametric Uncertainties . International Journal of Automation and Computing,  doi: 10.1007/s11633-014-0802-9
    [8] Chang-Jin Xu,  Yu-Sen Wu. Chaos Control of a Chemical Chaotic System via Time-delayed Feedback Control Method . International Journal of Automation and Computing,  doi: 10.1007/s11633-014-0804-7
    [9] Hudyjaya Siswoyo Jo,  Nazim Mir-Nasiri. Development of Minimalist Bipedal Walking Robot with Flexible Ankle and Split-mass Balancing Systems . International Journal of Automation and Computing,  doi: 10.1007/s11633-013-0739-4
    [10] Ling-Feng Sang, Hong-Bo Wang, Dian-Fan Zhang, Zhen-Hua Tian, Fu-Hai Deng, De-Lei Fang. Application of Parallel Mechanism in Varistructured Quadruped/Biped Human-carrying Walking Chair Robot . International Journal of Automation and Computing,  doi: 10.1007/s11633-013-0741-x
    [11] Hassan A. Yousef, Mohamed Hamdy. Observer-based Adaptive Fuzzy Control for a Class of Nonlinear Time-delay Systems . International Journal of Automation and Computing,  doi: 10.1007/s11633-013-0721-1
    [12] Xiao-Fang Hong, Yu-Zhen Wang, Hai-Tao Li. Robust H Filter Design for Time-delay Systems with Saturation . International Journal of Automation and Computing,  doi: 10.1007/s11633-013-0733-x
    [13] Saïda Bedoui,  Majda Ltaief,  Kamel Abderrahim. New Results on Discrete-time Delay Systems Identification . International Journal of Automation and Computing,  doi: 10.1007/s11633-012-0681-x
    [14] Fei Li, Hua-Long Xie. Sliding Mode Variable Structure Control for Visual Servoing System . International Journal of Automation and Computing,  doi: 10.1007/s11633-010-0509-5
    [15] Lun Xie,  Zhi-Liang Wang,  Wei Wang,  Guo-Chen Yu. Emotional Gait Generation for a Humanoid Robot . International Journal of Automation and Computing,  doi: 10.1007/s11633-010-0064-0
    [16] Xin Li,  Xue-Ping Zhao,  Jie Chen. Controller Design for Electric Power Steering System Using T-S Fuzzy Model Approach . International Journal of Automation and Computing,  doi: 10.1007/s11633-009-0198-0
    [17] Mou Chen,  Chang-Sheng Jiang,  Qing-Xian Wu. Sensor Fault Diagnosis for a Class of Time Delay Uncertain Nonlinear Systems Using Neural Network . International Journal of Automation and Computing,  doi: 10.1007/s11633-008-0401-8
    [18] You-Qing Wang, Dong-Hua Zhou, Li-Heng Liu. Reliable Memory Feedback Design for a Class of Nonlinear Fuzzy Systems with Time-varying Delay . International Journal of Automation and Computing,  doi: 10.1007/s11633-007-0169-2
    [19] Qi-Wen Deng, Qing Wei, Ze-Xiang Li. Analysis of Absolute Stability for Time-delay Teleoperation Systems . International Journal of Automation and Computing,  doi: 10.1007/s11633-007-0203-4
    [20] H State Feedback Delay-dependent Control for Discrete Systems with Multi-time-delay . International Journal of Automation and Computing,  doi: 10.1007/s11633-005-0107-0
通讯作者: 陈斌,
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索

Figures (11)


Abstract Views (18) PDF downloads (7) Citations (0)

Delayed Teleoperation with Force Feedback of a Humanoid Robot

Abstract: Teleoperation systems allow the extension of human capabilities to remote-control devices by providing the operator with conditions similar to those at the remote site through a communication channel that sends information from one site to the other. This article aims to present an analysis of the benefits of force feedback applied to the bilateral teleoperation of a humanoid robot with time-varying delay. As a control scheme, we link adaptive inverse dynamics compensation, balance control, and P+d like controllers. Finally, a test is performed where an operator simultaneously handles the locomotion (forward velocity and turn angle) and arm of a simulated 3D humanoid robot to do a pick-and-place task using two master devices with force feedback, where indexes such as time to complete the task, coordination errors, path tracking error, and percentage of successful tests are reported for different time-delays. We conclude with the results achieved.

Citation: V. Moya, , . Delayed teleoperation with force feedback of a humanoid robot. International Journal of Automation and Computing, doi:  10.1007/s11633-020-1267-7
Citation: Citation: V. Moya, , . Delayed teleoperation with force feedback of a humanoid robot. International Journal of Automation and Computing , doi:  10.1007/s11633-020-1267-7
    • The methods for remotely controlling robots have evolved over time, and new research and developments have contributed to the fact that the common problems encountered in this area have been decreasing, while the efficiency and stability of communication between the human operator, the robot, and the environment have been improving considerably, allowing the human operator, through a variety of master devices, to explore remote environments and control a robot to complete a task while receiving many kinds of feedback through a bilateral communication channel, that links both sites[1]. Therefore, robot teleoperation allows a human operator, located at a local site, to send control commands to a robot at a remote site, which could be dangerous or inaccessible to a human, such as nuclear sites, fires, disaster scenarios, or even other planets while receiving important information that helps him to successfully complete the proposed task. However, the main problem with these systems is the presence of time delays which decrease the performance and transparency of the system[2, 3].

      On the other hand, techniques to include the various nonlinearities and uncertainties in the teleoperation system to achieve system stability, such as adaptive control[4, 5], neural networks[6], and sliding strategies can be applied to get a more robust compensation in front of parametric uncertainties and sensor noise, but the main studies in this area have been focused on manipulators, while the need for a robot capable of interacting with interfaces designed for humans is crucial. Teleoperated humanoid robots have been proposed as a potential solution in different applications such as nuclear plant inspection and telemedicine, among others. Designing controllers applicable to humanoid robots is challenging and time-consuming in the design of control systems for bipedal motion, such as zero moment point (ZMP)[7], hybrid zero dynamics (HZD)[8], artificial intelligence or adaptive control techniques[9].

      Teleoperation can be performed in two ways, either unilateral or bilateral. Our work is focused on bilateral teleoperation, especially on the use of the force feedback and its effects and advantages during the teleoperation. Many studies use force feedback in different ways to communicate data to the operator depending on the task on hand. The work presented in [10] uses a vibrotactile belt to inform the operator about the position of the ZMP within the humanoid support polygon. Another work that builds upon the previous paper was presented in [11], in which a haptic interface for humanoid teleoperation is introduced and the effects of the manipulation task on the balance of the robot are considered. The work presented in [12] uses a kinesthetic system that applies forces on the operator′s waist, this haptic force is based on the translation of the robot′s center of pressure (CoP), as a direct measurement of balancing stability using the support polygon. Another way to feedback force to the operator is through stiffness, as shown in [13], where a bilateral wearable device uses adjustable muscle actuator modules in order to control a robot, sense the external force, and transmit the motion of contact with the environment. Popular devices used to feedback force to the operator are haptic joysticks such as the PHANToM Omni haptic used in [14] to implement a general strategy for generating the force feedback based on the ZMP method for a walking robot. Considering operations in dynamically stable conditions, the ZMP coincides with the CoP. Thus, its location is used to predict the dynamic behavior of the system and define the feedback force. More recent studies such as [15] propose using force feedback as a proportional value to the relative instantaneous velocity between the operator and robot, and this force is applied to the torso of the operator.

      As it has been shown in those papers, most studies use force feedback mainly to inform the human operator about the equilibrium of the robot using the ZMP criterion for static walks. Instead, our work uses force feedback during locomotion (dynamic walks) and manipulation tasks of a humanoid robot both based on the synchronism error, where the main contribution is to analyze how much the force feedback helps the delayed bilateral teleoperation system of a humanoid robot. The control scheme implemented is formed by an internal loop of adaptive reverse dynamics control plus a set of external loop algorithms based on controllers type P + d (Proportional plus damping), proportional differential (PD), and P + d + Fe (Fe is an external force), where the damping is established as a function of the delay time. Additionally, a balance dynamic control to hold the 3D posture adequately for the locomotion and manipulation tasks is developed. A test for simultaneous teleoperation of the locomotion and arm of a humanoid robot simulated in virtual robot experimentation platform (VREP) to perform a pick and place task was performed with a single operator driving two joysticks, the first for the lower body and the second for the humanoid arm. The human operator receives force feedback that improves the perception of the robot′s walk, and the control of the arm, helping him to regulate the commands he sends and preventing rapid movements of the master as the time delay increases. To report the achieved result, we evaluated the time to complete the task, coordination errors, path tracking error, and percentage of successful tests, comparing the use of force feedback against the non-force feedback case for delayed teleoperation of a humanoid robot.

      This document is organized as follows. Section 2 presents the master and slave models. Section 3 presents the assumptions and properties used. In Section 4, a bilateral teleoperation of a humanoid robot is proposed and the stability analysis for the system of bilateral teleoperation is presented in Section 5. Then, Section 6 shows the simulations results obtained when a human operator handles a humanoid NAO in the VREP simulation environment. And finally, the conclusions are presented in Section 7.

    • The master can have a serial or a parallel type configuration. In this paper, a Novint Falcon is used, which has a kinematic model described in [16] and its model can be represented in the task space. Here, the master model $ \left( m\right) $ for locomotion $ \left( l\right) $ and manipulation $ \left( m\right) $ are detailed:

    • $ {{{M}}_{{mi}}}{{\ddot{{x}}}_{{mi}}}+{{{C}}_{{mi}}}{{\dot{{x}}}_{{mi}}}+{{{g}}_{{mi}}} = {{{f}}_{{mi}}}+{{{f}}_{{h}}} $


      where the sub-index i can be equal to $ l $ for locomotion or $ m $ for manipulation, ${{{x}}_{{ml}}} ={\left[ {\begin{array}{*{20}{c}} {{x_{mv}}}&{{x_{m\delta }}} \end{array}} \right]^{\rm{T}}} \in {{\bf{R}}^{nml \times 1}}$ and ${{\dot{{x}}}_{{ml}}}$ are the master′s position and velocity for locomotion, ${{{x}}_{{mm}}} \in {{{\bf{R}}}^{nmm\times 1}}$ and ${{\dot{{x}}}_{{mm}}}$ are the master′s position and velocity for manipulation, ${{{M}}_{{m\left( l,m\right)}}}\in {{{\bf{R}}}^{n {{m\left( l,m\right)}}\times n {{m\left( l,m\right)}}}}$ is the inertia matrix, ${{{C}}_{{m\left( l,m\right)}}}$ is the matrix representing centripetal and coriolis forces, ${{{g}}_{{m\left( l,m\right)}}}\left( {{{x}}_{{m\left( l,m\right)}}} \right)$ is the gravitational force, ${{{f}}_{{h}}}$ is the force caused by the human operator and ${{{f}}_{{m\left( l,m\right)}}}$ is the control force applied to the masters which will be computed by the controllers.

    • For the kinematic model of a humanoid robot, different revised techniques are presented[17, 18]. This paper analyzes the model described in [18], where the humanoid has been modeled as the combination of four kinematic chains that share the same starting point called “ROOT”. With this, it is possible to construct the Denavit Hantenberg algorithm of all the kinematic chains respect to the reference frame of the root[18], as shown in Fig. 1.

      Figure 1.  Representation of all the joints

      The kinematic model is defined as

      $ {{v}} = {{J_{A{\rm{U}}}}}\dot{{q}} $


      $ \left[ {\begin{array}{*{20}{c}} \dot{{x}}_{{r}}\\ \dot{{x}}_{{sl}} \\ \dot{{x}}_{{nsl}} \\ \dot{{x}}_{{aa}} \\ \dot{{x}}_{{pa}}\end{array}} \right] \!\!\!=\!\!\! \left[ {\begin{array}{*{20}{c}} {{J}}_{{r}}& 0& 0 & 0 &0\\ {{J}}_{{sl1}}& {{J}}_{{sl2}} & 0 & 0 & 0\\ {{J}}_{{nsl1}}& 0 & {{J}}_{{nsl2}} & 0 & 0\\ {{J}}_{{aa1}}& 0 & 0 & {{J}}_{{aa2}} & 0\\ {{J}}_{{pa1}}& 0 & 0 & 0 & {{J}}_{{pa2}} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} \dot{{q}}_{{r}}\\ \dot{{q}}_{{sl}} \\ \dot{{q}}_{{nsl}} \\ \dot{{q}}_{{aa}}\\ \dot{{q}}_{{pa}}\end{array}} \right] $


      where ${{v}}$ represents the velocities of the final effectors in Cartesian coordinates, $ {{J_{AU}}} $ is the Jacobian augmented and $ \dot{{q}} $ represents the joint speeds. The acronym $ {{r}} $ is the root, $ {{sl}} $ is the stance leg, $ {{nsl}} $ is the non-stance leg, $ {{aa}} $ is the active arm and $ {{pa}} $ is the passive arm. The joints of the humanoid robot are defined as: qr_nsa is the roll non-stance ankle, qr_sa is the roll stance ankle, qp_nsa is the pitch non-stance ankle, qp_sa is the pitch stance ankle, qp_nsk is the pitch non-stance knee, qp_sk is the pitch stance knee, qp_nsh is the pitch non-stance hip, qp_sh is the pitch stance hip, qy_nsh is the yaw non-stance hip, qy_sh is the yaw stance hip, qr_nsh is the roll non-stance hip, qr_sh is the roll stance hip, qr_ash is the roll active shoulder, qr_psh is the roll passive shoulder, qp_ash is the pitch active shoulder, qp_psh is the pitch passive shoulder, qy_nel is the yaw active elbow, qy_pel is the yaw passive elbow, qr_ael is the roll active elbow and qr_pel is the roll passive elbow.

    • In the state of the art, there are different techniques for modeling a humanoid robot[19]. In this paper, we take the motion equation for a floating base humanoid robot described by[19, 20]

      $ {{M}}\left( {{q}} \right){\ddot{{q}}} + {{C}}\left( {{{q}},{\dot{{q}}}} \right){\dot{{q}}} + {{g}} = {{S}}{{\tau }} + {{{\tau }}_{{{ext}}}}. $


      with $ n $ representing the total number of joints of the robot, the variables are defined as: ${{q}} \in {{{\bf{R}}}}^{n+6}$ represents all degrees of freedom of the humanoid robot plus the degrees of freedom of the floating base (extended model), ${{M}} \left( q\right) \in {{{\bf{R}}}}^{\left( n+6\right)\times\left( n+6\right) }$ represents the inertia matrix of the extended model, ${{C}} \left( q,\dot{q} \right) \in {{{\bf{R}}}}^{\left( n+6\right)\times\left( n+6\right) }$ represents the centripetal and Coriolis matrix of the extended model, ${{g}} \in {{{\bf{R}}}}^{n+6}$ is the gravity vector, ${{S}} \in {{{\bf{R}}}}^{\left( n+6\right)\times \left( n+6\right)}$ is the driven joint selection matrix of the extended model, ${{\tau}} \in {{{\bf{R}}}}^{n+6}$ is the vector of powered torsion torques of the extended model, ${{\tau_{ext}}} \in {{{\bf{R}}}}^{n+6}$ and ${{\tau_{ext}}} = {{J_{AU}}}^{\rm{T}}{{F_{e}}}$ are the generalization of external forces, where ${{J_{AU}}}^{\rm{T}}$ is the augmented Jacobian and $ {{F_{e}}} $ represents the external forces. The dynamic humanoid model (4) between successive impacts can be represented in a state space as

      $ \dot{{x}} = {{f}}({{x}})+{{g_{l}}}\left( {{x}}\right){{u_{l}}}+{{g_{m}}}\left( {{x}}\right){{u_{m}}}+{{g_{e}}}\left( {{x}}\right){{F_{e}}} $


      where ${{x}}\!=\! \left[ {{q}},{\dot{{q}}}\right]^{{\rm{T}}}$, ${{f}}({{x}}) \!\!=\!\! \left[ {\begin{array}{*{20}{c}} {\mathop {{q}}\limits^. }\\ {{{{M}}^{ - {{1}}}}\left( {{q}} \right)\left[ { - {{C}}\left( {{{q}},{\dot{{q}}}} \right){\dot{{q}}} - {{g}}\left( {{q}} \right)} \right]} \end{array}} \right]$, ${{{g}}_{{l}}}\left( {{x}} \right) = \left[ {\begin{array}{*{20}{c}} 0\\ {{{{M}}^{ - {{1}}}}{{{B}}_{{l}}}} \end{array}} \right]$, ${{{g}}_{{m}}}\left( {{x}} \right) = \left[ {\begin{array}{*{20}{c}} 0\\ {{{{M}}^{ - {{1}}}}{{{B}}_{{m}}}} \end{array}} \right]$, ${{{g}}_{{e}}}\left( {{x}} \right) = \left[ {\begin{array}{*{20}{c}} 0\\ {{{{M}}^{ - {{1}}}}{{J}}_{{{AU}}}^{{{\rm{T}}}}\left( {{q}} \right)} \end{array}} \right]$, ${{{F}}_{{e}}} = \left[ {\begin{array}{*{20}{c}} {{{{F}}_{{{el}}}}}\\ {{{{F}}_{{{em}}}}} \end{array}} \right]$, $ {{F_{el}}} $ is the external force of locomotion, and $ {{F_{em}}} $ is the external force of manipulation. In general, the external force can be obtained using force sensors, or estimated with distributed tactile skin sensors[21].

    • Properties 1−3 and Assumptions 1−3 will be used in this paper[22, 23]:

      Property 1. The inertia matrices $ {{{M}}_{{ml}}}\left( {{{x}}_{{ml}}} \right) $ and $ {{{M}}_{{mm}}}\left( {{{x}}_{{mm}}} \right) $ are symmetric positive defined.

      Property 2. The matrices ${{\dot{{M}}}_{{ml}}}\left( {{{x}}_{{ml}}} \right)- 2{{{C}}_{{ml}}}\left( {{{x}}_{{ml}}}, {{{\dot{{x}}}}_{{ml}}} \right)$ and $ {{\dot{{M}}}_{{mm}}}\left( {{{x}}_{{mm}}} \right)-2{{{C}}_{{mm}}}\left( {{{x}}_{{mm}}},{{{\dot{{x}}}}_{{mm}}} \right) $ are skew-symmetric, ie., ${{\dot{{M}}}_{{ml}}}\left( {{{x}}_{{ml}}}\right) = {{{C}}_{{ml}}}\left( {{{x}}_{{ml}}},{{{\dot{{x}}}}_{{ml}}}\right)+{{{C}}_{{ml}}^{{{\rm{T}}}}}\left( {{{x}}_{{ml}}},{{{\dot{{x}}}}_{{ml}}} \right)$ and ${{\dot{{M}}}_{{mm}}}\left( {{{x}}_{{mm}}}\right) = {{{C}}_{{mm}}}\left( {{{x}}_{{mm}}},{{{\dot{{x}}}}_{{mm}}}\right)+{{{C}}_{{mm}}^{{{\rm{T}}}}}\left( {{{x}}_{{mm}}},{{{\dot{{x}}}}_{{mm}}} \right)$.

      Property 3. If ${{{\dot{{x}}}}_{{ml}}}\;{\rm{and}}\;{{{\ddot{{x}}}}_{{ml}}}$ are bounded, then $ {{\dot{{C}}}_{{ml}}} $ is bounded. If ${{{\dot{{x}}}}_{{mm}}}\;{\rm{and}}\;{{{\ddot{{x}}}}_{{mm}}}$ are bounded, then $ {{\dot{{C}}}_{{mm}}} $ is bounded.

      Assumption 1. The communication channel adds a forward time delay $ {{h}_{1}} $ (from master to slave) and a backward time delay $ {{h}_{2}} $ (from slave to master). These delays are time-varying, bounded and asymmetric. Therefore, there are positive values for $ {{\bar{h}}_{1}} $ and $ {{\bar{h}}_{2}} $ such as $ 0\le {{h}_{1}}\left( t \right)\le {{\bar{h}}_{1}} $ and $ 0\le {{h}_{2}}\left( t \right)\le {{\bar{h}}_{2}} $ for all $ t $.

      Assumption 2. The human operator has finite energy:

      $ {{{E}}_{{h}}} = \varphi_{h}+\int_{0}^{t}-{{{\dot{{x}}}}_{{mi}}}^{{{\rm{T}}}}\left( \sigma \right){{F_{h}}}\left( \sigma \right){\rm{d}}\sigma >0. $


      Assumption 3. The environment is represented by a finite energy solution:

      $ {{{E}}_{{e}}} = \varphi_{e}+\int_{0}^{t}-{{{\dot{{y}}}}_{{sm}}}^{{{\rm{T}}}}\left( \sigma \right){{F_{em}}}\left( \sigma \right){\rm{d}}\sigma >0 $


      where $ \varphi_{h} $ and $ \varphi_{e} $ are positive values and it is assumed that ${{F_{el}}} $ and ${{F_{em}}} \in L_{\infty}$.

    • This paper analyzes the effect of force feedback over the teleoperation of forwarding speed and turn angle of a humanoid robot (locomotion) and its active arm (manipulation). The implemented control scheme and their main parts are shown in Fig. 2.

      Figure 2.  General scheme of the teleoperation control of the humanoid robot

      1) On the local site, there is a human operator that handles two joysticks, one for locomotion control and the other for manipulation control, both have force feedback which improves the perception for teleoperating the entire humanoid robot, preventing rapid movements of the master depending on time delay, as well as leading the human operator hand to directions of fewer errors between the master and humanoid robot.

      2) Walk and turn references: For the control of the forward speed, a foot analysis based on a standard human walk in Cartesian coordinates is performed to generate the human-inspired knee and hip references. The turn angle of the humanoid robot is set based on [24], where the turn is carried out during the double support phase.

      3) Balance control of the torso is carried out depending on the hip speed and a static force analysis.

      4) A cascade control structure is applied, using an adaptive inverse dynamics control internal loop plus a set of external loop algorithms based on P+d like controllers, where the damping is set depending on the time delay.

    • First, it is necessary to define the errors between successive impacts to manage the outputs of the robot to be able to control the signals of forwarding velocity, turn angle, and Cartesian coordinates of the end effector of the active arm to follow the generated references by the human operator, whose errors are defined below.

    • The forward velocity error is defined by

      $ \dot{\tilde{y}}_{1}\left ( {\dot{{q}}} \right ) = \dot{y}_{1}\left ( {{\alpha}} \right )- v_{hip}\left ( {\dot{{q}}} \right ) $


      where $ \dot{y}_{1} $ is the forward velocity reference, given by

      $ \dot{y}_{1}\left ( {{\alpha}} \right ) = k_{gl}x_{mv}\left ( t-h_{1} \right) = v_{ref}. $


      $ v_{hip} $ is the real velocity of the humanoid robot and it is obtained through a linearization-based procedure used in [25], where the hips position depends on the angle of the stance foot $ q_{sf} $ and the angle of the stance knee qp_sk, and its derivative is computed by

      $ v_{hip}\left ( {\dot{{q}}} \right ) = -a_{1}\dot{q}_{sf}-a_{2}\dot{q}_{sf}-a_{2}\dot{q}_{p{{\_}}sk} $


      where $a_{1}\;{\rm{and}}\;a_{2}$ are the leg link lengths.

    • The gait length error is represented as

      $ {\tilde{{y}}}_{{2}}\left ( {{q}} \right ) = {{y_{2}}}\left ( t,{{\alpha}} \right)-{{y_{r,2}}}\left ( t \right ). $


      For angles references ${{y_{2}}}\left ( t,{{\alpha}} \right )$, the standard human walk is analyzed where the foot trajectory in Cartesian coordinates is taken as a pattern. At all times, at least one of the two feet is in contact with the ground. According to [26], each limb spends approximately $ 40\% $ of the walking cycle as a single support leg, $ 20\% $ as part of the double support and $ 40\% $ as a swinging leg. Once the foot trajectory data has been analyzed, human-inspired references are obtained through an approximation by a Bezier curve plus a straight line. The Bezier function depends on the velocity reference, whose coefficients are found by optimization. The goal is to ensure that the trajectory of the robot′s foot is as close as possible to the typical trajectory of a person (scaled to the size of the robot used). It is also important to verify the constraints of the partial zero dynamics surface (PHZD), given in Section 4.4. To obtain the references of stance knee angle and stance hip angle during a walking cycle, inverse kinematics[27] is used, which is calculated from the generated Cartesian trajectory. From this, the angles qp_sh and qp_sk can be obtained. Taking into account the percentages with respect to the period of the single support and double phase and considering that the legs are symmetrical, angles qp_nsh and qp_nsk are calculated using a phase shift of the angles qp_sh and qp_sk. Therefore, the references are ${{y}}_{{2}}\left ( t,{{\alpha}} \right ) \!=\! \left[ {{q_{p\_nsk}}} \quad{{q_{p\_sk}}} \quad{{q_{p\_nsh}}}\quad{{q_{p\_sh}}} \right]^{\rm{T}}$. And ${{y_{r,2}}}\left ( t \right)$ represents the real angles belonging to the sagittal plane, such as ${{y_{r,2}}}\left ( t \right ) = \left[ \begin{array}{*{20}{c}} {{q_{p\_nskr}}}\;\;\;{{q_{p\_skr}}}\;\;\; {{q_{p\_nshr}}}\;\;\; {{q_{p\_shr}}} \end{array} \right]^{\rm{T}}$.

    • The turn angle error is described as

      $ {\tilde{y}}_{3}\left ( {{q}} \right ) = {y}_{3}\left (t \right )-\delta\left (t \right ) $


      where $ \delta\left (t \right ) = {\rm{f}}$ (qy_sh, qy_nsh) is the real angle of rotation of the humanoid robot and the reference $ {y}_{3}\left( t\right) $ is given by

      $ {y}_{3}\left ( t\right ) = k_{g}x_{m\delta}\left ( t-h_{1} \right) = \delta_{ref}\left( t\right). $


      This yaw rotation reference does not affect the control in the sagittal plane due to the invariance property given in Proposition 1 of [28] where the within-stride feedback does not depend on the yaw orientation of the robot. This reference is mapped to (qy_sh, qy_nsh) using the proposal described in Fig. 15 of [24].

    • The manipulation error in Cartesian coordinates is represented as

      $ {\tilde{{y}}}_{{4}} = {{y}}_{{4}}-{{y_{sm}}} $


      where ${{y}}_{{4}}$ is the manipulation reference of the final effector in Cartesian coordinates, given by

      $ {{y}}_{{4}}(t) = {{k_{gm}x_{mm}}}\left ( t-h_{1} \right). $


      And ${{y_{sm}}}$ is obtained using the direct kinematics to calculate the actual position of the final effector (active hand) concerning a coordinate system that is taken as a reference (shoulder) and knowing the values of the joints and the geometric parameters of the robot elements. Finally, the coordination error of the delayed bilateral tele-operation system is defined by

      $ {{e}}\left( {{x}} \right)= \left[ {\begin{array}{*{20}{c}} {{{\dot{\tilde{y}}}_1}}&{{{{\tilde{{y}}}}_{{2}}}}&{{{\tilde y}_3}}&{{{{\tilde{{y}}}}_{{4}}}} \end{array}} \right]. $

    • For the balance control, the torso inclination is changed depending on the tasks of locomotion and manipulation.

      $ \theta_{torso} = \theta_{torsol}+\theta_{torsom} $


      where $ \theta_{torsol} $ is calculated using a linear function that depends on the reference forward speed. This is done for the torso to compensate the dynamic forces caused by different robot forward velocities:

      $ \theta_{torsol} = k_t\left(av_{ref}+b \right). $


      This represents $ \theta_{torsol} $ is a linear function of the reference forward velocity $ v_{ref} $ set by the human operator, where $ a $ is the slope of the line, $ b $ is the interjection with $ \theta_{torsol} $ axis and $ k_t $ is a proportional constant. $ \theta_{torsom} $ is calculated from the static force′s analysis to keep the torso at an angle of inclination such that it compensates the forces caused by the position of the arms and the object gripped, as shown in Fig. 3.

      Figure 3.  Static analysis of torques

      $ \sum \tau_{aa}+\sum \tau_{nsl} = \sum \tau_{\left( torso+head\right) }\left( \theta_{torsom}\right) $


      where the subscript $ aa $ is the active arm and the subcript $ nsl $ indicates the non-stance leg. Therefore, the ankle references will be used to indirectly control the balance of the torso and these references are related to the other joints as

      $ \theta_{ankle} = \theta_{torso}-\theta_{hip}-\theta_{knee}. $


      The references $ q_{sa} $ and $ q_{nsa} $ are given by $q_{sa} = q_{nsa} = \theta_{ankle}$.

    • The dynamic model (5) is used to represent the error dynamics ${{y}} \left ( {{x}}\right ) = {{h}} \left ( {{x}}\right )$ through the Lie derivative notation as follows:

      $ {\ddot{{y}}}\left ( {{x}}\right ) = L_{f}^{2}{{h}}\left ( {{x}}\right )+L_{g}L_{f}{{h}}\left ( {{x}}\right ){{u}}\left ( {{x}}\right ) + L_{g_{e}}L_{f}{{h}}\left ( {{x}} \right ) {{F_{e}}} $


      where ${ {h}}\left ({ {x}}\right)$ is based on the error vector (16) and this is adapted to work simultaneously with speed and position errors. Then, we define ${{h}}\left( {{x}}\right)= \left[ {\begin{array}{*{20}{c}} {\int {{{\dot {\tilde{y}}}_1}} }&{{{\widetilde {{y}}}_{{2}}}}&{{{\tilde y}_3}}&{{{\widetilde {{y}}}_{{4}}}} \end{array}} \right]$, and ${{g}}\left( {{x}} \right) = \left[ {\begin{array}{*{20}{c}} {{{g}}_{{l}}^{{{\rm{T}}}}\left( {{x}} \right)\quad{{g}}_{{m}}^{{{\rm{T}}}}\left( {{x}} \right)} \end{array}} \right]$. The feedback linearization controller based on [8] is stated as

      $ {{u}}\left ( {{x}}\right ) \!=\! -L_{g}L_{f}{{h}}\left ( {{x}} \right )^{-1}\left (L_{f}^{2}{{h}}\left ( {{x}}\right )+ L_{g_{e}}L_{f}{{h}}\left ( {{x}} \right ) {{ F_{e}}} + {{\upsilon}} \left ( {{x}}\right ) \right ). $


      A disadvantage of (22) is that an accurate dynamic model of the system is required. If we consider uncertainties in the dynamics and assume that the functions $f\left({{x}}\right)\;{\rm{and}}\; g\left ( {{x}}\right )$ of (5) are estimated, the controller must be designed based on the functions $\hat{f} \left({{x}}\right)\;{\rm{and}}\; \hat{g} \left ( {{x}}\right )$. Thus, the law of control (22) is reformulated as[9]

      $ {{u}}\left ( {{x}}\right ) \!=\! -L_{\hat{g}}L_{\hat{f}}{{h}}\left ( {{x}} \right )^{-1}\left (L_{\hat{f}}^{2}{{h}}\left ( {{x}}\right )+ L_{\hat{g}_{e}}L_{\hat{f}}{{h}}\left ( {{x}} \right ) {{F_{e}}}+ {{\upsilon}} \left ( {{x}}\right ) \right ). $


      Therefore, by replacing (23) in (21),

      $ {\ddot{{y}}}\left ( {{x}}\right ) = {{\upsilon}} \left ( {{x}}\right )+ {{\theta}} $



      $ {{\theta}} = \Delta_{1}+\Delta_{2}+\Delta_{3}{{\upsilon}}, \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; $

      $ \Delta_{1} = L_{f}^{2}{{h}}\left ( {{x}}\right )-L_{g}L_{f}{{h}}\left ( {{x}} \right )\left( L_{\hat{g}}L_{\hat{f}}{{h}}\left ( {{x}} \right )\right)^{-1}L_{\hat{f}}^{2}{{h}}\left ( {{x}}\right) $

      $\begin{split} \Delta_{2} =& L_{g_{e}}L_{f}{{h}}\left ( {{x}} \right ) {{F_{e}}}-\\ &L_{g}L_{f}{{h}}\left ( {{x}} \right )\left( L_{\hat{g}}L_{\hat{f}}{{h}}\left ( {{x}} \right )\right)^{-1}L_{\hat{g}_{e}}L_{\hat{f}}{{h}}\left ( {{x}} \right ) {{F_{e}}}\;\;\;\;\;\;\; \end{split}$

      $ \Delta_{3} = -L_{g}L_{f}{{h}}\left ( {{x}} \right )\left( L_{\hat{g}}L_{\hat{f}}{{h}}\left ( {{x}} \right )\right)^{-1}L_{\hat{f}}^{2}{{h}}\left ( {{x}}\right )-I. \;\;\;\;\;\; $

      To compensate the uncertainties, a combined controller is proposed with ${{\upsilon}} = {{\upsilon}}_{1}+{{\upsilon}}_{2}$, where the first part ${{\upsilon}}_{1}$ allows following the desired reference model considering a perfect knowledge of inverse dynamics and the second part ${{\upsilon}}_{2}$ compensates the nonlinear uncertainty ${{\theta}}$. Next, from (24), the closed-loop dynamics of the error in the state space is represented as

      $ {\dot{{\omega}}} = {{F}}{{\omega}}+{{\upsilon}}_{{1}}+\left({{\upsilon}}_{{2}}+ {{\theta}} \right) $


      where ${{\omega}} = \left[ \dot{\tilde{y}}_{1}, \tilde{y}_{2}, \dot{\tilde{y}}_{2}, \tilde{y}_{3}, \dot{\tilde{y}}_{3}, \tilde{y}_{4}, \dot{\tilde{y}}_{4} \right]^{\rm{T}}$, ${{\upsilon}}_{{1}} = {{K}}{{\omega}}+ {{HF_e}},$

      $\begin{split} &{{F}} = \left[ {\begin{array}{*{20}{c}} 0&0&0&0&0&0&0\\ 0&0&I&0&0&0&0\\ 0&0&0&0&0&0&0\\ 0&0&0&0&I&0&0\\ 0&0&0&0&0&0&0\\ 0&0&0&0&0&0&I\\ 0&0&0&0&0&0&0 \end{array}} \right]\\ &{{H}} = \left[ {\begin{array}{*{20}{c}} 0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0\\ 0&0&0&0&0&0&0\\ 0&0&0&0&0&0&{ - {{{M}}_{{{sm}}}}^{ - 1}} \end{array}} \right] \end{split}$

      $\begin{split} \\{}{{K}} = \left[ {\begin{array}{*{20}{c}} { - \dfrac{{{k_{{v_{hip}}}}}}{{1 + {\sigma _{real}}}}}&0&0&0&0\\ 0&0&0&0&0\\ 0&{ - \dfrac{1}{{{{{\varepsilon }}_{{2}}}^2}}}&{ - \dfrac{1}{{{{{\varepsilon }}_{{2}}}}}}&0&0\\ 0&0&0&0&0\\ 0&0&0&{ - {I^{ - 1}}{k_\delta }}&{ - {I^{ - 1}}\tau }\\ 0&0&0&0&0\\ 0&0&0&0&0 \end{array}} \right.\\ \left. {\begin{array}{*{20}{c}} &0&0\\ &0&0\\ &0&0\\ &0&0\\ &0&0\\ &0&0\\ &{ - {{M}}_{{{sm}}}^{ - {{1}}}{{{k}}_{{{sm}}}}}&{ - {{{M}}_{{{sm}}}}^{ - 1}{{{\sigma }}_{{{sm}}}}} \end{array}} \right] \end{split}$

      and $\sigma_{real},\;\tau$ and $ {\sigma_{sm}} $ are the injected damping, $ k_{v_{hip}},k_{\delta} $ and ${{ {k}_{sm}}}$ are proportional gains, $ \epsilon_{2} $ is a control gain, $ I $ and ${{{M}_{sm}}}$ are the inertia matrices of the reference model, ${{\theta}} = {{\theta}}\left( {{\omega}},t \right)$ defines the non-linear uncertainty. The forward speed and turning angle references are kept constant between impacts.

      From [9], we can find out ${{\psi}} \left( t\right)$, ${{\phi}} \left( t\right)$ and ${{\vartheta}} \left( t\right)$ such that:

      $ {{\theta}} \left( {{\omega}},t \right) = {{\psi}} \left( t\right)\left \| {{\omega}} \right \|+ {{\phi}} \left( t\right) \left \| {{F_e}} \right \| + {{\vartheta}} \left( t\right). $


      From (25), the state predictor is expressed as follows:

      $ \dot{\hat{{{\omega}}}} = {{F}} \hat{{{\omega}}}+\hat{{{\upsilon}}}_{1}+ \left({{\upsilon}}_{2}+ \hat{{\theta}} \right) $



      $ \hat{{\theta}} = \hat{{{\psi}}}\left( t\right)\left \| {{\omega}} \right \|+ \hat{{{\phi}}} \left( t\right) \left \| {{F_e}} \right \| + \hat{{{\vartheta}}} \left( t\right) $


      and ${{\upsilon}}_{2}$ is chosen as ${{\upsilon}}_{2} = -\hat{{\theta}}$. Defining $\tilde{{{\omega}}} = \hat{{{\omega}}}-{{\omega}}$, its evolution can be written from (25) and (27) as follows:

      $ \begin{split} &\dot{\tilde{{{\omega}}}} = {{F}}\tilde{{{\omega}}}+\tilde{{{\upsilon}}}_{1}+ \left( \tilde{{{\psi}}}\left \| {{\omega}} \right \|+ \tilde{{{\phi}}} \left( t\right) \left \| {{F_e}} \right \| + \tilde{{{\vartheta}}}\right) \\ &\dot{\tilde{{{\omega}}}} = {{F}}\tilde{{{\omega}}}+\tilde{{{\upsilon}}}_{1}+ \tilde{{\theta}} \end{split}$


      where $ \tilde{{{\upsilon}}}_{1} = \hat{{{\upsilon}}}_{1}-{{{\upsilon}}}_{1} $, $\tilde{{\theta}} = \hat{{\theta}}-{{\theta}} = [ \tilde{{\theta}}_v \;\; 0 \;\; \tilde{{\theta}}_g \;\; 0 \;\; \tilde{{\theta}}_\delta \;\; 0 \;\; \tilde{{\theta}}_m ]^{{\rm{T}}}$, $ \tilde{{{\psi}}} = \hat{{{\psi}}}-{{\psi}} $, $ \tilde{{{\phi}}} = \hat{{{\phi}}}-{{\phi}} $ and $ \tilde{{{\vartheta}}} = \hat{{{\vartheta}}}-{{\vartheta}} $. As a result, $ {{\theta}} $ is estimated indirectly through $ {{\psi}} $, $ {{\phi}} $ and $ {{\vartheta}} $, and the values of $ \hat{{{\psi}}} $, $ \hat{{{\phi}}} $ and $ \hat{{{\vartheta}}} $ are computed by the following adaptations laws based on the projection operators[29]:

      $ \begin{split} &\dot{\hat{{{\psi}}}} = {{\Gamma}}{{Proj}} \left( \hat{{{\psi}}}, {{y}}_{\psi}\right) \\ &\dot{\hat{{{\phi}}}} = {{\Gamma}}{{Proj}} \left( \hat{{{\phi}}}, {{y}}_{\phi}\right) \\ &\dot{\hat{{{\vartheta}}}} = {{\Gamma}}{{Proj}} \left( \hat{{{\vartheta}}}, {{y}}_{\vartheta}\right) \end{split} $


      where ${{\varGamma}}$ is a symmetric positive defined matrix and the projection functions are defined as:

      $\begin{split} &{{y}}_{\psi} = -\tilde{{{\omega}}} \left \| {{{\omega}}} \right \|\\ &{{y}}_{\phi} = - \tilde{{{\omega}}} \left \| {{F_e}} \right \|\\ &{{y}}_{\vartheta} = - \tilde{{{\omega}}}. \end{split}$

    • To analyze the estimated errors, the following candidate Lyapunov function is considered:

      $ {\tilde{{V}}} = \tilde{{{\omega}}}^{\rm{T}}\tilde{{{\omega}}}+\tilde{{{\psi}}}^{\rm{T}}{{\varGamma}}^{-1}\tilde{{{\psi}}}+\tilde{{{\phi}}}^{\rm{T}}{{\varGamma}}^{-1}\tilde{{{\phi}}}+\tilde{{{\vartheta}}}^{\rm{T}}{{\varGamma}}^{-1}\tilde{{{\vartheta}}}. $


      With a procedure similar to [9], we get:

      $ \dot{{\tilde{{V}}}}+ {{\lambda}} \tilde{V} \leq {{\lambda}} {{ \delta}}_{{\tilde{{V}}}} $


      where $ {\bf{\lambda}} $ depends on eigenvalues of ${{F}}+{{K}}$ and

      $ \begin{split} {{\delta}}_{\tilde{V}} =& 2\left \| {{\varGamma}} \right \|^{-1}\left( \tilde{{{\psi}}}^{\rm{T}}\tilde{{{\psi}}} + \tilde{{{\phi}}}^{{\rm{T}}}\tilde{{{\phi}}} + \tilde{{{\vartheta}}}^{{\rm{T}}}\tilde{{{\vartheta}}}\right)+ \\ & 2\left \| {{\varGamma}} \right \|^{-1}\left( \frac{1}{{{\lambda}}}\tilde{{{\psi}}}\dot{{{\psi}}}+ \frac{1}{{{\lambda}}}\tilde{{{\phi}}}\dot{{{\phi}}}+ \frac{1}{{{\lambda}}}\tilde{{{\vartheta}}}\dot{{{\vartheta}}}\right) \end{split} $


      where $ \tilde{{{\psi}}},\;\tilde{{{\phi}}} $ and $ \tilde{{{\vartheta}}} $ are bounded (see [30] for a detailed proof of these properties). Thus, if $ {{\tilde{{V}}}} \geq {{\delta}}_{{\tilde{{V}}}} $, then $ {{\tilde{{V}}}} \leq 0 $. In addition, by having the adaptation gain $ \varGamma $ sufficiently large, we can lower errors (33) to a bounded ball given by:

      $ \begin{split} &\left \| \tilde{{{\omega}}} \right \| \leq \sqrt{{{\delta}}_{{\tilde{{V}}}}} \\ &\left \| \tilde{{{\psi}}} \right \| \leq \sqrt{\left \| {{\varGamma}} \right \| {{\delta}}_{{\tilde{{V}}}} } \\ &\left \| \tilde{{{\phi}}} \right \| \leq \sqrt{\left \| {{\varGamma}} \right \| \delta_{{\tilde{{V}}}} } \\ &\left \| \tilde{{{\vartheta}}} \right \| \leq \sqrt{\left \| {{\varGamma}} \right \| {{\delta}}_{{\tilde{{V}}}} }. \end{split} $


      Next, to analyze the evolution of ${{\omega}}$, the following candidate Lyapunov function is used:

      $ {{V}} = \frac{1}{2} {{\omega}}^{{\rm{T}}}{{\omega}}. $


      Deriving (35) and then inserting (25), it is obtained:

      $ \dot{{{V}}} = {{\omega}}^{{\rm{T}}}\left( {{F}} {{\omega}} + {{K}} {{\omega}} + {{HF_e}}+ \tilde{{{\psi}}}\left \| {{\omega}} \right \|+ \tilde{{{\phi}}} \left( t\right) \left \| {{F_e}} \right \| + \tilde{{{\vartheta}}} \right) $


      and defining $ {\tilde{{p}}} = \left( \tilde{{{\phi}}} \left( t\right) \left \| {{F_e}} \right \| + \tilde{{{\vartheta}}}\right) $, $ {{F}}+{{K}} = {{K}}_{{1}} $, we get:

      $ \dot{{{V}}} \leq \left({{K_1}} + \tilde{{{\psi}}}\right) {{\omega}}^{{\rm{T}}}{{\omega}} + \left( {{HF_e}} + {\tilde{{p}}}\right) \left| {{\omega}}\right|. $


      If $ {{K_1}} + \tilde{{{\psi}}} <0 $, then $ {{\omega}} $ is bounded. Through this, it can be concluded that the adaptation errors $ \tilde{{{\psi}}},\tilde{{{\phi}}},\tilde{{{\vartheta}}} $ and $ \tilde{{{\omega}}} $ of (34), and the state $ {{\omega}} $ of (37) are bounded and therefore $ \left( {{\upsilon}}_{2}+{{\theta}} \right) = \left(-\hat{{\theta}}+{{\theta}} \right) = \tilde{{\theta}} \in L_{\infty} $ and becomes smaller as the gain ${{\varGamma}}$ is higher. Additionally with the values set in the $ {{K}} $ controller, the convergence rate of the system can be arbitrarily set considering a perfect compensation of inverse dynamics. Instead, when we do not have an ideal estimation, the rate of convergence must be established sufficiently high with respect to the estimator errors to satisfy (37).

    • Because the humanoid robot has a hybrid behavior (continuous and discrete), $ v_{hip} $ changes between impacts, so the continuous virtual slave is used to limit this hybrid behavior. In addition, the error $ {\tilde{{y}}}_{{2}} $ remains invariant during the impact and hence, a PHZD is defined like [25], where the condition $ {\dot{\tilde{{y}}}}_{{2}} = {{0}} $ holds because the foot arrives at the ground with a soft acceleration profile causing a null joint velocity reference at the time that the foot reaches the ground. To teleoperate the forward velocity $ v_{hip} $ and turning angle $ \delta $, a virtual slave of closed-loop continuous dynamics will be defined in (39) that it bounds above the hybrid dynamic response, which stably converges if $\epsilon_{real} = \dfrac{k_{v_{hip}}}{1+\sigma _{real}}$ is sufficiently high (see Fig. 3 of [8]). From Proof Theorem 2 of [8], it is possible to infer the existence of $ \rho > 0 $, with T as a walking cycle time:

      $ \epsilon_{real}T > \rho. $


      Then, the hybrid dynamics tend to a stable limit cycle. It is important to point out that the walking cycle time has to be small enough to allow the robot walks at higher forward velocities, and big enough to ensure that the phases of the walk could be accomplished rapidly to retain stability. We propose a closed-loop virtual system that limits the response of stable hybrid dynamics, as shown in Fig. 4, where it is possible to appreciate qualitatively that the real robot convergence rate $ \epsilon_{real} $ must be sufficiently greater than the convergence rate $ \epsilon_{virtual} $ of the virtual system. The open-loops continuous slave for teleoperation is defined by

      Figure 4.  Evolution of the behavior of hybrid and continuous systems.

      $ {{D}} {\dot{{\eta}}} = {{{f}}_{{{sl}}}} - {{{f}}_{{v}}}+\tilde{{\theta}}_l $


      $ {\ddot{{y}}}_{{sm}} = {{M}}_{{{sm}}}^{ - {{1}}}\left( {{{f}}_{{{sm}}}}+ {{F_{em}}}+ \tilde{{\theta}}_m \right) $


      where (40) is based on (25) and ${{\eta }} = {\left[ {\begin{array}{*{20}{c}} {{v_{virtual}}}&\delta \end{array}} \right]^{\rm{T}}}$ is a vector that links the linear virtual velocity represented with $ {v}_{virtual} $ and the turn angle of the humanoid robot represented with $ \delta $, $ {{D}} $ is the matrix of inertia of the virtual slave, the control action $ {{f_{sl}}} $ involves a force applied to the virtual robot and $ {{f_v}} $ is the external virtual force that holds $ {{f_v}}\in {{\cal{L}}_{\infty}}\cap {{\cal{L}}_{2 }} $ with $ \left\| {{f_v}} \right\|\le {\bar{{f}}}_{{v}} $ (positive constant), ${\widetilde {{\theta }}_l} = {\left[ {\begin{array}{*{20}{c}} {{{\widetilde {{\theta }}}_v}}&{{{\widetilde {{\theta }}}_\delta }} \end{array}} \right]^{\rm{T}}}$ and $ \tilde{{\theta}}_m $ are the non-linear uncertainties that hold $ \tilde{{\theta}}_l, \tilde{{\theta}}_m \in {\cal{L}}_{\infty} $, and the convergence rate of the virtual system is computed by $\epsilon_{virtual} = \dfrac{k_{s}}{D+\sigma _{virtual}}$.

      Finally, ${{y_{sm}}}$ represents real end-effector positions and ${{f_{sm}}}$ is the force applied to the active arm.

    • As described in the literature, the P+d controllers are simple structures that have adequate performance in practice for bilateral teleoperation systems of manipulator robots[23, 31] as well as bilateral teleoperation of mobile robots[32, 33], where a sufficiently high value of damping is injected to assure the stability of the delayed system. The bilateral loop is formed by force feedback applied to the master of locomotion $ {{f_{ml}}} $ and master of manipulation $ {{f_{mm}}} $. The slave uses $f_{sl} $ for locomotion and $ f_{sm}$ for manipulation. Finally, an elastic force $f_v $ (that links the virtual slave and the closed-loop humanoid robot) is applied. The controller equation are:

      $ \begin{split} {{f_{ml}}} =& -{{k_{ml}}}\left ( {{k_{gl}}}{{x_{ml}}}-{{\eta}}\left ( t-h_{2} \right )\right )+\\ &{{g_{ml}}\left ( x_{ml} \right )}-{{\alpha}}_{{ml}}{{\dot{{x}}}_{{{ml}}}} \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \end{split}$


      $ \begin{split} {{f_{mm}}} =& -{{k_{mm}}}\left ( {{k_{gm}}}{{x_{mm}}}-{{{y}_{sm}}}\left ( t-h_{2} \right )\right )+\;\;\;\\ &{{g_{mm}\left ( x_{mm} \right )}}-{{\alpha}}_{{mm}}{\dot{{x}}}_{{mm}} \end{split} $


      $ {{f_{sl}}} = {{k_{sl}}}\left ( {{k_{gl}}}{{x_{ml}}}\left ( t-h_{1} \right )-{{\eta}}\right )-{{{\sigma }}_{{{sl}}}}{{z}}\;\;\;\;\;\;\;\;\;\;\;\; $


      $ {{f_{v}}} = {{\Omega }}\left ( \cdot \right )\bar{{f}}_{v}\tanh \left ( {{\beta }}_{{1}} \left ( \varUpsilon-{{{\eta}}}\right )+ {{\beta }}_{{2}} \left ( \dot{\varUpsilon}-{\dot{{\eta}}} \right )\right ) $


      $ {{f_{sm}}} = {{k_{sm}}}\left( {{k_{gm}}}{{y}}_{{4}} \left ( t-h_{1} \right ) -{{y_{sm}}} \right) -{{{\sigma }}_{{{sm}}}}{{\dot{{y}}}_{{{sm}}}} \;\;\; $


      where $\varUpsilon = \left[ {\begin{array}{*{20}{c}} {{v_{vip}}}&\delta \end{array}} \right]$. The parameters have positive values, where $ {{k_{ml}}} $, $ {{ k_{mm}}} $, $ {{ k_{sm}}} $, and $ {{{k}}_{{{sl}}}} = \left[ {\begin{array}{*{20}{c}} {{k_{vhip}}}&0\\ 0&{{k_\delta }} \end{array}} \right]$, represent positive gain matrixes, $ {{k_{gl}}} $ linearly maps the master position to a speed and turn angle references, $ {{k_{gm}}} $ linearly maps the master position of manipulation to the references of the end effector, and z represents a vector that links the acceleration and the angular velocity $ {\dot{{\eta}}} $ at an infinitesimal time before $ t $, i.e., $ {\dot{{\eta}}} = {{z}}+\gamma{\dot{{z}}} $ with $ \gamma \rightarrow 0^{+} $[32]. $ {{\alpha}}_{{ml}} $ and $ {{\alpha}}_{{mm}} $ are coefficients of damping injected in the master of locomotion and manipulation, $ {{{\sigma }}_{{{sl}}}} = \left[ {\begin{array}{*{20}{c}} {{\sigma _{virtual}}}&0\\ 0&\tau \end{array}} \right] $ is the damping injected in the virtual slave composed by the virtual damping of the forward speed $ \sigma_{virtual} $ and the damping of the turn of the real robot $ \tau $, $ {{{\sigma }}_{{{sm}}}} $ is the damping of the active arm, $ {{\beta }}_{{1}} $ is the elasticity coefficient and $ {{\beta }}_{{2}} $ is the damping coefficient of the coupling impedance. The function $ {{\Omega }}\left ( \cdot \right ) $ has a bounded output that tends to zero for $ t\rightarrow \infty $ in order to assure that $ {{f_{v}}} \in L_{2} $. This function represents a turn-off function after T seconds, i.e., the function is equal to one from 0 to T, and later it decays to zero in a finite duration time interval.

    • A positive definite functional $V\left(t,{{x_{lm}}}\right) = {{V}_{l1}}+ {{V}_{l2}}+ {{V}_{l3}}+{{V}_{l4}}+{{V}_{m1}}+{{V}_{m2}}+{{V}_{m3}}$ with ${{x_{lm}}}= \left [ {{{\dot{{x}}}}_{{ml}}},{{{k}_{gl}}}{{{x}}_{{ml}}}-{{\eta}}, {{z}}, {{{\dot{{x}}}}_{{mm}}},{{{x}}_{{mm}}}-{{{{y}}}_{{sm}}}, {{{\dot{{y}}}}_{{sm}}} \right ]$ is built to evaluate the evolution of the locomotion and manipulation system from a finite initial condition. The functional $ V $ is formed by

      $ {{V}_{l1}} = \frac{1}{2}{{\dot{{x}}}_{{ml}}}^{{{\rm{T}}}}{{{M}}_{{ml}}}{{\dot{{x}}}_{{ml}}}+E_{h} $


      $ {{V}_{l2}} = \frac{1}{2}\frac{{{k}_{ml}}}{{{{k}_{gl}}}}{{\left( {{{k}_{gl}}}{{{x}}_{{ml}}}-{{\eta}} \right)}^{{{\rm{T}}}}}\left( {{{k}_{gl}}}{{{x}}_{{ml}}}-{{\eta}} \right) $


      $ {{V}_{l3}} = \frac{1}{2} \gamma \frac{{{k}_{ml}}}{{{{k}_{sl}}}{{{k}_{gl}}}} {{{z}}}^{{{\rm{T}}}}{{{D}}}{{{z}}} $


      $ \begin{split} \quad\quad\;{{V}_{l4}} =& {{\int\limits_{-{\bar{h}_{2}}}^{0}{{}}}_{{}}}\int\limits_{t+\theta }^{t}{{{z}}{{\left( \xi \right)}^{{{\rm{T}}}}}{{z}}\left( \xi \right)}{\rm{d}}\xi {\rm{d}}\theta+ \\ & {{{k}_{gl}}}^{2} {{\int\limits_{-{\bar{h}_{1}}}^{0}{{}}}_{{}}}\int\limits_{t+\theta }^{t}{{{{\dot{{x}}}}_{{ml}}}{{\left( \xi \right)}^{{{\rm{T}}}}}{{{\dot{{x}}}}_{{ml}}}\left( \xi \right)}{\rm{d}}\xi {\rm{d}}\theta \end{split} $


      $ \begin{split} \quad\quad\;{{V}_{m1}} =\;& {{\dot{{x}}}_{{mm}}}^{{{\rm{T}}}}{{{M}}_{{mm}}}{{\dot{{x}}}_{{mm}}}+\frac{{{k_{mm}}}}{{{k_{sm}}}}{{\dot{{y}}}_{{sm}}}^{{{\rm{T}}}}{{{M}}_{{sm}}}{{\dot{{y}}}_{{sm}}}+\\ \;\; &E_{h}+\frac{{{k_{mm}}}}{{{k_{sm}}}} E_{e} \end{split} $


      ${{V}_{m2}} = {{{k}_{mm}}}{{\left( {{{x}}_{{mm}}}-{{{{y}}}_{{sm}}} \right)}^{{{\rm{T}}}}}\left( {{{x}}_{{mm}}}-{{{{y}}}_{{sm}}} \right) $


      $ \begin{split} \quad\quad\; {{V}_{m3}} =& {{\int\limits_{-{\bar{h}_{1}}}^{0}{{}}}_{{}}}\int\limits_{t+\theta }^{t}{{{{\dot{{x}}}}_{{mm}}}{{\left( \xi \right)}^{{{\rm{T}}}}}{{{\dot{{x}}}}_{{mm}}}\left( \xi \right)}{\rm{d}}\xi {\rm{d}}\theta+\\ &{{\int\limits_{-{\bar{h}_{2}}}^{0}{{}}}_{{}}}\int\limits_{t+\theta }^{t}{{{{\dot{{y}}}}_{{sm}}}{{\left( \xi \right)}^{{{\rm{T}}}}}{{{\dot{{y}}}}_{{sm}}}\left( \xi \right)}{\rm{d}}\xi {\rm{d}}\theta. \end{split} $


      Following the procedure similar to the test described in [32] and [22], $ \dot{V} $ along the trajectories of the closed loop system, considering the virtual slave (39), locomotion master dynamics (1), time delay, the human operator (6), the slave (active arm) (40), manipulation master dynamics (1), and environment forces (7), is bounded by:

      $ \begin{split} \dot{V}\leq &-\lambda _{ml}{\dot{{x}}}_{{{ml}}}^{{{{\rm{T}}}}}{\dot{{x}}}_{{{ml}}}-\lambda_{sl} {{z}}^{{{{\rm{T}}}}} {{z}}+\rho_{sl}\left | {{z}} \right | -\\& \lambda _{mm}{\dot{{x}}}_{{{mm}}}^{{{{\rm{T}}}}}{\dot{{x}}}_{{{mm}}}-\lambda_{sm}{{\dot{{y}}}_{{{sm}}}}^{{{{\rm{T}}}}}{{\dot{{y}}}_{{{sm}}}}+\rho_{sm}\left | {{\dot{{y}}}_{{{sm}}}} \right | \end{split} $



      $ \begin{split} &\lambda _{ml} = {{\alpha}} _{{ml}}{{I}}-{{k_{gl}}}^{2}\bar{h}_{1}{{I}}-\frac{{{k_{ml}}}^{2}}{4}{{h}}_{2}{{I}}\\ &\lambda _{sl} = \frac{{{k_{ml}}}}{{{k_{sl}}}{{k_{gl}}}}\left( {{\sigma _{sl}}}{{I}}+{{D}}\right)- \frac{1}{4}\bar{h}_{1}\frac{{{k_{ml}}}^{2}}{{{k_{gl}}}^{2}}{{I}}-\bar{h}_{2}{{I}} \end{split} $

      $ \begin{split} &\lambda _{mm} = 2{{\alpha}}_{{mm}} {{I}}- \bar{h}_{1}{{I}}-\bar{h}_{2}{{k_{mm}^2}}\\ &\lambda _{sm} = \frac{2{{k_{mm}}}{{\sigma _{sm}}}}{{{k_{sm}}}} {{I}}- \bar{h}_{2}{{I}}-{\bar{h}_{1}}{{k_{mm}^2}}\\ &\rho_{sl} = \frac{{{k_{ml}}}}{{{k_{sl}}}{{k_{gl}}}} \left( {{f_{v}}}+\tilde{{\theta}}_l\right)\\ &\rho_{sm} = \frac{2{{k_{mm}}}}{{{k_{sm}}}}\tilde{{\theta}}_m . \end{split} $


      Remark 1. If $ {{\alpha}}_{{ml}} $, $ {{{\sigma }}_{{{sl}}}} $, $ {{\alpha}}_{{mm}} $ and $ {{{\sigma }}_{{{sm}}}} $ are sufficiently high damping to fulfill $ \lambda_{ml},\lambda_{sl},\lambda_{mm}, \lambda_{sm}>0 $ of (54), then we can point out from (53) that the variables $ {\dot{{x}}}_{{{ml}}},{{z}},{\dot{{x}}}_{{{mm}}},{\dot{{y}}}_{{{sm}}} \in L_{\infty} $. As $ \lambda_{ml},\lambda_{sl},\lambda_{mm}, \lambda_{sm} $ are higher, increasing the damping injected, then $ {\dot{{x}}}_{{{ml}}},{{z}},{\dot{{x}}}_{{{mm}}},{\dot{{y}}}_{{{sm}}} $ will remain in a smaller origin-centered ball. Next, if (53) is integrated with respect to time, we get:

      The term $\int_{0}^{{\rm{T}}}{{z}}\left(\epsilon\right)^{\rm{T}} \rho_{sl} \left(\epsilon\right){\rm{d}}\epsilon$ and $\int_{0}^{{\rm{T}}}{{\dot{{y}}}_{{{sm}}}}\left(\epsilon\right)^{\rm{T}} \rho_{sm} \left(\epsilon\right){\rm{d}}\epsilon$ are bounded since $ {\dot{{x}}}_{{{ml}}},{{z}},{\dot{{x}}}_{{{mm}}},{\dot{{y}}}_{{{sm}}} \in L_{\infty} $, $ {{f_v}} \in L_{2} $ (finite energy) and using the adaptive control stability demonstration of Section 4.3.1, it is concluded that the estimation errors are bounded $ \left( \tilde{{\theta}} \in L_{\infty}\right) $. So, from (55), we infer that $ V\left( t\right) $ will be bounded for all t and therefore $ {\dot{{x}}}_{{{ml}}},{{z}},{\dot{{x}}}_{{{mm}}},{\dot{{y}}}_{{{sm}}} \in L_{2} $ and ${{{\dot{{x}}}}_{{ml}}},{{{k}_{gl}}}{{{x}}_{{ml}}}-{{\eta}},{{\eta}},{{z}},{{{\dot{{x}}}}_{{mm}}}, {{{k}_{gm}}}{{{x}}_{{mm}}}-{{{{y}}}_{{sm}}}, {{{\dot{{y}}}}_{{sm}}} \in L_{\infty}$.

      $ \begin{split} &V\left ( t \right )-V\left ( 0 \right )\leq -\lambda _{ml}\left \| {{\dot{{x}}}_{{{ml}}}} \right \|_{2}^{2}-\lambda _{sl}\left \| {{z}} \right\|_{2}^{2} -\lambda _{mm}\left \| {\dot{{x}}}_{{mm}} \right \|_{2}^{2} -\\ &\;\;\lambda _{sm}\left \| {{\dot{{y}}}_{{{sm}}}} \right \|_{2}^{2}+\int_{0}^{t}{{z}}\left(\epsilon\right)^{\rm{T}} \rho_{sl} \left(\epsilon\right){\rm{d}}\epsilon+\int_{0}^{t}{{\dot{{y}}}_{{{sm}}}}\left(\epsilon\right)^{\rm{T}} \rho_{sm} \left(\epsilon\right){\rm{d}}\epsilon . \end{split} $

    • The control parameters must be calibrated correctly to comply with stable behavior. The following guidelines to qualitatively calibrate the controller are established:

      1) The parameters $ {{k_{ml}}}, {{k_{gl}}},{{k_{mm}}}, {{k_{gm}}}, {{k_{sl}}} $ and $ {{k_{sm}}} $ are positive matrices calibrated with null delay. On the other hand, parameters ${{k_{ml}}}\;{\rm{and}}\;{{k_{mm}}}$ set the desired level of force feedback while $ {{k_{gl}}} $ and $ {{k_{gm}}} $ establish the maximum speed or position, $ \beta_1 $ and $ \beta_2 $ are positive constants to set the coupling between the virtual robot and the real one.

      2) The damping injected $ \sigma_{real} $ is set as a function of $ {{{\sigma }}_{{{sl}}}} $ which in turn depends on the time delay (54) to hold a stable delayed bilateral teleoperation. In addition, the value of $ \sigma_{real} $ also depends on the errors of the adaptive control law. If adaptation errors increase, then $ \sigma_{real} $ must be higher to hold (37). Thus, if the time delay increases $ \left( h_1, h_2 \right) $, then the master damping $ \left( {{\alpha}}_{{ml}}\right) $ as well as the virtual damping $ \left({{{\sigma }}_{{{sl}}}}\right) $ must be raised too and therefore an increment of the real damping $ \left( \sigma_{real}\right) $ must be performed based on Section 4.4. This slows the convergence rate $ \epsilon_{real} $ of the continuous part and therefore this procedure, retaining a constant cycle period, can be applied while $ \epsilon_{real} $ is greater than a minimum bound $ {\epsilon} $[25]. If necessary, the designer could change the walking cycle time depending on the real slave damping in order to assure a stable hybrid dynamic.

    • In this section, different tests are presented to verify the theoretical results achieved. These tests were performed in a structural environment where a human operator drives a simulated humanoid robot in VREP. The task scenario as shown in Fig. 5 consists of the following phases: First, the forward speed and angle of the robot are controlled to achieve locomotion to the first black mark on the ground, which signals the place where the humanoid should be positioned to grab the red object of 50 grams of mass. Once in position, it takes the red object and transports the red object to the next black mark, where a manipulation task is performed to put the red object on the new location. Next, the NAO walks a straight line to the third black mark, and then it starts to turn to reach the final goal. During the whole task, the robot must follow the black path.

      Figure 5.  Workplace (each square has a 50 cm side), where the path that the robot must follow is shown.

      Fig. 6 shows a sequence of images of the phases of the test. On the local site (left side of Fig. 6), two Novint Falcon robots are used as master devices, one for locomotion with 2 DOF and the other for manipulation with 3 DOF. The controllers for these devices were implemented on Matlab simulink. The right side of Fig. 6 shows the behavior of the simulated NAO during the different stages. A video of a test non-delay without force feedback and delayed with force feedback is presented in Three variants of the experiment were performed: 1) non-delay, 2) delayed without force feedback, and 3) delayed with force feedback. The time delay is generated as a time variable signal plus a filtered Gaussian noise $ \left( M_{1}\left( t\right),M_{2}\left( t\right)\right) $. The delays generated are $h_1 = \left( 1+0.5 {\rm{sin}}\left( 2\pi 0.25t\right) + M_{1}\left( t\right)\right)$ and $h_2 = \left( 1+ M_{2}\left( t\right)\right)$. These delays are incorporated into the communication channel. Each experiment is repeated five times with one human operator and the average value of the synchronization error for locomotion, path error, time to complete the task, and the number of successful tests are calculated using the following expressions:

      Figure 6.  Phases of the experiments

      1) Synchronization error: error between the position or reference speed (measured at the local site) and real signals of the slave robot (measured at the remote). ${\varepsilon _s} = {\left[ {\begin{array}{*{20}{c}} {{\varepsilon _v}}&{\;{\varepsilon _\delta }} \end{array}} \right]^{\rm{T}}}$, which measures both velocity and turning errors, is computed as follows:

      $ {{\varepsilon }_{s}} = \frac{1}{n}\sum\limits_{i = 1}^{n}{\frac{1}{{{t}_{{{f}_{i}}}}}}\int\nolimits_{0}^{{{t}_{{{f}_{i}}}}}{\left| {{k}_{gl}}{{{x}}_{{{{m_l}}_{i}}}}\left( t \right)-{{{{\eta}}}_{i}}\left( t \right) \right|}{\rm{d}}t. $

      2) Path error $ {{\varepsilon }_{path}} $ is the average error between the reference path and the path followed by the robot.

      3) Average time to complete the task $ T_{task} $ is defined by

      $ {{T}_{task}} = \frac{1}{n}\sum\limits_{i = 1}^{n}{{{t}_{{{f}_{i}}}}} $

      where $ n $ is the number of tests performed for each case.

      4) The amount of successful tests is an indirect metric. It indicates that of the five tests that were carried out in each experiment, all of them were finished (all 5 phases).

      For cases A and B, the proposed controller in (41)−(43) and (45) has $ {{\alpha}}_{{ml}} = {{0}},\;{{{\sigma }}_{{{mm}}}} = {{0}}, \;{{k_{ml}}} = {{0}} $ and $ {{k_{mm}} = {{0}}} $, and the slave damping ${{{\sigma }}_{{{sl}}}} = {{0}}.{{01}}\;{\rm{and}}\; {{{\sigma }}_{{{sm}}}} = {{0.02}}$ are the same in all cases. For case C in which force feedback is used, the controller parameters are $ {{\alpha}}_{{ml}} = {{0.2}} $ and $ {{{k}}_{{{ml}}}} = \left[ {\begin{array}{*{20}{c}} {0.3}&0\\ 0&{0.2} \end{array}} \right]$ for locomotion and $ {{\alpha}}_{{mm}} = {{0.2}} $ and $ {{{k}}_{{{mm}}}} = \left[ {\begin{array}{*{20}{c}} {0.1}&0&0\\ 0&{0.2}&0\\ 0&0&{0.1} \end{array}} \right]$ for the active arm. The simulation results for cases A, B and C are shown in Fig. 7, where the average values of the performance indexes are shown. Here it can be observed that the test for the time-delayed without force feedback has the worst performance since the movements that are generated by the human operator are fast and more energy is injected into the system, then the robot moves without control. Then, when the force feedback is used in the delayed tests, the indexes improve and approach the values of the test without delay. This improvement is produced by a sufficiently high value of damping injected to assure stability and with force-feedback applied to the human operator. This helps to prevent sudden moves of the master and push it to a location with less synchronization error. Also, Fig. 8 shows the path error, in case A, the operator can easily make the robot follow the black path to perform the test. But in case B, where a time delay is added to the communication channel without force feedback, it is harder for the operator to make the robot follow the reference path. But in case C, thanks to the force feedback used for the teleoperation, the test can be successfully completed through all its phases.

      Figure 7.  Performance indexes of the tests

      Figure 8.  Path error in the three schemes

      Using the proposed scheme in case C, Figs. 9(a) and 9(b) show the linear velocities and turning angles of the robot, where it can be observed that synchronization error is bounded even though the time delay is variable. Also, Fig. 10 shows the different stages of the test, where the first locomotion phase goes from 0 to 50 s, the locomotion plus manipulation phases go from 50 s to 180 s, and the last locomotion phase go from 180 s to 360 s. In the locomotion phase, the active arm is teleoperated through the second joystick (Fig.6) to take the object and transport it to the next box. Throughout the experiment, the angle of inclination of the torso plays a very important role, as shown in Fig. 11, the torso adapts according to the different forward speeds and the torque produced by the arm when it is raised or/and takes the object (see Fig. 3).

      Figure 9.  Synchronization error for locomotion: a) Robot linear velocity and reference from human operator; b) Robot turn angle and command.

      Figure 10.  Synchronization error for manipulation.

      Figure 11.  Balance control for locomotion and manipulation tasks

      We remark that the use of force feedback helps to keep both manipulation and walking stable, since the human operator can feel the synchronism error as well as avoid rapid motions of a dual master when the time delay increases, via a controlled-damping injection.

    • In this paper, a control scheme for delayed bilateral teleoperation of a humanoid robot was performed to analyze the benefits of the use of force feedback. The proposed scheme links adaptative inverse dynamics compensation, P+d, PD, and P+d+Fe controllers, balance control, and coupling between a humanoid robot and a virtual slave. Using the parameter setting procedure proposed for the controller, which is based on delay-dependent injected damping, we obtained a stable behavior of the teleoperated system in both manipulation and walking despite the communication delays between the master robot and the slave robot. Section 6 reported that the use of force feedback provides a better response from the system as opposed to not using it.

      It is difficult to generalize the results obtained for any human operator (trained, expert, novice), therefore future researchers should be focused on a multiple operator analysis, experiments with real teleoperated humanoid robots in front of several tasks, and different time delays to get more statistical information about the impact of the force feedback in multiple scenarios. However, we point out that the reason for performing human-in-the-loop simulations on simple tests was to verify the advantages of using force feedback in delayed teleoperation of humanoid robots, and with the indexes such as elapsed time to complete the task, synchronization error, and successful tests rate, this was corroborated in a quantitative manner.

    • This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

      The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

      To view a copy of this licence, visit

Reference (33)



    DownLoad:  Full-Size Img  PowerPoint