Personalized sensor- and robot-supported training for upper limbs
Monitoring of upper limbs motor skills is important throughout the rehabilitation process. The doctoral thesis, consisting of three studies, deals with the analysis of upper limb activity in the clinical setting, during activities of daily living, and the evaluation of upper limb interaction during robot training. Upper limb movement was measured with a wearable sensory system consisting of wireless inertial-magneto measurement units (IMU) and electromyography sensors. Sensors are small and do not interfere with upper limb movement. The methodology for computation of upper limb kinematics based on IMU data is presented. First study focuses on measuring and quantifying upper limb and trunk movement while executing ARAT and WMFT motor tasks. In the second study of the doctoral thesis we used wearable sensory system for monitoring upper limbs movement while performing activities of daily living. In the first step, time quantization of movement is used for computation of activity counts, counts of muscle activity and power counts for each upper limb. In the second step, upper limb motion was segmented into individual movements based on changes in velocity and direction of movement of the upper limbs. In the third study, the analysis of the interaction of the upper limbs during a bima- nual tasks with a robot was performed. The system was used with a group of healthy volunteers and a group of patients after stroke. The analysis was based on measurements of the interaction forces between the upper limbs and the robot. We performed a comparison between groups of subjects at a given ro- bot resistance and a comparison of the impact of robot resistance within each group of subjects. The proposed method enables quantification of activities of each upper limb and differentiates between the groups with different degrees of impairment. Analysis of robot resistance shows to a large extent statistical significant differences for most of the computed parameters within the group of healthy subjects and to lesser extent within the groups of patients after stroke.
Analysis and synthesis of methods for safe, sensory supported unimanual robot based training
This doctoral thesis is dealing with robotic exercise for support in motor rehabilitation of patients with movement disorders of the upper extremities. The first part of the thesis is focused on an unobtrusive measurement of the persons’ physiological response. First, different physiological processes, that manifest themselves as bioelectric and other types of signals, are described. State of the art technologies for measuring these signals are presented, based on the literature review a new measurement system is proposed. Unobtrusive measurement methods are described for different physiological signals, followed by proposed signal analysis and processing. The proposed system was experimentally validated. Validation was performed by using a reference measurement system, considered as the gold standard for physiological measurement. Raw signal correlation was performed for all physiological signals, as well as parameter comparison. A thorough analysis of motion artifacts and signal quality during tasks was conducted. An adaptive task controller for a physical robot interaction task si validated in the last part of the chapter. Controller was based on a tree structure, using physiological and biomechanical signals from the proposed system as inputs. Second part of the thesis is concerned with analysis of a variable stiffness actuator and methodology for control. Variable stiffness actuators enable a safe human-robot interaction. The chapter starts with a review of different variable stiffness configurations. It is followed by a detailed description of a antagonistic variable stiffness configuration, utilizing non-linear springs for stiffness adjustment. Detailed description of LinWWCVSA actuator is presented, along with the stiffness and force models. These can be implemented in actuator control. Further, a new version of a haptic interface is presented, using the LinWWC-VSA configuration. Hardware and software used by the interface are described in detail, followed by kinematic and dynamic modeling of the interface. Different performance measures for haptic interfaces are described, followed by an experimental review of the haptic interface. Position and force control methods are proposed and analyzed. Force calibration and measurement is presented together with control. Finally, active compliance control is presented and analyzed. Impedance and admittance control strategies are evaluated.
Modeling and evaluation of ski jump from data acquired with wearable measurement system
Progress and minimization of sensory technologies used in wearable systems has enabled a rich development development of new applications. Information acquired with wearable sensors can be used in different areas such as medicine, biomechanics, sport, sociology, psychology, and engineering. A combination of gyroscope, accelerometer, and magnetometer as an inertial measurement unit is considered to be the most useful for the assessment of orientation of the body segments and study of the biomechanics of an athlete’s movement. The raw signals measured by each sensor are processed using sensory fusion to assess the orientation of the sensor, or simply improve the basic measurement of the sensor’s physical quantity.
In addition to a small size, which makes them wearable, sensors are also energy efficient and therefore suitable for monitoring athletes over longer periods of time.
Sensory fusion of pressure mattress data and wireless inertial and magnetic measurement units (IMUs)
Assessment of infant motor patterns during early infancy is important and needs to be objective, accurate and sufficiently reliable. To accomplish this, typically applied clinical methods are nowadays often used in combination with advanced measurement systems. The PhD thesis first presents a dedicated sensor-supported measurement system, which was developed by the FP7 EU project CareToy consortium and includes pressure mattresses, IMUs, sensorized toys, and other modules. The thesis further focuses on presentation of numerous novel data processing and sensor fusion algorithms for adequate transformation of raw unprocessed data to final numerical motor pattern parameter results, such as trunk, head, and arm posture analysis, grasping evaluation and postural stability assessment. Proposed methods were validated in three consequent studies, comprising validation with a referential optoelectronic measurement system, evaluation of algorithms on a pilot study, and final assessment of suitability with an extensive clinical trial study. Study results verify that the proposed measurement system in combination with the developed algorithms is appropriate for stimulation and simultaneous assessment of infant activity. Effects of sensor related drawbacks are successfully removed. High correlation of proposed numerical parameters and the obtained clinical scores implies adequacy for complete, accurate and reliable motor pattern evaluation of infants.
Wearable sensory system for measurement and evaluation of sit-to-stand movements
Inertial and magnetic measurement units are typical representatives of wearable sensors. Such sensors are used for orientation assessment by means of different methods of sensory fusion (e.g. Kalman filter). We presented an extended Kalman filter based method which included the kinematic model. The method was updated to include with an adaptive magnetic compensation algorithm which prevented the heading drift of the assessed orientation when used in a disturbed magnetic field. The performance and long-term stability of the proposed method was experimentally evaluated. Result showed that the wearable system is capable of assessing the orientation with an absolute median error below 5°, no expressed drift over time, and with enhanced performance while operating in a disturbed magnetic field. The sit-to-stand manoeuvre of people following transtibial amputation was studied with regard to asymmetry of joint angles and joint torques. The wearable measurement system was used to assess the kinematic and dynamic parameters of sit-to-stand movement before and after hip replacement. Results showed that after the medical procedure the subject was loading his lower extremities more symmetrically.
Ergometer rowing exercise with real-time feedback information
PhD thesis proposes and evaluates an alternate method for supervision and learning of proper ergometer rowing technique that could complement the traditional method of learning with a trainer. Kinetic and kinematic data are acquired during rowing, processed, and compared with reference models based on skilled rowers. Feedback information provides knowledge of performance using concurrent feedback, video feedback, video modelling, and error correction strategies. Based on the real-time feedback, the rower modifies movement towards a proper technique. 36 participants in three groups took part in an evaluation study. One group trained without supervision, one with a trainer, and one with the real-time biomechanical feedback. The results show that participants were able to utilize the provided feedback. The results of training with the biomechanical feedback were much better than training without supervision and comparable to training with a trainer.
Calibration of an accelerometer and magnetometer using an adaptive method
The inertial and magnetic measurement units are commonly used for the determination of orientation. Due to a presence of nonidealities, calibration of the sensors is thus needed to determine parameters such as bias, misalignment and gain. The PhD thesis describes an automatic online calibration method for a three-axial accelerometer and magnetometer. The accelerometer is placed in a number of different orientations using a robotic arm. The orientations are calculated online from the parameter covariance matrices that represent estimated optimal sensor orientations for parameter estimation. A similar approach is used to estimate the magnetometer parameters. Instead of rotating the sensor, the magnetometer is exposed to different directions of the magnetic field created by a 3D Helmholtz coil. The described method provides better accuracy of parameter estimation using a low number of iterations without the need to manually predefine the orientations of the sensor. The last part of the thesis describes a combination of both calibration methods with a use of a custom developed nonmagnetic two-axis manipulator which is placed on a level surface inside of a 3D Helmholtz coil enabling even higher accuracy of the sensor parameter estimation.
Robot and sensor enhanced robot training
A motor impaired limb severely limits voluntary motor control. The thesis focuses on combining robot rehabilitation and bimanual training into adaptive interactive bimanual rehabilitation systems capable of evaluating the movements and giving feedback to the patient and therapist. The main subject of the thesis is the concept of adaptive bimanual training. The concept is based on the ability of patients to assist movements of the affected limb using the unaffected limb where the level of assistance is adjusted. The concept has been implemented and analyzed on three different systems. The first, based on the standard haptic robot, was used in a study on bimanual movements with four hemiparetic stroke subjects. All subjects improved performance on simple bimanual movements. The second system focuses on the processes behind bimanual training – motor learning. The analysis of bimanual motor learning in healthy subjects showed positive motor learning. The third system is a system for independent sensor-enhanced bimanual training that is capable of measuring interaction between the hands and providing feedback information to the user. The system has proven to be suitable for training of patients of different pathologies and levels of motor disorders.
Haptic primitives in rehabilitation robotics
Robotic systems are becoming increasingly common in upper extremity stroke rehabilitation. Recent studies have already shown that the use of rehabilitation robots can improve recovery. This thesis evaluates the effect of three different modes of robot-assistances in a complex virtual environment on the subjects’ ability to complete the task as well as on various haptic parameters arising from the human-robot interaction. In the second part of the thesis, two haptic interfaces were used, which have already been applied in rehabilitation training, to model and render the surface properties. Parameters of surface friction, stiction and texture models were estimated for the contact of a steel end-effector mounted on the robot tip and ten interacting materials. A surface discrimination virtual task using estimated parameters in the models of different materials has been developed. Healthy subjects performed the task of surface discrimination in virtual environment as well as using real materials. The study showed similar efficiency discrimination abilities among various materials in the virtual and real environment.
Impacts of a small industrial robot with a human lower arm
Robots operating in industrial environment are known as dangerous mechanisms that need to be separated from human workspace with fences or safety control systems. However, lately there has been increased interest in “safe” robotic mechanisms, which would enable safe coexistence and cooperation with humans in industry and domestic environments. Safety is a primary concern in human-robot cooperation. One of the key points is the knowledge regarding the consequences of unintended physical human-robot interaction as a result of a human error or failure of a robotic system. The doctoral thesis presents a research focused on impacts of a small industrial robot with a human lower arm and development of a mechanical model of human lower arm.
Adaptive integration of psychophysiological variables for robotic training
The dissertation titled “Adaptive integration of psychophysiological variables in robotic training” covers the use of four psychophysiological measurements in upper extremity rehabilitation: heart rate, skin conductance, respiration and skin temperature. The ultimate goal is to control the difficulty of a rehabilitation task using data fusion of various features extracted from these four measurements. The research is divided into two parts. The first covers the analysis of rehabilitation-specific factors which could adversely affect psychophysiological responses: the presence of physical activity and pathological conditions. The second part of the dissertation deals with data fusion and biocooperative control. A review of the existing psychophysiological literature was performed, and a number of different dimension reduction and classification methods were selected for implementation. These were first tested in a physically undemanding cognitive task and then transferred to an upper extremity rehabilitation task performed with the HapticMaster robot. Results suggest that psychophysiological measurements are not reliable as a primary data source in motor rehabilitation, but can provide supplementary information that complements task performance and biomechanical measurements.
fMRI compatible haptic robot
Research on the human motor control and haptic perception have seen a strong upswing due to the wide access of methods that can provide noninvasive insight into human brain. One of the most important methods in this area of research is a functional magnetic resonance imaging (fMRI). In addition to the noninvasive method that capture events in the brain, a tool capable of measuring trajectories, velocities and exerting forces on a subject’s upper limb during brain imaging is needed. Preferred tool capable of preforming these controlled hand movements is haptic robot. However, most of the haptic robots available on the market are not compatible with the environment in which examinations with magnetic resonance technique are carried out. The basic condition that enables the observation of magnetic resonance phenomenon at the macroscopic level is a strong magnetic field, which exceeds 1 T in the isocenter of today’s fMRI scanners. There is also a strong radio frequency electromagnetic radiation during fMRI scan. In addition, the space inside a bore of a fMRI scanner is limited. The doctoral thesis, which is divided into five sections, presents two haptic robot designed for use inside fMRI environment.
dr. Justin Činkelj
Robotic control system for hydraulic telescopic handler
The development of a robotized handler (crane) for use in the construction industry is presented in the thesis. The handler is used as a macromanipulator in a system for semi-automated montage of facade panels. The handler is required to manipulate a payload with a mass of 2000 kg with accuracy of a few cm to prevent panel damage, like scratches of the surface. Development was based on a ommercially available handler, with load-sensing hydraulic pump and proportional valves with overlap. The mechanical structure of the handler also drawbacks, in particular backlachs and flexibility of the telescopic boom. Developed software accounted for problems introduced by hydraulic system and compensated accordingly. Achived performance during straight line motion was verified accoring ISO 9283 standard, using an independent measurement system.
Lower-extremities training in multimodal virtual environment
We developed a kinematic model of a human body for real-time visualization in graphical virtual environments. The model was used to create a virtual mirror – a VR application for real-time visualization of body movements by displaying virtual figures enabling visual feedback. An investigation was conducted with a group of healthy subjects, who performed the stepping-in-place tracking test. The results obtained included basic spatial and temporal parameters, which provided quantitative measures of a subject’s adaptation to the environment. Next, we compared the effects of visual and haptic modalities on the subjects’ adaptation capabilities. Haptic feedback was provided by the actuated orthosis Lokomat. Different modalities were engaged separately as well as combined We observed a strong bias towards haptic modality compared to visual modality. Finally, we realised the virtual mirror using stereo camera system and three-dimensional video visualization combined with virtual target tracking. Enhanced visual feedback improved the tracking results and showed that such setup has a potential in emerging telerehabilitation environments.
Training and assessment of the sensorimotor capability of the hand
The work presented in doctoral thesis is focused on development of a system for training and assessment of the sensorimotor capability of the hand. The system is based on measurement of isometric hand forces and force tracking tasks. The training with the system is combined with functional electrical stimulation, which augments tracking capability of the patient. The system was first evaluated as training tool. Two incomplete tetraplegic patients participated in the study and the results show that improvement of muscle control is possible with presented system. The system was also tested out as an assessment tool in clinical study in which two incomplete tetraplegic patients participated. Their progress was assessed with the system and two clinically established tests. The comparison of results shows, that the used methods are complementary to each other. A new prototype of the force measuring device was built in the last part of the doctoral work. The prototype is designed upon magnetorheological device, which has one degree of freedom and can generate variable resistance or can be locked to provide isometric conditions.
Grasping and reaching in haptic virtual environment
The core of this thesis is HEnRiE device (Haptic Environment for Reaching and Grasping Exercise), which is intended for use in robot-aided neurorehabilitation and for training of reaching, grasping, and transporting virtual objects in haptic environments. System combines haptic interface and module for grasping, which is mounted at the end-point of the haptic interface. This allows combined training of the upper extremity movements and grasping. High level of reality is achieved with use of the graphic and haptic visual environments. HEnRiE system was evaluated in a group of healthy persons and two post-stroke subjects during a one-month period of training. Positive outcome of training is reported for the strength of closing and opening of the hand and also for arm movements. Experiments have confirmed that HEnRiE allows training of reaching and grasping movements, such that beside arm movement treatment, the therapy can be expanded to grasp training and therapies can be carried out jointly at the same time.
A laser system for lateral deflection measurement
The thesis is focused on optical measuring of lateral displacements in precision mechanical systems. First, a theoretical analysis of coordinate measuring arm (CMA) error sources is performed and elastic deformations are identified as one of main contributions. A novel system for direct measurement of mechanical link deformations is then presented, capable of high resolution and high frequency real-time output of 2 DOF deformation data. The system was calibrated to achieve 1 um accuracy and low temporal drift. It was integrated in a CMA in order to measure deformations caused by different external and internal sources. Specific effects were quantified for the first time. In the last part, a straightness measurement system is presented, based on the newly developed components. Various geometric and dynamic measurements of machines are possible. It features excellent stability in non-controlled (industrial) environment and accuracy performance similar to complex and expensive systems, such as a laser interferometer.
Fast and accurate contactless distance measurements in industrial environment
The goal of the thesis is development of fast and very accurate distance measurements in demanding industrial environment. Analysis, testing, and results are shown on two completely different industrial applications. The first is a development of a quick non-contact robotic measurement system with use of a laser distance sensor for gray-iron grates dimensional measurements before their deburring with another industrial robot, located in a foundry. The second example shows a development of a very accurate noncontact distance measurement system for diastat membrane expansion measurement in a dirty industrial environment with plenty of vibrations, dirt and temperature fluctuations. The measurement results confirmed that the fast robot measurement cell and the very accurate measurement system for membrane extension monitoring correspond to the predefined goals. Unfortunately, the first system was never introduced into production, but the second one brought significant production economic savings.
Assessment, modelling and evaluation of kinematic properties of multifingered grasping
Four tasks were selected to study the dexterous manipulation. The visual information of the subject was augmented by displaying the posture of the real object on the computer screen. We developed a kinematic model of a finger and thumb, and proposed a model based method for assessing angles in the finger and thumb joints. In the method one marker for each finger, three markers on the thumb and three on the dorsal aspect of hand are needed to acquire the angles through inverse kinematics. The accuracy of the inverse kinematics method was evaluated by using the reference method, where large number of markers was used to estimate the angles from the centers of rotation of joints. An instrumented glove was employed as a complementary system to overcome the problem of occlusion of markers and methods for its calibration proposed.
Haptice interface for finger exercise and virtual rehabilitation
A virtual rehabilitation system, including a haptic device and virtual environments, was developed. The system was designed for the hand finger rehabilitation and finger functionality assessment. Along with the mechanism, accurate kinematic and dynamic models of the haptic device were developed. Furthermore, the experimental virtual environment setup and the methodology for the assessment were developed. The setup utilizes the haptic device and the virtual environments as a visual feedback. The system was evaluated in a group of stroke patients during a one-month period of therapy. The second part of the thesis presents the system for processing large quantities of the measured data and an automatic report generator, which automatically produces condensed reports in a printable form. The numerical results are stored in a database for further analysis. The results of the study are compared to the Functional Independence Measure – FIM clinical scores, and are well correlated. The recorded data can be used to follow the therapy effects on site as well as at a distance – telerehabilitation.
Measurement and Evaluation of Grasping in Virtual Reality
The thesis is focused on the assessment and rehabilitation of human grasping through measurement of force in virtual environments (VEs). We present a tracking system for the evaluation of grip force control. The system consists of a grip-measuring device with end-objects of different shapes which was used as input to a tracking task. The thesis presents the results obtained in healthy subjects, patients with neuromuscular diseases, head-injury patient after BTX treatment and in a group of post-stroke patients. The presented tracking method is aimed to be used in connection with the existing rehabilitation therapies to follow the influence of the therapy or progress of disease. The second part of the work presents a novel rehabilitation system for the assessment and training of multi-fingered grasping in VE. We designed an isometric finger device to simultaneously assess forces applied by three fingers. Mathematical model of grasping was applied to achieve multi-fingered interaction in VE. Four virtual reality tasks were developed with the aim to improve grip force coordination and increase muscle strength of patients after stroke through repetitive exercises. The presented virtual system was evaluated in a group of healthy subjects and a post-stroke patient.
Estimation of Upper Extremity Biomechanical Parameters using a Robot Manipulator
The thesis deals with studying the internal biomechanical characteristics of the human upper extremity. The problem of determining biomechanical properties in the upper extremity was dealt with in a laboratory environment using industrial robots which are normally used for other tasks in different environments. In the presented experiments we took advantage of the fact that robot movements can be accurately repeated as many times as desired and that several other sensory devices can also be incorporated into the experimental setup enabling us a very wide range of experimental possibilities. In our experiments we used two different types of industrial robots (Yaskawa – Motoman sk6 and Motomation – Stäubli – RX90) to impose accurately controlled robot motions into the upper extremity. The upper extremity was modelled as a simple 3 degree of freedom (3DOF) planar manipulator while all the experiments were performed in the sagittal plane of the studied subject. The upper extremity was then lead through a specified trajectory in space. During this process angle measurements were taken by means of an optical 3D positioning system and end-point forces were measured with a force sensor attached to the robot end-effector.
Arm movement analysis in a haptic environment
The thesis focuses on the design, control, and evaluation of a six-degree of freedom (DOF) haptic device for studying multi-joint arm movements. A general purpose haptic interface with an industrial robot Stäubli RX90 was developed to display a wide range of impedances. The admittance controlled haptic interface is stable for displayed damping of 10 N/ms – 1 and a mass of 1 kg at the control loop sampling rate of 4 kHz. In combination with our powerful haptic interface, a virtual haptic environment in the form of virtual pipelines was developed. For various sets od simulated dynamic conditions were performed arm movement experiments with included on-line electromyographical (EMG) estimation.
Haptic interface for the purpose of quantitative assessment of the upper limb functional state
We are employing haptic technology for the purpose of quantitative assessment of the upper limb functional state. The measurement setup that was developed consists of a powerful multimodal virtual reality simulator, capable of providing quality haptic, visual and audio feedback and capable of network distributed platform independent real-time execution as well. The comprehensive study, using the developed measurement setup within the haptic virtual environment, was carried out. 19 healthy subjects and a total of 75 patients with various neurological and neuro-muscular disorders took part in the study. Data mining techniques have been employed to show the objectivity, repeatability and content validity of the proposed measurement methodology. This mesurement setup and methodology serves as a basis for the EC project I-Match (IST-37280).
Sensory information in gait re-education
The research comprises the use of sensory information in gait reeducation of incomplete spinal cord injured persons. The sensory information was assessed by multisensor device and processed (Kalman filter, Neural Network). The swing phase was estimated during treadmill walking, classified and provided as cognitive feedback. Simultaneously the information was employed in FES control. Whether the person was able to maintain the required swing quality using the cognitive feedback the motor augmentation was decreased and if failed increased. The PhD thesis also comprises the gait analysis with optical measurement system Optotrak.
Optimal standing-up trajectories
Dynamic optimization as a tool to compute standing-up trajectories was investigated. Sit-to-stand manoeuvres in five intact persons and five trans-femoral amputees were measured. In each particular subject, the optimization criterion which yielded trajectories that best resemble the measured standing-up movement was determined. Since the intact persons used considerably different criteria in choosing the standing-up trajectories than the amputees, the optimal trajectories were computed by minimizing cost functionals with distinctive structures for each group of individuals. The PhD thesis also comprises a comprehensive analysis of the sit-to-stand manoeuvre in amputees and a study of robot assisted standing-up in intact persons and amputees.
Control of Unsupported Standing in Complete Spinal Cord Injured Subjects
In the past, limited unsupported standing has been restored in patients with thoracic spinal cord injury through open-loop functional electrical stimulation of paralyzed knee extensor muscles and the support of intact arm musculature. Here an optimal control system for paralyzed ankle muscles was designed that enables the subject to stand without hand support in a sagittal plane. The paraplegic subject was conceptualized as an underactuated double inverted pendulum structure with an active degree of freedom in the upper trunk and a passive degree of freedom in the paralyzed ankle joints. Control system design is based on the minimization of a cost function that estimates the effort of ankle joint muscles via observation of the ground reaction force position, relative to ankle joint axis. Furthermore, such a control system integrates voluntary upper trunk activity and artificial control of ankle joint muscles, resulting in a robust standing posture. Benefits of the presented methodology are prolonged standing sessions and in the fact that the subject is able to maintain voluntary control over upper body orientation in space, enabling simple functional standing.