This dissertation focuses on reinforcement learning (RL) controller design aiming for real-life applications in continuous state and control problems. It involves three major research investigations in the aspect of design, analysis, implementation, and evaluation. The application case addresses automatically configuring robotic prosthesis impedance parameters. Major contributions of the dissertation include the following. 1) An “echo control” using the intact knee profile as target is designed to overcome the limitation of a designer prescribed robotic knee profile. 2) Collaborative multiagent reinforcement learning (cMARL) is proposed to directly take into account human influence in the robot control design. 3) A phased actor in actor-critic (PAAC) reinforcement learning method is developed to reduce learning variance in RL. The design of an “echo control” is based on a new formulation of direct heuristic dynamic programming (dHDP) for tracking control of a robotic knee prosthesis to mimic the intact knee profile. A systematic simulation of the proposed control is provided using a human-robot system simulation in OpenSim. The tracking controller is then tested on able-bodied and amputee subjects. This is the first real-time human testing of RL tracking control of a robotic knee to mirror the profile of an intact knee. The cMARL is a new solution framework for the human-prosthesis collaboration (HPC) problem. This is the first attempt at considering human influence on human-robot walking with the presence of a reinforcement learning controlled lower limb prosthesis. Results show that treating the human and robot as coupled and collaborating agents and using an estimated human adaptation in robot control design help improve human walking performance. The above studies have demonstrated great potential of RL control in solving continuous problems. To solve more complex real-life tasks with multiple control inputs and high dimensional state space, high variance, low data efficiency, slow learning or even instability are major roadblocks to be addressed. A novel PAAC method is proposed to improve learning performance in policy gradient RL by accounting for both Q value and TD error in actor updates. Systematical and comprehensive demonstrations show its effectiveness by qualitative analysis and quantitative evaluation in DeepMind Control Suite.

Although previous studies have elucidated the role of position feedback in the regulation of movement, the specific contribution of Golgi tendon organs (GTO) in force feedback, especially in stabilizing voluntary limb movements, has remained theoretical due to limitations in experimental techniques. This study aims to establish force feedback regulation mediated by GTO afferent signals in two phases. The first phase of this study consisted of simulations using a neuromusculoskeletal model of the monoarticular elbow flexor (MEF) muscle group, assess the impact of force feedback in maintaining steady state interaction forces against variable environmental stiffness. Three models were trained to accurately reach an interaction force of 40N, 50N and 60N respectively, using a fixed stiffness level. Next, the environment stiffness was switched between untrained levels for open loop (OL) and closed loop (CL) variants of the same model. Results showed that compared to OL, CL showed decreased force deviations by 10.43%, 12.11% and 13.02% for each of the models. Most importantly, it is also observed that in the absence of force feedback, environment stiffness is found to have an effect on the interaction force. In the second phase, human subjects were engaged in experiments utilizing an instrumented elbow exoskeleton that applied loads to the MEF muscle group, closely mimicking the simulation conditions. The experiments consisted of reference, blind and catch trial types, and 3 stiffness levels. Subjects were first trained to reach for a predetermined target force. During catch trials, stiffness levels were randomized between reaches. Responses obtained from these experiments showed that subjects were able to regulate forces with no significant effects of trial type or stiffness level. Since experimental results align closely with that of closed loop model simulations, the presence of force feedback mechanisms mediated by GTO within the human neuromuscular system is established. This study not only unveils the critical involvement of GTO in force feedback but also emphasizes the importance of understanding these mechanisms for developing advanced neuroprosthetics and rehabilitation strategies, shedding light on the intricate interplay between sensory inputs and motor responses in human proprioception.

A current thrust in neurorehabilitation research involves exogenous neuromodulation of peripheral nerves to enhance neuroplasticity and maximize recovery of function. This dissertation presents the results of four experiments aimed at assessing the effects of trigeminal nerve stimulation (TNS) and occipital nerve stimulation (ONS) on motor learning, which was behaviorally characterized using an upper extremity visuomotor adaptation paradigm. In Aim 1a, the effects of offline TNS using clinically tested frequencies (120 and 60 Hz) were characterized. Sixty-three participants (22.75±4.6 y/o), performed a visuomotor rotation task and received TNS before encountering rotation of hand visual feedback. In Aim 1b, TNS at 3 kHz, which has been shown to be more tolerable at higher current intensities, was evaluated in 42 additional subjects (23.4±4.6 y/o). Results indicated that 3 kHz stimulation accelerated learning while 60 Hz stimulation slowed learning, suggesting a frequency-dependent effect on learning. In Aim 2, the effect of online TNS using 120 and 60 Hz were characterized to determine if this protocol would deliver better outcomes. Sixty-three participants (23.2±3.9 y/o) received either TNS or sham concurrently with perturbed visual feedback. Results showed no significant differences among groups. However, a cross-study comparison of results obtained with 60 Hz offline TNS showed a statistically significant improvement in learning rates with online stimulation relative to offline, suggesting a timing-dependent effect on learning. In Aim 3, TNS and ONS were compared using the best protocol from previous aims (offline 3 kHz). Additionally, concurrent stimulation of both nerves was explored to look for potential synergistic effects. Eighty-four participants (22.9±3.2 y/o) were assigned to one of four groups: TNS, ONS, TNS+ONS, and sham. Visual inspection of learning curves revealed that the ONS group demonstrated the fastest learning among groups. However, statistical analyses did not confirm this observation. In addition, the TNS+ONS group appeared to learn faster than the sham and TNS groups but slower than the ONS only group, suggesting no synergistic effects using this protocol, as initially hypothesized. The results provide new information on the potential use of TNS and ONS in neurorehabilitation and performance enhancement in the motor domain.

Multisensory integration is the process by which information from different sensory modalities is integrated by the nervous system. This process is important not only from a basic science perspective but also for translational reasons, e.g., for the development of closed-loop neural prosthetic systems. A mixed virtual reality platform was developed to study the neural mechanisms of multisensory integration for the upper limb during motor planning. The platform allows for selection of different arms and manipulation of the locations of physical and virtual target cues in the environment. The system was tested with two non-human primates (NHP) trained to reach to multiple virtual targets. Arm kinematic data as well as neural spiking data from primary motor (M1) and dorsal premotor cortex (PMd) were collected. The task involved manipulating visual information about initial arm position by rendering the virtual avatar arm in either its actual position (veridical (V) condition) or in a different shifted (e.g., small vs large shifts) position (perturbed (P) condition) prior to movement. Tactile feedback was modulated in blocks by placing or removing the physical start cue on the table (tactile (T), and no-tactile (NT) conditions, respectively). Behaviorally, errors in initial movement direction were larger when the physical start cue was absent. Slightly larger directional errors were found in the P condition compared to the V condition for some movement directions. Both effects were consistent with the idea that erroneous or reduced information about initial hand location led to movement direction-dependent reach planning errors. Neural correlates of these behavioral effects were probed using population decoding techniques. For small shifts in the visual position of the arm, no differences in decoding accuracy between the T and NT conditions were observed in either M1 or PMd. However, for larger visual shifts, decoding accuracy decreased in the NT condition, but only in PMd. Thus, activity in PMd, but not M1, may reflect the uncertainty in reach planning that results when sensory cues regarding initial hand position are erroneous or absent.

This project investigates the gleam-glum effect, a well-replicated phonetic emotion association in which words with the [i] vowel-sound (as in “gleam”) are judged more emotionally positive than words with the [Ʌ] vowel-sound (as in “glum”). The effect is observed across different modalities and languages and is moderated by mouth movements relevant to word production. This research presents and tests an articulatory explanation for this association in three experiments. Experiment 1 supported the articulatory explanation by comparing recordings of 71 participants completing an emotional recall task and a word read-aloud task, showing that oral movements were more similar between positive emotional expressions and [i] articulation, and negative emotional expressions and [Ʌ] articulation. Experiment 2 partially supported the explanation with 98 YouTube recordings of natural speech. In Experiment 3, 149 participants judged emotions expressed by a speaker during [i] and [Ʌ] articulation. Contradicting the robust phonetic emotion association, participants judged more frequently that the speaker’s [Ʌ] articulatory movements were positive emotional expressions and [i] articulatory movements were negative emotional expressions. This is likely due to other visual emotional cues not related to oral movements and the order of word lists read by the speaker. Findings from the current project overall support an articulatory explanation for the gleam-glum effect, which has major implications for language and communication.

With an aging population, the number of later in life health related incidents like stroke stand to become more prevalent. Unfortunately, the majority those who are most at risk for debilitating heath episodes are either uninsured or under insured when it comes to long term physical/occupational therapy. As insurance companies lower coverage and/or raise prices of plans with sufficient coverage, it can be expected that the proportion of uninsured/under insured to fully insured people will rise. To address this, lower cost alternative methods of treatment must be developed so people can obtain the treated required for a sufficient recovery. The presented robotic glove employs low cost fabric soft pneumatic actuators which use a closed loop feedback controller based on readings from embedded soft sensors. This provides the device with proprioceptive abilities for the dynamic control of each independent actuator. Force and fatigue tests were performed to determine the viability of the actuator design. A Box and Block test along with a motion capture study was completed to study the performance of the device. This paper presents the design and classification of a soft robotic glove with a feedback controller as a at-home stroke rehabilitation device.

Information processing in the brain is mediated by network interactions between anatomically distant (centimeters apart) regions of cortex and network action is fundamental to human behavior. Disruptive activity of these networks may allow a variety of diseases to develop. Degradation or loss of network function in the brain can affect many aspects of the human experience; motor disorder, language difficulties, memory loss, mood swings, and more. The cortico-basal ganglia loop is a system of networks in the brain between the cortex, basal ganglia, the thalamus, and back to the cortex. It is not one singular circuit, but rather a series of parallel circuits that are relevant towards motor output, motor planning, and motivation and reward. Studying the relationship between basal ganglia neurons and cortical local field potentials may lead to insights about neurodegenerative diseases and how these diseases change the cortico-basal ganglia circuit. Speech and language are uniquely human and require the coactivation of several brain regions. The various aspects of language are spread over the temporal lobe and parts of the occipital, parietal, and frontal lobe. However, the core network for speech production involves collaboration between phonologic retrieval (encoding ideas into syllabic representations) from Wernicke’s area, and phonemic encoding (translating syllables into motor articulations) from Broca’s area. Studying the coactivation of these brain regions during a repetitive speech production task may lead to a greater understanding of their electrophysiological functional connectivity. The primary purpose of the work presented in this document is to validate the use of subdural microelectrodes in electrophysiological functional connectivity research as these devices best match the spatial and temporal scales of brain activity. Neuron populations in the cortex are organized into functional units called cortical columns. These cortical columns operate on the sub-millisecond temporal and millimeter spatial scale. The study of brain networks, both in healthy and unwell individuals, may reveal new methodologies of treatment or management for disease and injury, as well as contribute to our scientific understanding of how the brain works.

Repetitive practice of functional movement patterns during motor rehabilitation are known to drive learning (or relearning) of novel motor skills, but the learning process is highly variable between individuals such that responsiveness to task-specific training is often patient-specific. A number of neuroimaging and neurophysiological methods have been proposed to better predict a patient’s responsiveness to a given type or dose of motor therapy. However, these methods are often time- and resource-intensive, and yield results that are not readily interpretable by clinicians. In contrast, standardized visuospatial tests may offer a more feasible solution. The work presented in this dissertation demonstrate that a clinical paper-and-pencil test of visuospatial function may improve predictive models of motor skill learning in older adults and individuals with stroke pathology. To further our understanding of the neuroanatomical correlates underlying this behavioral relationship, I collected diffusion-weighted magnetic resonance images from 19 nondemented older adults to determine if diffusion characteristics of white matter tracts explain shared variance in delayed visuospatial memory test scores and motor skill learning. Consistent with previous work, results indicated that the structural integrity of regions with the bilateral anterior thalamic radiations, corticospinal tracts, and superior longitudinal fasciculi are related to delayed visuospatial memory performance and one-week skill retention. Overall, results of this dissertation suggest that incorporating a clinical paper-and-pencil test of delayed visuospatial memory may prognose motor rehabilitation outcomes and support that personalized variables should be considered in standards of care. Moreover, regions within specific white matter tracts may underlie this behavioral relationship and future work should investigate these regions as potential targets for therapeutic intervention.

In recent years, brain signals have gained attention as a potential trait for biometric-based security systems and laboratory systems have been designed. A real-world brain-based security system requires to be usable, accurate, and robust. While there have been developments in these aspects, there are still challenges to be met. With regard to usability, users need to provide lengthy amount of data compared to other traits such as fingerprint and face to get authenticated. Furthermore, in the majority of works, medical sensors are used which are more accurate compared to commercial ones but have a tedious setup process and are not mobile. Performance wise, the current state-of-art can provide acceptable accuracy on a small pool of users data collected in few sessions close to each other but still falls behind on a large pool of subjects over a longer time period. Finally, a brain security system should be robust against presentation attacks to prevent adversaries from gaining access to the system. This dissertation proposes E-BIAS (EEG-based Identification and Authentication System), a brain-mobile security system that makes contributions in three directions. First, it provides high performance on signals with shorter lengths collected by commercial sensors and processed with lightweight models to meet the computation/energy capacity of mobile devices. Second, to evaluate the system's robustness a novel presentation attack was designed which challenged the literature's presumption of intrinsic liveness property for brain signals. Third, to bridge the gap, I formulated and studied the brain liveness problem and proposed two solution approaches (model-aware & model agnostic) to ensure liveness and enhance robustness against presentation attacks. Under each of the two solution approaches, several methods were suggested and evaluated against both synthetic and manipulative classes of attacks (a total of 43 different attack vectors). Methods in both model-aware and model-agnostic approaches were successful in achieving an error rate of zero (0%). More importantly, such error rates were reached in face of unseen attacks which provides evidence of the generalization potentials of the proposed solution approaches and methods. I suggested an adversarial workflow to facilitate attack and defense cycles to allow for enhanced generalization capacity for domains in which the decision-making process is non-deterministic such as cyber-physical systems (e.g. biometric/medical monitoring, autonomous machines, etc.). I utilized this workflow for the brain liveness problem and was able to iteratively improve the performance of both the designed attacks and the proposed liveness detection methods.