Audio descriptions (AD) make videos accessible for blind and low vision (BLV) users by describing visual elements that cannot be understood from the main audio track. AD created by professionals or novice describers is time-consuming and lacks scalability while offering little control to BLV viewers on description length and content and when they receive it. To address this gap, this work explores user-driven AI-generated descriptions, where the BLV viewer controls when they receive descriptions. In a study, 20 BLV participants activated audio descriptions for seven different video genres with two levels of detail: concise and detailed. Results show differences in AD frequency and level of detail BLV users wanted for different videos, their sense of control with this style of AD delivery, its limitations, and variations among BLV users in their AD needs and perception of AI-generated descriptions. The implications of these findings for future AI-based AD tools are discussed.

This thesis explores the impact of artificial intelligence (AI) interventions on player decision-making in an investment-themed serious game, The Strategists. This Monopoly®-inspired game incorporates rule-based advice, machine learning-based predictions, and data-driven visualizations to create a unique platform for studying user interactions with AI-assisted decision support systems. The research investigates how players perceive and utilize AI-driven features in a game environment and how these interventions influence gameplay outcomes. Through a series of user studies involving 37 participants across 29 game sessions, the study compares player behavior and performance between control groups with and without access to AI features. Key findings reveal that players who followed AI advice more frequently were likelier to win the game. However, the study also uncovered an interesting paradox: players with more gaming experience tend to distrust and undervalue AI assistance despite evidence of its effectiveness. The findings show gender differences in gameplay outcomes, with female players more likely to win than male players. Players who finished last viewed AI advice more frequently but were less likely to follow it, suggesting a complex relationship between advice visibility and following. This research adds to human-computer interaction and game studies by providing insights into AI-driven decision support systems design in gaming contexts. It also addresses the need for interdisciplinary approaches in game research by combining elements from game design, artificial intelligence, data visualization, and financial education. The study highlights the significance of understanding user perceptions and trust in AI systems.

Artificial Intelligence (AI) technology has advanced significantly, enabling AI models to learn from diverse data and automate tasks previously performed solely by humans. This capability has excited people from various domains outside the AI research community and has driven technical experts from non-AI backgrounds to leverage AI for their domain-specific tasks. However, when these experts attempt to use AI, they face several challenges: understanding AI models' behavior and results in intuitive ways, adapting pre-trained models to their own datasets, and finding comprehensive guidelines for AI integration practices. This dissertation takes a user-centered approach to address these challenges by designing and developing interactive systems and frameworks that make AI more interpretable and accessible. The dissertation focuses on three key areas: Explaining AI Behavior: For domain experts from a non-AI background, understanding AI models is challenging. Automated explanations alone often fall short, as users need an iterative approach to form, test, and refine hypotheses. We introduce two visual analytics systems, ConceptExplainer and ASAP, designed to provide intuitive explanations and analysis tools, helping users better comprehend and interpret AI's inner workings and outcomes. Simplifying AI Workflows: Adapting pre-trained AI models for specific downstream tasks can be challenging for users with limited AI expertise. We present InFiConD, an interactive no-code interface for streamlining the knowledge distillation process, allowing users to easily adapt large models to their specific needs. Integrating AI in Domain-Specific Tasks: The integration of AI into domain-specific visual analytics systems is growing, but clear guidance is often lacking. Users with non-AI backgrounds face challenges in selecting appropriate AI tools and avoiding common pitfalls due to the vast array of available options. Our survey, AI4DomainVA, addresses this gap by reviewing existing practices and developing a roadmap for AI integration. This guide helps domain experts understand the synergy between AI and visual analytics, choose suitable AI methods, and effectively incorporate them into their workflows.

Machine Learning and Artificial Intelligence algorithms are deeply embedded in our everyday experiences, from the moment we wake up to tracking our REM sleep through wearable devices. These technologies are not only applied to a wide array of challenges but have also become ubiquitous, with many people interacting and relying on AI decisions - often without fully understanding how these models work.Despite the widespread use of AI, the reasoning behind its decisions is frequently opaque. This lack of transparency can lead to users either underestimating or overestimating the capabilities of AI systems. Even in scenarios where AI models provide explanations for their decisions, the impact of these justifications on end-users' perceptions remains unclear. These issues raise important questions about ways to improve model transparency, methods to aid in decision making and understanding the impact of such explanations on user's trust. To address these issues, this thesis focuses on: 1. Explainability of Reinforcement Learning for non expert users: From 2019 to 2023, HCI saw 154 Explainable ML papers in many HCI conferences like IEEEVis, EuroVis and CHI. Comparatively, field of explainable RL (XRL) has however been underdeveloped. I contributed two novel visualization-driven systems for explainable RL: PolicyExplainer that provides a visual explanations and PolicySummarizer that provides policy summaries for novice users. 2. Accessibility to perform downstream decision making tasks: As AI becomes more accessible, users may struggle to leverage these tools within their limits. I studied how to design effective interfaces for identifying underlying coverage diversity among different news organizations, operationalized as NewsKaleidoscope that helps users identify coverage biases in news and PromptAid to offer AI-based prompt recommendations for better task performance. 3. User perceptions and impact of machine generated rationales: Finally, explanations can be a double-edged sword, potentially increasing trust in flawed models. I explored how user perceptions are affected by rationales generated by ChatGPT, especially when it justifies incorrect predictions due to hallucinations and ways to circumvent the issue.

Robot learning aims to enable robots to acquire new skills and adapt to their environment through advanced learning algorithms. As an embodiment of AI, robots continue to face the challenges of precisely estimating a robot’s state across varied environments and executing actions based on these state estimates. Although many approaches focus on developing end-to-end models and policies, they often lack explainability and do not effectively integrate algorithmic priors to understand the underlying robot models. This thesis addresses the challenges of robot learning through the application of state-space models, demonstrating their efficacy in representing a wide range of robotic systems within a differentiable Bayesian framework that integrates states, observations, and actions. It establishes that foundational state-space models possess the adaptability to be learned through data-driven approaches, enabling robots to accurately estimate their states from environmental interactions and to use these estimated states to execute more complex tasks. Additionally, the thesis shows that state-space modeling can be effectively applied in multimodal settings by learning latent state representations for sensor fusion. Furthermore, it demonstrates that state-space models can be utilized to impose conditions on robot policy networks, thereby enhancing their performance and consistency. The practical implications of deep state-space models are evaluated across a variety of robot manipulation tasks in both simulated and real-world environments, including pick-and-place operations and manipulation in dynamic contexts. The state estimation methods are also applied to soft robot systems, which present significant modeling challenges. In the final part, the thesis discusses the connection between robot learning and foundation models, exploring whether state-space agents based on large language models (LLMs) serve as a more conducive reasoning framework for robot learning. It further explores the use of foundation models to enhance data quality, demonstrating improved success rates for robot policy networks with enriched task context.

While the growing prevalence of robots in industry and daily life necessitatesknowing how to operate them safely and effectively, the steep learning curve of programming languages and formal AI education is a barrier for most beginner users. This thesis presents an interactive platform which leverages a block based programming interface with natural language instructions to teach robotics programming to novice users. An integrated robot simulator allows users to view the execution of their high-level plan, with the hierarchical low level planning abstracted away from them. Users are provided human-understandable explanations of their planning failures and hints using LLMs to enhance the learning process. The results obtained from a user study conducted with students having minimal programming experience show that JEDAI-Ed is successful in teaching robotic planning to users, as well as increasing their curiosity about AI in general.

This research project seeks to develop an innovative data visualization tool tailored for beginners to enhance their ability to interpret and present data effectively. Central to the approach is creating an intuitive, user-friendly interface that simplifies the data visualization process, making it accessible even to those with no prior background in the field. The tool will introduce users to standard visualization formats and expose them to various alternative chart types, fostering a deeper understanding and broader skill set in data representation. I plan to leverage innovative visualization techniques to ensure the tool is compelling and engaging. An essential aspect of my research will involve conducting comprehensive user studies and surveys to assess the tool's impact on enhancing data visualization competencies among the target audience. Through this, I aim to gather valuable insights into the tool's usability and effectiveness, enabling further refinements. The outcome of this project is a powerful and versatile tool that will be an invaluable asset for students, researchers, and professionals who regularly engage with data. By democratizing data visualization skills, I envisage empowering a broader audience to comprehend and creatively present complex data in a more meaningful and impactful manner.

Socially assistive robots (SARs) can act as assistants and caregivers, interacting and communicating with people through touch gestures. There has been ongoing research on using them as companion robots for children with autism as therapy assistants and playmates. Building touch-perception systems for social robots has been a challenge. The sensors must be designed to ensure comfortable and natural user interaction while recording high-quality data. The sensor must be able to detect touch gestures. Accurate touch gesture classification is challenging as different users perform the same touch gesture in their own unique way. This study aims to build and evaluate a skin-like sensor by replicating a recent paper introducing a novel silicone-based sensor design. A touch gesture classification is performed using deep-learning models to classify touch gestures accurately. This study focuses on 8 gestures: Fistbump, Hitting, Holding, Poking, Squeezing, Stroking, Tapping, and Tickling. They were chosen based on previous research where specialists determined which gestures were essential to detect while interacting with children with autism. In this work, a user study data collection was conducted with 20 adult subjects, using the skin-like sensor to record gesture data and a load cell underneath to record the force. Three different types of input were used for the touch gesture classification: skin-like sensor & load cell data, only skin-like sensor data, and only load cell data. A Convolutional Neural Network-Long Short Term Memory (CNN-LSTM) neural network architecture was developed for inputs with skin-like sensor data, and an LSTM network for only load cell data. This work achieved an average accuracy of 94% with skin-like sensor & load cell data, 95% for only skin-like sensor data, and 45% for only load cell data after a stratified 10-fold validation. This work also performed subject-dependent splitting and achieved accuracies of 69% skin-like sensor & load cell data, 66% for only skin-like sensor data, and 31% for only load cell data, respectively.

Mid-air ultrasound haptic technology can enhance user interaction and immersion in extended reality (XR) applications through contactless touch feedback. However, existing design tools for mid-air haptics primarily support the creation of static tactile sensations (tactons), which lack adaptability at runtime. These tactons do not offer the required expressiveness in interactive scenarios where a continuous closed-loop response to user movement or environmental states is desirable. This thesis proposes AdapTics, a toolkit featuring a graphical interface for the rapid prototyping of adaptive tactons—dynamic sensations that can adjust at runtime based on user interactions, environmental changes, or other inputs. A software library and a Unity package accompany the graphical interface to enable integration of adaptive tactons in existing applications. The design space provided by AdapTics for creating adaptive mid-air ultrasound tactons is presented, along with evidence that the design tool enhances Creativity Support Index ratings for Exploration and Expressiveness, as demonstrated in a user study involving 12 XR and haptic designers.

Mid-air haptic technologies enhance user interaction by projecting tactile feedback directly onto a user’s skin without requiring physical contact with a feedback device. This thesis details the research and development of a graphical editor which designs 2D curves and 3D surfaces for mid-air haptic parameter control. The editor enables developers working on applications that interface with mid-air haptic technologies to graphically model the relationship between one or two numeric parameters and a modulating phased array output—namely amplitude, amplitude modulation frequency, or drawing frequency. The editor leverages interaction models tailored to the purposes of phased array parameter control, and incorporates just noticeable differences to give developers visual information about end user haptic perception.