Robust and Reliable Deep Learning by Synergizing with Pre-Trained Models

Over the past decade, the success of deep learning models has largely depended on the availability of extensive training data and the assumption that training and target data distributions are independent and identically distributed (i.i.d.). However, deviations from these conditions often reveal the models' brittleness, as they struggle with distribution shifts—discrepancies between training and testing datasets arising from various factors. To address this challenge, this dissertation leverages Generative Adversarial Networks (GANs) to parameterize distribution shifts, using a novel single-shot target aware (SiSTA) adaptation technique. This approach updates a GAN with a target domain example to generate synthetic samples that facilitate the effective adaptation of predictive models to target conditions. Beyond synthetic data generation, GANs are integral to digital imaging restoration tasks such as image denoising and super-resolution. However, they often perform poorly when the input data deviates from the training distribution. To address this, a technique called SPHInX (Style Projection Heads for Inverting X) is developed to enhance GAN inversion capabilities, thereby improving the model's ability to handle out-of-distribution images. However, GANs struggle with semantic shifts caused by label shifts and class imbalances. Vision-Language Models (VLMs) are more effective in these scenarios. This dissertation introduces CREPE (CLIP Representation Enhanced Predicate Estimation), a framework that leverages VLMs to improve nuanced visual relationship prediction by better contextualizing the visual representations. A key part of mitigating model failures involves understanding when and why these failures occur. This dissertation proposes a novel strategy for estimating failures by parameterizing the decision rules learned by predictive models through VLMs. This approach refines the mechanism for failure estimation, allowing for more precise identification and correction of failures across various scenarios. When there is a scarcity of data, the challenge becomes even more pronounced. In such cases, the problem is commonly modeled as a distribution of small tasks. This dissertation addresses this issue by exploring the use of knowledge graphs to dynamically modulate the weights of predictive models. This approach enables the models to adapt their decision rules effectively, enhancing flexibility and effectiveness in real-world applications. Overall, this dissertation presents robust methodologies for understanding and mitigating adverse effects of distribution shifts {on the performance of deep learning models, significantly advancing the adaptability and reliability of these models in dynamic environments. These contributions lay a foundation for future research into developing artificial intelligence systems that are capable of sustaining reliable performance across varying conditions.