Fortifying Foundation Models: Fine-Tuning for Robustness, Compositionality and Safety

Abstract

This thesis addresses the critical challenges hindering the safe and reliable de- ployment of modern vision models and foundation models, specifically image classifiers, Vision-Language Models (VLMs), Large Vision-Language Models (LVLMs), and Large Lan- guage Models (LLMs). We investigate four interconnected domains essential for trustworthy AI: establishing strong adversarial robustness in both discriminative image classifiers and multi-modal systems, improving compositional reasoning, and establishing mechanisms for unlearning to mitigate harmful, outdated, or unintended behaviors, thereby strengthening model reliability and accountability. While training massive models from scratch is often computationally prohibitive, we demonstrate that targeted adaptations of pre-trained models can effectively resolve these challenges without compromising their general capabilities. Our first contribution revisits adversarial training for modern ImageNet architectures. Prior work on adversarially robust image classifiers focused almost exclusively on ResNets and small datasets like CIFAR-10, leaving open how modern architectures such as Vision Transformers (ViTs) and ConvNeXts behave under adversarial training on ImageNet. We show that seemingly minor architectural choices, most notably replacing the standard patch-based input stem with a convolutional stem, combined with strong clean pre-training and heavy data augmentation, have a large and consistent impact on the achieved robustness. Crucially, these changes not only improve robustness in the threat model seen at training time, but substantially improve generalization to unseen ℓ1 and ℓ2 threat models, a problem that had received little attention before. Our second contribution centers on adversarial robustness in multi-modal systems. We robustify the vision encoder of CLIP models, making them capable of defending against visual adversaries using an unsupervised fine-tuning technique. Crucially, we demonstrate that this robustness transfers to downstream LVLMs, securing them against stealth attacks where malicious image manipulations aim to deceive users. Beyond model robustness, we establish that these adversarially robust encoders serve as superior perceptual metrics that are better aligned with humans, resilient to manipulation, and essential for reliable content filtering and dataset curation in real-world applications. In the third contribution, we tackle the limitations of compositional reasoning in vision- language models. While current models excel at recognizing individual objects, they often fail to grasp the nuanced relationships between the objects in the image. This deficiency can lead to failure modes which propagate to downstream LVLMs that inherit these problems through frozen encoders. We introduce CLIC (Compositionally-aware Learning in CLIP), 1 a method that enhances both lexical sensitivity and semantic understanding. By refining how the model processes complex relationships, CLIC improves the compositional reasoning abilities of the encoder, which is vital for the performance of downstream LVLMs that rely on CLIP for visual understanding. Finally, we address reliability, privacy and compliance in Large Language Models through unlearning, the selective removal of specific knowledge. We introduce JensUn, a novel framework leveraging Jensen-Shannon Divergence to achieve stable knowledge removal while preserving the model’s general ability. To rigorously benchmark progress in this nascent field, we introduce LPF (Less Prominent Facts), a curated dataset of niche topics, and an evaluation protocol that takes the worst case over different query formats using an LLM as a semantic judge. These evaluations reveal significant flaws in existing unlearning methods, highlighting the need for the more robust approach offered by JensUn. Collectively, these works reveal fundamental connections between seemingly disparate chal- lenges. The robustness techniques used for adversarial defense in both classifiers and multi- modal models share conceptual roots, as both require the model to distinguish between valid and malicious patterns. Similarly, the knowledge manipulation required for unlearning in LLMs parallels the targeted parameter adjustments studied across all our robustness work. This thesis suggests that the principles of safety and reliability, whether in robustness, reasoning, or privacy, span across modalities and architectures.