Abstract:
This thesis addresses the critical challenges hindering the safe and reliable de-
ployment of modern vision models and foundation models, specifically image classifiers,
Vision-Language Models (VLMs), Large Vision-Language Models (LVLMs), and Large Lan-
guage Models (LLMs). We investigate four interconnected domains essential for trustworthy
AI: establishing strong adversarial robustness in both discriminative image classifiers and
multi-modal systems, improving compositional reasoning, and establishing mechanisms for
unlearning to mitigate harmful, outdated, or unintended behaviors, thereby strengthening
model reliability and accountability. While training massive models from scratch is often
computationally prohibitive, we demonstrate that targeted adaptations of pre-trained models
can effectively resolve these challenges without compromising their general capabilities.
Our first contribution revisits adversarial training for modern ImageNet architectures. Prior
work on adversarially robust image classifiers focused almost exclusively on ResNets and small
datasets like CIFAR-10, leaving open how modern architectures such as Vision Transformers
(ViTs) and ConvNeXts behave under adversarial training on ImageNet. We show that
seemingly minor architectural choices, most notably replacing the standard patch-based input
stem with a convolutional stem, combined with strong clean pre-training and heavy data
augmentation, have a large and consistent impact on the achieved robustness. Crucially,
these changes not only improve robustness in the threat model seen at training time, but
substantially improve generalization to unseen ℓ1 and ℓ2 threat models, a problem that had
received little attention before.
Our second contribution centers on adversarial robustness in multi-modal systems. We
robustify the vision encoder of CLIP models, making them capable of defending against visual
adversaries using an unsupervised fine-tuning technique. Crucially, we demonstrate that this
robustness transfers to downstream LVLMs, securing them against stealth attacks where
malicious image manipulations aim to deceive users. Beyond model robustness, we establish
that these adversarially robust encoders serve as superior perceptual metrics that are better
aligned with humans, resilient to manipulation, and essential for reliable content filtering and
dataset curation in real-world applications.
In the third contribution, we tackle the limitations of compositional reasoning in vision-
language models. While current models excel at recognizing individual objects, they often
fail to grasp the nuanced relationships between the objects in the image. This deficiency can
lead to failure modes which propagate to downstream LVLMs that inherit these problems
through frozen encoders. We introduce CLIC (Compositionally-aware Learning in CLIP),
1
a method that enhances both lexical sensitivity and semantic understanding. By refining
how the model processes complex relationships, CLIC improves the compositional reasoning
abilities of the encoder, which is vital for the performance of downstream LVLMs that rely on
CLIP for visual understanding.
Finally, we address reliability, privacy and compliance in Large Language Models through
unlearning, the selective removal of specific knowledge. We introduce JensUn, a novel
framework leveraging Jensen-Shannon Divergence to achieve stable knowledge removal while
preserving the model’s general ability. To rigorously benchmark progress in this nascent field,
we introduce LPF (Less Prominent Facts), a curated dataset of niche topics, and an evaluation
protocol that takes the worst case over different query formats using an LLM as a semantic
judge. These evaluations reveal significant flaws in existing unlearning methods, highlighting
the need for the more robust approach offered by JensUn.
Collectively, these works reveal fundamental connections between seemingly disparate chal-
lenges. The robustness techniques used for adversarial defense in both classifiers and multi-
modal models share conceptual roots, as both require the model to distinguish between
valid and malicious patterns. Similarly, the knowledge manipulation required for unlearning
in LLMs parallels the targeted parameter adjustments studied across all our robustness
work. This thesis suggests that the principles of safety and reliability, whether in robustness,
reasoning, or privacy, span across modalities and architectures.
Robustness