|
Athulith Paraselli
Brown University |
Etha Tianze Hua
Brown University |
Ellie Pavlick
Brown University |
|
|
|
| We investigate how vision-language models (VLMs) handle context-memory conflicts; that is, situations in which the model is given information in context that differs from what was stored parametrically during training. We document asymmetric biases: models tend to prefer in-context information about entities which appear in text, but prefer parametric information about entities which appear in images. We relate this asymmetry to the late representational alignment across modalities, showing that the longer processing time associated with resolving visual entities prevents the suppression of the model's usual factual recall mechanism, thus resulting in more parametric answers. Chain-of-thought reasoning does not appear to resolve the gap, but increasing the amount of visual information in the context does show an effect. These results illustrate the complexity of ensuring consistent behavior as models become increasingly multimodal and retrieval-augmented. |
| We investigate how VLMs handle context-memory conflicts; that is, situations in which the model is given information in context that differs from what was stored parametrically during training. We compare the model's performance when the subjected is prompted with a visual reference versus a textual one. The context is restricted to the textual inputs to reflect default settings of many RAG frameworks. |
|
|
| We evaluate context-memory conflicts by presenting a model with counterfactual in-context information about an entity shown either as text (a) or as an image (b), then measure whether it follows the context or its stored knowledge. |
| We tested Qwen2.5-VL, Gemma 3, and Ministral 3 on three datasets (celebrities, buildings, and artwork). Our results show that VLMs bias towards in-context information when prompted with textual references, however prefer parametric when prompted with visual ones. |
|
| Percentage of parametric answers by relation under context-memory conflict. Δ denotes Vision - Text. The final column reports the observed modality gap together with its 95% bootstrap confidence interval. |
| Our causal experiments reveal parametric enrichment over the MLPs at the entity token is suppressed under conflict for textual entities. Whereas for visual entities the parametric enrichment process is relatively unaffected. We provide further evidence to suggest the robustness of the visual modality is due to a lack of early alignment to textual references. |
|
| Slower visual-entity resolution weakens suppression of parametric enrichment. (a) In standard factual recall, MLPs enrich the entity representation with parametric knowledge, enabling retrieval of associated attributes. (b) For textual entities under context-memory conflict, early attention to the conflicting context suppresses downstream MLP-mediated parametric enrichment, leading the model to favor the in-context answer. (c) For visual entities, entity resolution occurs more slowly, so this suppression arrives too late: parametric enrichment occurs largely unaffected, leading the model to favor the parametric answer. |
| We find that ablating entity token MLPs narrows the modality gap by forcing the model to rely on context for visual entities, while leaving textual entities relatively unaffected. This observation suggests that MLPs for visual entities drive parametric preference, but play a significantly less important role for textual entities. |
|
| Behavioral impact of MLP ablation across entity positions. The x-axis indicates the starting layer of the ablated MLP span. The y-axis represents the models preference. We utilize a constant span of five layers for all the models. |
While MLPs appear responsible for retrieving information about image entities, they have less of
an effect on text entities. To account for
this divergence, we consider two potential explanations:
|
|
| Effect on model preference when masking entity-token attention to the retrieved context. For cutoff layer k, attention is masked in layers 0-k. Solid lines show the standard intervention; dashed lines restore clean-run entity-token MLP outputs. Dash-dotted horizontal lines indicate clean-run margins for each modality. |
| The results strongly support E2 (context sensitive suppression) over E1 (baseline asymmetry). Masking attention to context results in parametric shift for textual case, suggesting parametric signal undergoes suppression under conflict. Masking attention over visual entities produces a small shift, suggesting that visual representations are less affected by the conflicting context. |
| We hypothesize that failure of MLP suppression for visual entities is due to processing time; specifically, visual features may require more layer depth to resolve into text-aligned concepts, allowing them to escape early-layer suppression. To test this late entity alignment hypothesis, we utilize backpatching. Specifically, we patch all the visual tokens' representations from layer K -> layer L/2 . |
|
| Parametric answer reporting under back-patching. Activations from the visual-token span at source layer K (shown on x-axis) are patched into the layer L/2 . The solid line shows the resulting parametric-answer rate; dashed lines mark the default baselines. |
| Across all models, back-patching from later layers decreases the visual parametric preference, ultimately dropping near the text baseline. These findings suggest that the behavioral divergence under conflict stems from a lack of early alignment between the representations of the two modalities. |
| While back-patching demonstrates that the modality gap can be mitigated through internal activation alignment, we investigate whether prompt-based, "black-box" interventions can induce similar convergence across modalities. Specifically, we replace the context textual reference with an image. For example, instead of stating "The Eiffel Tower is located in Tokyo", we provide an image of the Eiffel Tower and state "The entity pictured is located in Tokyo". By requiring the context to first resolve the image, we hypothesize that many of the early entity-enriching MLPs in the textual setting will no longer be suppressed, switching those answers back to parametric. Conversely, when the image in the query matches the image in the context, we expect the model to easily connect them. This clear representation alignment should allow the visual entity to detect the conflict and suppress the MLPs, switching the final answer from parametric to contextual. |
|
| Effect of visual-context prompting on parametric answer rates under context-memory conflict. Filled bars show visual-context results relative to outlined textual-context baselines; labels report the change. Dashed lines indicate the textual baseline. |
| replacing the textual context with an image increases the parametric response rate for text entities across all models, supporting our hypothesis. When we query visual entities using the exact same image found in the context, the parametric rate drops for all three models compared towards the text-entity-text-context rate. However, this effect weakens when we switch to a different image of the same entity. These results further support our mechanistic findings that context-driven suppression requires an early aligned representation between the queried entity and the contextual information. |
| [arXiv] | [Code & Data] |
AcknowledgementsThis template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here. |
