Language-Guided Invariance Probing of Vision-Language Models

Researchers at an academic institution announced a new benchmark called Language-Guided Invariance Probing (LGIP) on November 23, 2025, to evaluate how vision-language models respond to linguistic perturbations. The benchmark measures invariance to meaning-preserving paraphrases and sensitivity to meaning-changing semantic flips in image-text matching using 40k MS COCO images with five different linguistic variations. The development of LGIP comes as vision-language models like CLIP, OpenCLIP, EVA02-CLIP and SigLIP have demonstrated strong zero-shot performance but have not been thoroughly tested for their reliability when faced with controlled linguistic changes. The research team recognized this gap in evaluation methodologies and created LGIP to provide a standardized way to assess how these models handle different formulations of the same semantic content. By testing against both meaning-preserving paraphrases and meaning-changing semantic flips, the benchmark offers comprehensive insights into the linguistic robustness of current VLMs. The LGIP benchmark utilizes the large-scale MS COCO dataset, which contains 40,000 images, each paired with five different linguistic variations, allowing researchers to systematically evaluate whether vision-language models maintain consistent performance when semantic content remains unchanged but wording differs, and whether they can detect when semantic meaning has been altered.