EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

Hugging Face Daily Papers Papers
face-recognition explainability vision-language-alignment embedding-space adapter model-auditing semantic-signature

Summary

EXPL-FR is a method that explains face recognition models by aligning vision-language embeddings to the recognition space, enabling label-free auditing of semantic attributes and model comparison without requiring architecture access.

Deep face recognition (FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model's own embedding space. A lightweight adapter aligns a vision-language model's (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM's encoders share one space, the same adapter applies to the text encoder, turning 978 attribute prompts in 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only the adapter isolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-free detectability measure compares each concept's separability in FR space against the VLM space, and the 100 most detectable form the model's readable semantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fully prompt-driven audit, against real verification behavior. With no labels, the prompt-driven audit ranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.
Original Article
View Cached Full Text

Cached at: 08/25/26, 12:35 PM

Paper page - EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

Source: https://huggingface.co/papers/2608.21486

Abstract

EXPL-FR explains face recognition similarity scores by aligning vision-language embeddings to the recognition space, enabling label-free auditing of semantic attributes and model comparison.

Deep face recognition(FR) models reach near-saturated accuracy but remain opaque: a practitioner cannot ask which semantic attributes a similarity score relied upon. EXPL-FR answers this inside the FR model’s ownembedding space. A lightweightadapteraligns avision-language model’s (VLM) image encoder with the frozen FR space, trained on face images alone and never on text. Because the VLM’s encoders share one space, the sameadapterapplies to the text encoder, turning 978attribute promptsin 22 categories, also extendable, into FR-space anchors at no extra cost. We do not assume this transfer works: a face-verification protocol measures it, and an ablation changing only theadapterisolates its contribution. Not every concept survives, because an FR model earns its invariances by discarding the factors it must verify identities across. A label-freedetectability measurecompares each concept’s separability in FR space against the VLM space, and the 100 most detectable form the model’s readablesemantic signature, which separates identities better than the full vocabulary. We cover four FR backbones and two VLM encoders, EXPL-FR needs no architecture access, and supports identity-level, per-image, and differential explanations. We benchmark attribute-level auditing under three supervision settings, human labels (current practice), VLM pseudo-labels, and our fullyprompt-driven audit, against real verification behavior. With no labels, theprompt-driven auditranks four FR models by their measured per-ethnicity RFW errors and ranks controlled attribute changes by their true verification cost.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2608\.21486

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.21486 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.21486 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.21486 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models

arXiv cs.LG

This paper introduces KODA (Kernel Optimization for Discrepancy Analysis), a kernel-based framework for comparing and aligning vision-language model representations by identifying sample subsets that are clustered differently across models like CLIP, SigLIP, and BLIP. The method uses contrastive embedding clustering and randomized low-dimensional approximations to scale to large datasets while providing interpretable structural differences between representations.

VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation

arXiv cs.CL

This paper proposes Vision-Free Adaptation (VFA), a framework that enhances multilingual capabilities in multimodal large language models by merging multilingual and vision-aligned task vectors without visual data, demonstrating improved performance and data efficiency.

Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Models

arXiv cs.CL

This paper presents the first systematic study of multilingual instruction following in Vision-Language-Action (VLA) models, revealing significant performance degradation when models trained on English are evaluated on other languages. The authors propose Multilingual Principal Component Alignment (MPCA) to reduce the multilingual performance gap.

Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

Hugging Face Daily Papers

Investigates spatial representation in vision-language models, revealing a consistent bias where models conflate vertical image position with distance, and introduces SpatialTunnel synthetic benchmark to expose this shortcut; finds that better disentangled spatial representations improve robustness.