Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Hugging Face Daily Papers Papers
computer-vision video-understanding compositional-generalization zero-shot-learning action-recognition shortcut-learning

Summary

This paper investigates object-driven shortcuts that hinder compositional generalization in zero-shot compositional action recognition, proposing RCORE to mitigate verb-collapse and improve unseen composition generalization.

Zero-Shot Compositional Action Recognition (ZS-CAR) requires recognizing novel verb-object combinations composed of previously observed primitives. In this work, we tackle a key failure mode: models predict verbs via object-driven shortcuts (i.e., relying on the labeled object class) rather than temporal evidence. We argue that sparse compositional supervision and verb-object learning asymmetry can promote object-driven shortcut learning. Our analysis with proposed diagnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weak generalization to unseen compositions. To address object-driven shortcuts, we propose Robust COmpositional REpresentations (RCORE) with two components. Co-occurrence Prior Regularization (CPR) adds explicit supervision for unseen compositions and regularizes the model against frequent co-occurrence priors by treating them as hard negatives. Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity to learn temporally grounded verb representations. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consequently improves compositional generalization.
Original Article
View Cached Full Text

Cached at: 07/10/26, 06:15 AM

Paper page - Why Can’t I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Source: https://huggingface.co/papers/2601.16211 TL;DR: We quantify how object-driven shortcuts sabotage compositional generalization in video understanding.

A model that has seen “Open Window” and “Close Drawer” should be able to recognize “Open Drawer”—an unseen yet plausible verb–object combination. But we found that when existing models fail, it’s mostly a verb-collapse phenomenon: they get the object right and the verb wrong. The model simply assumes the verb that is most often paired with that object in training and thus fails to generalize to unseen compositions.

We show that these object-driven shortcuts are not a matter of insufficient model capacity or pretrained knowledge. They stem from two intertwined, structural problems: (i) sparse compositional supervision and (ii) the learning asymmetry between verbs and objects (objects are simply easier to learn than verbs). We prove this empirically, and we further propose a suite of diagnostic tools that measure exactly how far a model overfits to co-occurrence statistics.

Our diagnostic-driven fix—RCORE (Robust COmpositional REpresentations)—directly mitigates both root causes and delivers consistent gains in unseen-composition generalization across multiple benchmarks.

Similar Articles