Learning to compose diversified prompts for image emotion classification
摘要
Image emotion classification(IEC)aims to extract the abstract emotions evoked in images.Recently,language-supervised methods such as con-trastive language-image pretraining(CLIP)have demonstrated superior performance in image under-standing.However,the underexplored task of IEC presents three major challenges:a tremendous training objective gap between pretraining and IEC,shared suboptimal prompts,and invariant prompts for all instances.In this study,we propose a general framework that effectively exploits the language-supervised CLIP method for the IEC task.First,a prompt-tuning method that mimics the pretraining objective of CLIP is introduced,to exploit the rich image and text semantics associated with CLIP.Subsequently,instance-specific prompts are automatically composed,conditioning them on the categories and image content of instances,diversifying the prompts,and thus avoiding suboptimal problems.Evaluations on six widely used affective datasets show that the proposed method significantly outperforms state-of-the-art methods(up to 9.29％accuracy gain on the EmotionROI dataset)on IEC tasks with only a few trained parameters.The code is publicly available at research purposes.
Author：
Sinuo Deng
Lifang Wu
Ge Shi
Lehao Xing
Meng Jian
Ye Xiang
Ruihai Dong
作者单位：
Faculty of Information Technology,Beijing University of Technology,Beijing 100124,China
Insight Centre for Data Analytics,University College Dublin,Belfield,Dublin D04 V1W8,Ireland
年，卷(期)：
2024, 10(6)
机标分类号：
R338.1
G43
H0
在线出版日期：
2025年4月23日
