Personalized Image Aesthetics Assessment Leida Li School of Artificial Intelligence Xidian University 2020.05.27 Collaborators: Hancheng Zhu Jinjian Wu Sicheng Zhao Guiguang Ding Guangming Shi Weisi Lin CUMT XDU UC Berkeley Tsinghua XDU NTU
Con onte tent nts • Introduction • Related Work • Personality-assisted Multi-task Learning for Personalized Image Aesthetics Assessment • Personalized Image Aesthetics Assessment via Meta- learning • Conclusion 2020/5/28 2
Con onte tent nts • Introduction • Related Work • Personality-assisted Multi-task Learning for Personalized Image Aesthetics Assessment • Personalized Image Aesthetics Assessment via Meta- learning • Conclusion 2020/5/28 3
Introducti roduction on to Aes esthetic hetic Qua uality ity Chin ina has a prover verb The love e of be beauty y is common on to all peopl ple on: how to to jud udge aes esthetics? etics? A n natural ural questi stion: 2020/5/28 4
Introducti roduction on to Aes esthetic hetic Qua uality ity • Photog otogra raphy phy rules les Rule of Thirds Symmetry Depth of Field Color Harmony K. Michal, et al., Leveraging expert feature knowledge for predicting image aesthetics. IEEE Trans. Image Process., 2018. 2020/5/28 5
Introducti roduction on to Aes esthetic hetic Qua uality ity • Photog otogra raphy phy rules les https://baijiahao.baidu.com/s?id=1608952443864950129&wfr=spider&for=pc 2020/5/28 6
Introducti roduction on to Aes esthetic hetic Qua uality ity • Ap Appl plicatio cation n scen enari rios os - Advertisement Alibaba’s Luban system • Launched on 11/11/2016 • Designed 170 million posters • Improved hit rate by 100% • Equipped with IAA engine http://www.mgzxzs.com/tmtbzxjc/2288.html 2020/5/28 7
Introducti roduction on to Aes esthetic hetic Qua uality ity • Ap Appl plicatio cation n scen enari rios os - Cover image selection 2020/5/28 8
Introducti roduction on to Aes esthetic hetic Qua uality ity • Ap Appl plicatio cation n scen enari rios os - Photo auto-cropping 2020/5/28 9
Con onte tent nts • Introduction • Related Work • Personality-assisted Multi-task Learning for Personalized Image Aesthetics Assessment • Personalized Image Aesthetics Assessment via Meta- learning • Conclusion 2020/5/28 10
Generi ric c Aesthetic tic Quality ity • Generic image aesthetics assessment (GIAA) • Aesthetics regression • Aesthetics classification Score = 4.0 Score = 3.8 Score = 2.4 • Aesthetics distribution -H. Zeng, et al., A unified probabilistic formulation of image aesthetic assessment. IEEE Trans. Image Process., 2020 -Y. Deng, et al., Image aesthetic assessment an experimental survey. IEEE Signal Process. Mag., 2017. 2020/5/28 11
Related Work • Conventional approaches with handcrafted features • Simple image features - Colorfulness - Contrast - Brightness • Image composition features - Low depth of field High colorfulness Low colorfulness - Salient object - Rule of thirds • General-purpose features - SIFT descriptors - Bag of visual words (BOV) - Fisher vector (FV) Good composition Bad composition Extracted handcrafted features for image aesthetics assessment -R. Datta, et al., Studying aesthetics in photographic images using a computational approach. ECCV 2006. -X. Tang, et al., Content-based photo quality assessment. IEEE Trans. Multimedia, 2013. -N. Murray, et al., AVA: A large-scale database for aesthetic visual analysis. CVPR 2012. 2020/5/28 12
Rel elated ed Work • Deep-learning approaches • Ranking deep network Aesthetics attribute Network architecture • Multi-task deep network Network architecture Significant progress has been achieved in GIAA. -S. Kong, et al., Photo aesthetics ranking network with attributes and content adaptation. ECCV 2016. -Y. Kao, et al., Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Process., 2017. 2020/5/28 13
Per ersona nalized ized Aes esthetic hetic Qua uality ity China na has as another other proverb overb One man's 's meat is is anoth ther er man's 's poison on 2020/5/28 14
Per ersona nalized ized Aes esthetic hetic Qua uality ity • Personalized image aesthetics assessment (PIAA) - People have different tastes on image aesthetics, depending on their subjective preferences . Score = 4.0 Score = 3.8 Score = 2.4 Ratings = (1,2,2,3,4) Ratings = (3,3,4,5,5) Ratings = (2,3,4,5,5) 2020/5/28 15
Rel elated ed Work • Adapting from generic aesthetics Personalized image aesthetics model with a residual-based model adaptation scheme. J. Ren, et al., Personalized image aesthetics. ICCV 2017. 2020/5/28 16
Rel elated ed Work • User interaction User-friendly aesthetic ranking framework via deep neural network and a small amount of interaction. P. Lv, et al., USAR: an interactive user-specific aesthetic ranking framework for images. ACM MM 2018. 2020/5/28 17
Rel elated ed Work Challenges ： 1. Existing works leverage objective visual features (e.g., contents and attributes) for modeling users’ subjective aesthetic preferences. This may be insufficient, because the subjective factors (e.g., personality traits) in rating image aesthetics are not fully investigated. 2. The generic model learned from average aesthetics cannot accurately capture the shared aesthetic prior knowledge when people gauge image aesthetics, since it simply uses the average score as the training target, which counteracts the differences of individual aesthetic perception. 2020/5/28 18
Con onte tent nts • Introduction • Related Work • Personality-assisted Multi-task Learning for Personalized Image Aesthetics Assessment • Personalized Image Aesthetics Assessment via Meta- learning • Conclusion 2020/5/28 19
PA PA_IAA IAA Personality-assisted Multi-task Learning for Generic and Personalized Image Aesthetics Assessment (L. Li, H. Zhu, et al., IEEE TIP, 2020) • As an important subjective trait , personality trait is believed as a key factor in modeling humans’ subjective preferences. • What is the relationship between aesthetics assessment and personality prediction from images? Images liked by users with high extraversion Images liked by users with low extraversion -H. Zhu, L. Li, et al., Evaluating attributed personality traits from scene perception probability, Pattern Recogn Lett, 2018. -S. C. Guntuku , et al., Who likes what, and why? insights into personality modeling based on image “likes“, IEEE Trans Affect Comput, 2018. 2020/5/28 20
PA PA_IAA IAA • Big-Five personality traits - Openness: tendency to be open, curious, etc. - Conscientiousness: tendency to be responsible and reliable. - Extraversion: tendency to interact and spend time with others. - Agreeableness: tendency to be kind, generous, etc. - Neuroticism: tendency to be anxious, sensitive, etc. • Multi-task learning An effective way in capturing useful information contained in multiple related tasks , which can be used to improve the generalization performance of all tasks. Task 1 Aesthetics scores . . . . . . Personality traits Task 2 Convolution Fully Connected Related tasks -B. Rammstedt, et al., Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German, J. Res. Pers. 2007. -S. Ruder, CoRR, 2017. Online: http://arxiv.org/abs/1706.05098 2020/5/28 21
PA PA_IAA IAA • A multi-task learning network with shared weights is proposed to predict the aesthetics distribution of an image and Big-Five (BF) personality traits of people who like the image • To capture the common representation of image aesthetics and people’s personality traits, a Siamese network is trained using aesthetics data and personality data jointly . • Inter-task fusion is introduced to generate individual’s personalized aesthetic scores. 2020/5/28 22
Ƹ Ƹ Ƹ Ƹ PA PA_IAA IAA 𝑗 ; 𝒕 𝑏 𝑗 } 𝑗=1 𝑗 = {𝑡 𝑏 𝑜 𝑗 } 𝑜=1 𝑂 𝑏 , where 𝒕 𝑏 𝑂 Generic aesthetics training samples: {𝐽 𝑏 ; Aesthetic deep features: 𝒆 𝑏 ⚫ - Estimated aesthetic distribution: 𝑈 𝒆𝑏 𝑓 𝑿𝒃𝒐 𝑗 𝑡 𝑏 𝑜 = 𝑈 𝒆𝑏 𝑿𝒃𝒌 𝑂 σ 𝑘=1 𝑓 - Generic aesthetics loss function: 1 1 2 𝑀 𝑏 = 𝑂 𝑏 σ 𝑜=1 𝑂 𝑗 𝑗 𝑂 σ 𝑗=1 𝑡 𝑏𝑜 −𝑡 𝑏𝑜 𝑂 𝑏 2 5 𝑣𝑛 } 𝑛=1 𝑁 𝑣𝑗 } 𝑗=1 𝑉 ⚫ Personality training samples: {{𝐽 𝑞 , {𝑇 𝑞 } 𝑣=1 , 𝑉 is the number of users, 𝑁 is the number of images liked by a user. Personality deep features: 𝒆 𝑞 - Predicted personality distribution: 𝑈𝒆 𝑞 −𝑓 −𝑿 𝑞 𝑈𝒆 𝑞 𝑣𝑛 = 𝑓 𝑿 𝑞 𝑻 𝑞 𝑈𝒆𝑞 +𝑓 −𝑿𝑞 𝑈𝒆𝑞 𝑓 𝑿𝑞 - Personality loss function: 𝑀 𝑞 = 1 1 1 2 5 𝑉 𝑁 𝑣𝑛𝑗 −𝑇 𝑞 𝑣𝑗 መ 𝑁 σ 𝑣=1 σ 𝑛=1 σ 𝑗=1 𝑇 𝑞 5 𝑉 2 𝑗 } 𝑗=1 𝑂 𝑐 ; 𝑗 ; 𝑡 𝑐 Personalized aesthetics training samples: {𝐽 𝑐 ⚫ - Estimated personalized aesthetic score: 𝑗 = Ƹ 𝑗 + 𝑿 𝑐 ො 𝑗 𝑡 𝑐 𝑡 𝑏 𝒕 𝑞 - Personalized aesthetic loss function: 2 1 𝑂𝑐 𝑗 −𝑡 𝑐 𝑀 𝑐 = 𝑗 𝑂 𝑐 σ 𝑗=1 𝑡 𝑐 2 2020/5/28 23
Recommend
More recommend
