META-FORUM 2016 04-July-2016 LiMe Motivation Knowledge in the EU - PowerPoint PPT Presentation

Ronald Denaux rdenaux@expertsystem.com META-FORUM 2016 04-July-2016

LiMe Motivation Knowledge in the EU is fragmented… So far, information can only be TV, social analysed videos video, visual photos, social independently for each images photos modalities dimension. audio This restricts the extractable audio from auditiv from TV social knowledge media and keeps it fragmented. news, tweets, textual annotations blogs, of comments, audio/video reviews mainstream/ social/ user generated professionally produced channels 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 2

LiMe Approach TV, Social Video Clips, Video , visual Photos, Social Images Photos Podcasts, Audio Audio auditiv from from TV Social Video News, Twitter, Multi- Cross- Multi-lingual Multi- Descriptions Multi-lingual Blogs, Natural lingual textual lingual text lingual of video or text Comment Languag Annotati text audio text anno annotations annotations e on mainstream/ social/ commercial/ user professional generated 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 3

LiMe Approach 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 4

LiMe Roles of Partners Research & Development of general functionality Iberia Integration Data & Use Cases & Product Specific Development 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 5

LiMe Approach 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 6

LiMe Use Case: Zattoo „mini embed“ For a given online article, find related TV programs from what’s currently airing 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 7

Use Case: Econda product LiMe recommendation For a given product catalog, recommend products based on recent mentions in supported media. Mappings of domain entities • Products/categories/brands to DBpedia • Identify mapped entities in messages Text/ASR annotations using domain entities • Include domain entities in annotation knowledge base • Use names and descriptions as surface forms Video annotations using domain entities • Visual index of all products in shop • Detect similar products in video frames 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 8

LiMe Use Case: VICO Brand Monitoring 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 9

LiMe Start: November 2013; End: October 2016 • Project Coordinator: Achim Rettinger rettinger@kit.edu • Project Manager: valentina.pavlova@kit.edu • Web: http://xLiMe.eu • 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 10

LiMe Y1 Benchmarks Entity Linking for Social Media : 220% CPU, 27% main memory, ~150,000 microposts per day, 12 • languages supported, throughput ~2.1kb/s Speech to Text : single 40s audio chunk processed in 2min 20s, throughput ~ 140kb/s • Text from Video : 6 frames per second, 24GB memory, throughput ~ 384 kb/s, accuracy 30% • Visual Object Type Recognition : 6GB GPU, NVIDIA Tesla K40c GPU, throughput ~ 2000 kb/s, • accuracy 99.5% Named Entity from Text : tokenization (48% one core CPU, 3.7GB memory), KIT wikifier (10-40 • cores used of 64), throughput ~ 112kb/s Syntactic annotations for News Articles : <10% CPU, 5.5GB, throughput ~ 112kb/s • Cross-media Recommendations : social media (1.01 million posts), TV programs (350), news • articles (40k), throughput ~ 232kb/s Named Entity Microposts Video Annotation 2,1 2000 Text from Video 384 Speech to Text Throughput 140 (kb/s)* 112 Named Entity Newsfeed 112 Sentiment 232 Annotations Cross-media Recommendations 1 10 100 1000 10000 04-July-16 Ronald Denaux (Expert System Iberia) – META-FORUM-16 11