Crossing Media for I m proved I nform ation Access the Reveal This - PowerPoint PPT Presentation

Crossing Media for I m proved I nform ation Access the Reveal This exam ple Stelios Piperidis ILSP spip@ilsp.gr LangTech, 29 February 20 08

• "The vision I have for the Web is about anything being potentially connected w ith anything . It is a vision that provides us with new freedom, and allows us to grow faster than we ever could. . . . it brings the w orkings of society closer to the w orkings of our m inds ." Tim Berners-Lee : Weaving the Web, 2000 • “European citizens should be able to watch or listen to audiovisual content anytim e, anyw here and on all technical platform s (TVset, computer, mobile phone, personal digital assistant, etc.)” European Commission i2010 initiative LangTech, 29 February 20 08 2

Vision Music Drama Educational News / Cinema TeleText The Web / Dig. Libraries Personal Digital Images and Video TV / Satellite TV Video/Text Video/Text/ Radio Images Home/Office PC Music/Voice Music/Voice Video/Text/ Stereo Ima ges/Music Local Pers. Entertainment Repositor System y E - book Car Entertainment Pers. Entertainment Reader System System PDA Laptop LangTech, 29 February 20 08 3

Multim edia Content Analysis Objectives • develop content processing systems that help people keep up with the explosion of digital content scattered over different platforms (radio, TV, World Wide Web), different media (speech, text, image, video) and different languages • develop technology able to semantically index, categorise, summarise and cross-link multiplatform, multimedia and multilingual digital content LangTech, 29 February 20 08 4

Use Scenaria TV, Radio, Web data WEB TV Radio A system that offers both types of service : a) Multimedia and Media Cross lingual Archive Information Multimedia Content technology Retrieval (pull) Aggregator b) Multimedia and Cross lingual information Search archive Delivery Filtering (push) Mobile phone and Web interfaces User Web Mobile User profile Local Archive LangTech, 29 February 20 08 5

Potential Users • end users to gather, filter and categorize information collected from a wide variety of sources in accordance with their preferences. o professionals (media monitoring experts, journalists and editors with demanding media retrieval needs – pull model) o Laymen (novice technology users with information collection/ consumption needs – push model) • content providers to add value to their content, restructure and re-purpose it and offer their clients (subscribers, viewers, etc) individual or corporate users, personalized content LangTech, 29 February 20 08 6

Medium specific m etadata � text: terms/ keywords, named entities (e.g. names of persons, places, organizations), events and topics � speech: speech/ nonspeech, speakers (e.g. speaker identity), transcriptions and stories � video and im ages: keyframes and thematic categories, faces and persons LangTech, 29 February 20 08 7

Cross-m ediality in m ultim edia analysis referring to different sources of information (radio and web text on sam e topic) → across docum ents referring to medium used to convey information within one source (audio, text, image of video segm ent) → within document Source A Source B Single Source TV Broadcast Radio Broadcast TV Broadcast Audio (Speech/Music) News on News on Vidoe/Images Video / I mages Elections Elections Text First Interpretation Second Interpretation LangTech, 29 February 20 08 8

Cross-m ediality in m ultim edia analysis referring to medium used to convey information within one source (audio, text, image of video segm ent) → within document Cross-media indexing • treat imprecisions & inconsistencies • process metadata of speech-image-text Cross-media categorisation • add to m etadata set • process text and images Cross-media summarisation • add to m etadata set • process video and text • present video+ text+ audio salient parts using a content/ domain specific multimedia discourse grammar LangTech, 29 February 20 08 9

Cross-m ediality in m ultim edia analysis referring to different sources of information (radio and web text on sam e topic) → across docum ents Semantic retrieval • retrieval of different m ultimedia documents for a specific query • multidocument summarisation LangTech, 29 February 20 08 1 0

w eb radio tv Media Manager video audio text I AC - video FDI C - and im age face keyfram e TPC - text SPC - audio processing analysis text s processing processing XML m etadata XML m etadata XML m etadata XML m etadata speaker turns, shotcuts, keyfram es, nam ed entities, faces & ids speakers, text im age features term s, events XML Merging Segm ent Unification Story Boundary Detection cross- m edia stories story-based text/ im age/ video analysis categorisation, sum m arisation, translation Sm art Content Media Server LangTech, 29 February 20 08 1 1

Exam ples of m ultim edia analysis m odules in a nutshell Speech Speech Image Face recognition Recognition Categorisat detection & in EN in EL ion identificatio n Fact Fact Cross- Cross- extraction extraction media media in EN in EL indexer Categorisat ion Text Textual Scenes and Cross- summarisa summarisa visual media tion in EN tion in EL summaries summaries Query Cross- Retrieving www. Translation lingual stories reveal-this. document org translation LangTech, 29 February 20 08 1 2

Cross-m edia sum m arisation architecture Cross-media Summarization Subsystem Analysis Analysis SCENE Grouping TEXTUAL-BASED Personalization ENGINE Mechanisms CLUSTERING Users Profiles Summarization Technologies Summary Enrichment Cross-lingual Translation Subsystem summary REVEAL Translation summary Engine Interfaces Summarization Interfaces LangTech, 29 February 20 08 1 3

Different dom ains: different m odels Anchor Reportage Interview Reportage Interview Reportage History Lifestyle Landscape History Fight against Arms … …. …. Human Rights Terrorism Embargo LangTech, 29 February 20 08 1 4

Audio Segmentor P2 P1 P3 P4 P5 P6 P7 P8 A A R R I R I R Scene Labeller Anchor Reportage Interview Visual Textual Textual Summariser Summariser Summariser Presentation Layer TV News HTML+TIME LangTech, 29 February 20 08 1 5

Euro-Parliam ent Sessions : m edia analysis EbS PLENARY 2005/04/27 15:00 Fight against Arms … …. …. Human Rights Terrorism Embargo LangTech, 29 February 20 08 1 6

Euro- Parliam ent Sessions structure & content T 1 T 1 T 1 T 2 T 2 T 1 T 3 T 4 T 5 T 5 Session S 1 S 2 S 3 S 1 S 4 S 4 S 1 S 5 S 1 S 2 Topic 1 Topic 2 Topic 3 Topic 4 Topic 5 Regions S1 S2 S1 S4 S1 S5 S1 S2 Speakers S3 S4 LangTech, 29 February 20 08 1 7

Term Extractor T 1 T 1 T 1 T 2 T 2 T 1 T 3 T 4 T 5 T 5 S 1 S 2 S 3 S 1 S 4 S 4 S 1 S 5 S 1 S 2 Speaker Identifier Topic 1 Topic 2 Topic 3 Topic 4 Topic 5 Textual Summariser S1 S2 S1 S4 S1 S5 S1 S2 S3 S4 Presentation Layer EbS Sessions HTML+TIME LangTech, 29 February 20 08 1 8

Travel docum entaries: m edia analysis BestOfGreece-DG-EN - Chapter: Athens History Lifestyle Landscape History LangTech, 29 February 20 08 1 9

Travel docum entaries bits and bolts Story P 2 P 1 P 3 P 4 P 5 P 6 P 7 C 1 C 2 C 1 C 2 C 2 C 3 C 1 C 1 Chapter Thematic C1 C2 C3 C1 Categories C1 C2 C3 Regions LangTech, 29 February 20 08 2 0

Audio Segmentor P 1 P 2 P 3 P 4 P 5 P 6 P 7 C 1 C 1 C 2 C 2 C 2 C 3 C 1 C 1 Image Clustering Categoriser Textual Travel Summariser Vocabulary Land History Lifestyle scape Presentation Layer Travel Documentaries HTML+TIME LangTech, 29 February 20 08 2 1

Crossing m edia in the future • Elaborate crossing media techniques for multimedia authoring and presentation • Cross-media based indexing and retrieval of multimedia content • Cross-media analysis for better understanding of communicated messages • Cross-media methods in robotic and cognitive systems • Cross-media techniques for better simulation of knowledge and/ or language acquisition processes LangTech, 29 February 20 08 2 2