Summarization and Annotation of Meetings Wessel Kraaij Martijn - PowerPoint PPT Presentation

Summarization and Annotation of Meetings Wessel Kraaij Martijn Spitters Hap Kolb

Objectives • Choose plausible interaction types of a user with archive generated by M4 • Exploratory effort to define annotation scheme for meetings data (e.g. M4 or parliament ) • Evaluation of annotation tools

User access to a meeting archive • View summary – Summary of a missed meeting • Indexed segments, high level annotation, textual summary? • Browse – Exploratory search, zooming in /out • Navigation structure • Search – Select meeting segments • Indexed segments

Issues • Segmentation of recordings – At which granularity? – Overlapping – Segment clustering (nested?) – Channel based? • Feature detection – Which low level features – Higher level (span multiple segments) • Indexing: use relational DB or XML DB?

* Summarizing different media types *

Goal of summarization • Preserve the “most important information” in a |S| Compression Ratio = document. |D| • Make use of i (S) redundancy in text Retention Ratio = i (D) • Maximize information density i (S) |S| Goal: > i (D) |D|

Summarization architecture • What do human summarizers do? – A: Start from scratch: analyze, transform, synthesize (top-down) – B: Select material and revise: “cut and paste summarization”: bottom-up (Jing & McKeown-1999) • Automatic systems: complexity – Extraction: selection of material – Revision: reduction, combination, syntactic transformation, paraphrasing, generalization, sentence reordering

Document Summarization: Extracts vs Abstracts • Sentence extracts: robust but poor coherence – Determine salience based on sentence position, cue phrases, #content terms, sentence length etc. • Abstracts: – polished extracts – Or use domain dependent generation from templates – Can apply generalisation

Required knowledge lexical level local context dependency discourse global world bag of words bigrams/trigrams structure referential links structure knowledge Ad Hoc IR QA Sentence selection Sentence reduction Sentence combination syntactic transformation lexical paraphrase generalization/specification sentence reordering

Dialogue Summarization (CMU Klaus Zechner) • Speech disfluency removal • Identification and insertion of sentence boundaries Much more difficult for multi person • Identification and linking of question-answer dialogue? regions • Topical segmentation (TextTiling/Hearst) • Information condensation • Genres: Call-me, group meetings (CMU), dialogue oriented television shows (e.g. Crossfire) • Transcript based, no prosody

Video Summarizaton • Video – Sequence of logical story units (scenes) • Sequence of shots – Sequence of frames • Storyboard summary – Presentation of one frame per shot (static shots) – More frames per shot (lots of movement) • Summary frames can be grouped into logical story units, e.g. by TextTiling based on transcripts • Video (or audio) summaries are extracts!

Evaluation • The big problem of evaluating generic summaries is that there is no single gold standard summary. • Possible way-out (DUC2003): use scenarios – Task based summary e.g triggered by a certain question – Viewpoint summary: e.g. financial perspective on a cluster of docs about an earthquake

Meeting Summaries • Goal: short textual summary, in addition some thumbnail images link to key fragments of the meeting – Textual summary: • Sentence extraction is probably not suitable. • Alternative: generation from domain specific templates in combination with topic spotting and information extraction – Storyboard summary: • Navigation points: opening, summary, decision, vote etc • Lively discussion, jokes etc.

Example viewpoints • Result oriented: e.g. decisions, actions assigned, (votes) • Focus on issues that triggered a lot of discussion • Focus on a certain participant • Etc etc.

Generation of an AV viewpoint summary • (Automatically) annotate a recorded meeting: XML file • Apply a viewpoint based XSLT transformation • Generate a SMIL file.

Dummy browse interface

* Annotating meetings *

Function of annotation • Features and higher level elements help to segment data in various granularity levels • Annotated elements can be indexed – Supports focused summaries – Supports retrieval • Hierarchical relations between elements enable browsing

Annotation structure • There’s many levels of annotation possible • Constrained by – What are realistically detectable features – What kind of annotation is necessary and interesting from the application perspective (summary, browsing, searching)

Levels of annotation • Features: directly observed from the visual or audio signal • Speech segments, prosody, laughter, silence movement (nodding, taking notes, pointing ….), shot change • Interpretation layers: are inferred bottom up (e.g by chunking using local or global context) • Low level elements: speech transcripts, utterances, dialogue acts, speaker turns, sentences • High level elements: Meeting structure: topics and agenda management, dynamics, mood, interaction types (e.g. monologue, dialogue, discussion etc) • (Hierarchical) • Relations between different elements and possibly between different levels

Top-down view MEETING STRUCTURE BLOCK STRUCTURE BLOCK STRUCTURE BLOCK STRUCTURE BLOCK HAS_A ID HAS_A TYPE INTERACTION INTERACTION INTERACTION HAS_A TYPE HAS_A ID • Monologue • Dialogue • Supervised (chaired) • Unsupervised

INTERACTION TYPE RECOGNITION TURN HAS_A ID 1. MONOLOGUE SPEAKER HAS_A Chairman HAS_A NAME/ID Speaker Interrupter Audience/others 2. (CHAIRED) DISCUSSION TURN ELEMENT Chairman Speaker 1 TURN ELEMENT Speaker 2 Audience/others HAS_A FUNCTION HAS_A SUBFUNCTION ID HAS_A SUBJECT HAS_A

Annotation standard for M4 • Reuse and extend existing efforts as much as possible e.g. – DAMSL for dialogue act markup – Gesture markup scheme from ANVIL – ICSI MR for speech transcripts? • Separate tracks for different speakers • Global track for features like structure, mood etc.

Annotation tools(1) IBM VideoAnnex: • Shot based (contains shot segmentation module) • MPEG-1 input, MPEG-7 annotation • Annotation scheme can be changed while editing • No relations between elements • Image oriented • No visual display of annotations

Annotation Tools(2) ANVIL (DFKI) • No automatic shot segmentation • Timeline gives visual overview of annotations • Extendible – No on-line editing of annotation DTD – Screen refresh problems (windows 2000) – No complex relations between elements – JMF based: limited number of codecs supported

Example viewpoint summary • ANVIL at work • Query: who made the best joke? – Darren? – Ian? – Daniel? – Steve? – Pierre?