Teamware: A Collaborative, Teamware: A Collaborative, Web-based - PowerPoint PPT Presentation

Teamware: A Collaborative, Teamware: A Collaborative, Web-based Annotation Environment Web-based Annotation Environment Kalina Bontcheva, Milan Agatonovic University of Sheffield

University of Sheffield NLP Outline Outline • Why Teamware? • What’s Teamware? • Teamware for annotation • Teamware for quality assurance and curation • Teamware for defining workflows, running automatic services, managing annotation projects • Outlook GATE Summer School - July 27-31, 2009 2

University of Sheffield NLP From Annotation Tools to From Annotation Tools to Collaborative Annotation Workflows Collaborative Annotation Workflows We have lots and lots of tools and algorithms for annotation; what we need is 1. methodological instead of purely technological 2. multi-role instead of single role 3. assistive instead of autonomous 4. service-orientated, not monolithic 5. usable by non-specialists GATE Teamware � Research users in several EU projects � External users at IRF and Matrixware � Interest from other commercial users as well GATE Summer School - July 27-31, 2009 3

University of Sheffield NLP GATE Teamware: Annotation GATE Teamware: Annotation Workflows on the Web Workflows on the Web GATE Teamware is: □ Collaborative, social, Web 2.0, has behaviour mining using Machine Learning □ Parallel and distributed (using web services) □ Scalable (via service replication) □ Workflow based with business process integration via BPEL GATE Summer School - July 27-31, 2009 4

University of Sheffield NLP Teamware – Teamware – Layer Cake Layer Cake Language Data Curation Manual Annotation Engineer User Interface User Interface User Interface User Interface Document Browser Schema Ontology Layer Annotation Annotation GATE Annotation ANNIC UI UI Developer UI Diff UI UI Teamware Authentication Workflow Executive And User Management Layer Management GATE GATE Annotation Machine Learning GATE GATE Services Services API Document Ontology Layer Service Service GATE Summer School - July 27-31, 2009 5

University of Sheffield NLP Division of Labour: Division of Labour: A Multi-role A Multi-role Methodology Methodology • (Human) Annotators - labour has to be cheap! � Bootstrap annotation process with JAPE rules or mixed-initiative learning • Curators (or super-annotators) � Reconcile differences between annotators, using IAA, AnnDiff, curator UI � Manager � Defining annotation guidelines and schemas � Choose relevant automatic services to pre-process � Toolset including performance benchmarking , progress monitoring tools, small linguistic customisations � Define workflow , manage annotators , liaise with language engineers and sys admins • Sys admin � Setup the Teamware system, users, etc. � Language engineer � Uses GATE Developer to create bespoke services and deploy online GATE Summer School - July 27-31, 2009 6

University of Sheffield NLP Teamware: Teamware: Manual Annotation Tool Manual Annotation Tool GATE Summer School - July 27-31, 2009 7

University of Sheffield NLP Manual Annotation Process Manual Annotation Process • Annotator logs into Teamware • Clicks on “Open Annotation Editor” • Requests an annotation task (first button) • Annotates the assigned document • When done, presses the “Finish task” button • If wants to save work and return to this task later – “Save” button, then close the UI. Next time a task is requested, the same document will be assigned, so it can be finished • Depending on the project setup, it might be possible to reject a document and then ask for another one to annotate (Reject button) GATE Summer School - July 27-31, 2009 8

University of Sheffield NLP Hands-on Hands-on • Open a web browser and Teamware • Login using you user name (from your reg.pack): � <cics-account-id>-annotator • Open the annotation UI • Try requesting tasks, editing annotations, saving your work, asking for another task, etc. • This is what Teamware looks like to a human annotator GATE Summer School - July 27-31, 2009 9

University of Sheffield NLP Teamware for Curators Teamware for Curators • Still being developed, so UI is in transition • Identify if there are differences between annotators using IAA • Inspect differences in detail using AnnDiff • Edit and reconcile differences if required � New curator UI in Teamware under development � Currently available in Developer GATE Summer School - July 27-31, 2009 10

University of Sheffield NLP IAA: Do my annotators agree? IAA: Do my annotators agree? GATE Summer School - July 27-31, 2009 11

University of Sheffield NLP IAA: Results IAA: Results GATE Summer School - July 27-31, 2009 12

University of Sheffield NLP IAA: Recap IAA: Recap • The IAA on IE tasks, such as named entity recognition, should be measured using f-measure across all annotators • For classification tasks, use Kappa to measure IAA • For details, see the evaluation lecture and the GATE user guide GATE Summer School - July 27-31, 2009 13

University of Sheffield NLP AnnDiff: Finding the differences AnnDiff: Finding the differences GATE Summer School - July 27-31, 2009 14

University of Sheffield NLP Where are these in Teamware? Where are these in Teamware? • Only visible to curators and their managers • Resources/Documents menu • Select the corpus worked on • Iterate through each document • Run IAA and AnnDiff, as required • Try for yourself: � Login as <cics-user-name>-curator � Corpus: annie-demo � The first or second documents GATE Summer School - July 27-31, 2009 15

University of Sheffield NLP Forthcoming curator facilities Forthcoming curator facilities • Have a corpus-level view of IAA • Extended AnnDiff to allow easy reconciliation of the differences between 2 annotators • Currently prototyped in Developer • Will be made available in Teamware soon GATE Summer School - July 27-31, 2009 16

University of Sheffield NLP New AnnDiff in Developer New AnnDiff in Developer GATE Summer School - July 27-31, 2009 17

University of Sheffield NLP Beyond Pair-wise Reconciliation Beyond Pair-wise Reconciliation • AnnDiff only handles 2 sets of annotations at a time – we often need more! • Towards an in-place, content-based reconciliation interface GATE Summer School - July 27-31, 2009 18

University of Sheffield NLP Current UI Prototype Current UI Prototype GATE Summer School - July 27-31, 2009 19

University of Sheffield NLP Teamware for Managers Teamware for Managers • Defining workflows • Running annotation projects • Tracking progress GATE Summer School - July 27-31, 2009 20

University of Sheffield NLP Teamware Workflows Teamware Workflows • Whole process is controlled by a workflow manager • Workflow may be simple: � Give the document to a human annotator � Information curator checks a sample of documents for QC • or more complex � Invoke one or more web services to produce automatic annotations � Pass each document to 2 annotators � Information curator to quickly check level of agreement between the annotators and reconcile any differences � Annotated documents used to train an ML model � When model is good enough, start making suggestions to the annotators GATE Summer School - July 27-31, 2009 21

University of Sheffield NLP Workflow Templates Workflow Templates GATE Summer School - July 27-31, 2009 22

University of Sheffield NLP Defining new workflows Defining new workflows • Select Projects/WF Templates • Opens the WF wizard • Choose which services you want to run • Choose whether you want manual annotation, how many annotators per doc, … GATE Summer School - July 27-31, 2009 23

University of Sheffield NLP Setting up a Manual Annotation Setting up a Manual Annotation Project Project • Upload the schemas • Upload the documents • Define the Workflow template • Run the project, choosing the corpus, the annotators, curators, etc. • DEMO! GATE Summer School - July 27-31, 2009 24

University of Sheffield NLP Setting up an Automatic Setting up an Automatic Annotation Project Annotation Project • Configure the web service(s) • Define the Workflow template • Run the project, choosing the corpus • DEMO! GATE Summer School - July 27-31, 2009 25

University of Sheffield NLP Semi-automatic Projects Semi-automatic Projects • Just combine the two sets of steps GATE Summer School - July 27-31, 2009 26

University of Sheffield NLP Teamware: Teamware: Monitoring Project Progress Monitoring Project Progress GATE Summer School - July 27-31, 2009 27

University of Sheffield NLP Outlook Outlook • Teamware is still under active development • Many features subject to change • If you’d like further information or to try it with your data for a particular project, please contact Hamish and Kalina GATE Summer School - July 27-31, 2009 28