Aut utom omat atic ic Cor orrecti ection on of of Adv dver - PowerPoint PPT Presentation

Aut utom omat atic ic Cor orrecti ection on of of Adv dver erb Pl Plac acem emen ent t Er Error ors s for or CAL ALL Marie Garnier CAS/IRIT, Université de Toulouse, France EuroCALL 2012, 22-25 August, Gothenburg

Background 1.  Focus on French speakers  Aspects of the project Predicting adverb placement 2.  Grammaticality judgment tests Implementation 3.  Evaluation: methods and results Limits and strengths  Corrective feedback 4.  Design and implementation 2

1. . Ba Back ckground ound Osborne (2008), "Adverb placement in post-intermediate learner English: a contrastive study of learner corpora"  Persistence of adverb placement errors at an intermediate to advanced level  Adverbs in V-Adv-O order are markedly more frequent, mostly with learners with L1 French , Italian and Spanish  Survey of 11 common grammar checkers: adverb placement errors not detected  Tricky grammatical issue 3

Focus on French speakers  French/English: partial overlap between canonical positions for adverbs as modifiers/adjuncts  Possible positions for adverb lentement (Fr.) / slowly (Eng.) :  , elle  ouvrit  la porte  .  , she  opened  the door  .  V-Adv-O accepted/necessary in some cases in English (collocations, weight of object NP, syntax of AdvP): I understood very clearly the meaning of the speech you gave yesterday.  May result in negative transfer (Odlin, 1989) and overgeneralisation (Osborne, 2008) 4

Aspects of the project  Objective: design automatic detection and correction strategies for adverb placement errors in the written productions of intermediate to advanced users of English with L1 French Manner adverbs as Modelling of adverb modifiers in the VP or placement adjuncts in the clause. Patterns and rewriting Generation of tailored rules programmed in corrective feedback Prolog , using <TextCoop> messages 6

2. Pred edicting icting correct rect adverb erb pl placement cement  Detecting errors and predicting correct placement means understanding adverb placement in English  Lack of definitive rules: Only rather broad and approximate flexible generalisations about adverb placement and sequence can be made. There is a great deal of variation in use, and features of context, style, prosody, and euphony play a role in some decisions. Huddleston and Pullum (2002). CGEL , p. 576  Multifactorial syntactic phenomenon (Garnier, 2012):  semantic type of adverb and scope (VP/clause)  weight and composition of the AdvP (long/short adverb, very/too/more )  sentence structure (transitive/intransitive VP, information packaging choices)  prosody (prosodically integrated/detached adverbs) 7

Grammaticality judgment tests (1)  Indicative study: 3 native English speakers (2 AmEng, 1 BrEng)  Objective: assess the influence of specific syntactic parameters on the placement of manner adverbs  Parameters:  Weight and structure of the AdvP (long/short adverb, modification by very , too , more , etc.)  Presence and weight of verb complements ( direct and indirect objects)  Verbs followed by prep. phrases  Presence of auxiliaries 8

Grammaticality judgment tests (2)  56 sentences organized in 13 sets of 3 to 5 sentences representing all possible positions for an adverb in a test sentence  NS asked to identify all correct placements + choose the most natural-sounding one  Results : "Correct": complete agreement reached in only 36% of cases "Best": complete agreement reached in 69% of cases 9

3. Im 3. Implem emen enta tati tion on  Creation of detection patterns and rewriting rules  22 detection patterns associated with 1 to 3 rewriting rules System can issue several propositions when more than one correction is possible  Resources:  Lexicons : adverbs, aux., verbs, prep., adjectives, nouns, determ.  Grammars : Noun Phrase, Adjective Phrase 10

Error She must remove carefully the bandage. Detection pattern NP1 (Aux) (Aux) Vlex (Prep) ADV NP2 Rewriting rules 1. NP1 (Aux) (Aux) ADV Vlex (Prep) NP2 2. NP1 (Aux) (Aux) Vlex (Prep) NP2 ADV Corrected sentence 1. She must carefully remove the bandage. 2. She must remove the bandage carefully. Table 1. Schematized example of a pattern/rewriting rule 11

forme (corr-adv a , [plus(aux( AUX ,_)), verb( V ,_), opt(prep( P1 ,_)), adv( ADV ,manner), opt(prep( P2 ,_)), np( NP ,_)], [conc(VE1,E2,E1), conc(NP11,S,E5), card(NP11, T), (T < 5)], ['<erreur 3a >', AUX , VE1 , P1 , ADV , P2 , NP11 , '</erreur>',' <correct>', AUX , ADV , VE1 , P1 , P2 , NP11 , '</correct>']). forme (corr-adv b , [plus(aux( AUX ,_)), verb( V ,_), opt(prep( P1 ,_)), adv( ADV ,manner), opt(prep( P2 ,_)), np( NP ,_)], [conc(VE1,E2,E1), conc(NP11,S,E5), card(NP11, T), (T < 5)], ['<erreur 3b >', AUX , VE1 , P1 , ADV , P2 , NP11 , '</erreur>',' <correct>', AUX , VE1 , P1 , P2 , NP11 , ADV ,'</correct>']). Table 2. Example of a pattern/rewriting rule in <TextCoop>  Patterns and rules implemented within <TextCoop>  <TextCoop>: discourse analysis platform programmed in Prolog (Saint-Dizier, 2011)  Logic-based programming means model can be implemented directly 12

Evaluation: Method (1)  Evaluation on an 80,000-word corpus of native English (British and American online newspapers, British and American blog posts, scientific publications)  279 occurrences of manner adverbs  False positives = < 4%  Main cause: words belonging to several grammatical categories (ex. check n./v., use n./v.) 14

Evaluation: Method (2)  Evaluation on a modified 20,000-word corpus of learner and user English (learner productions from ICLE and personal emails)  The availability of appropriate learner/user corpora for testing is one of the challenges of research on automatic grammar- checking (Foster and Andersen, 2009) : A proficient English user with L1 French was asked to introduce manner adverbs in authentic texts, producing correct and incorrect sentences 15

Evaluation: Results 1. Detection 2. Correction precision = 96% precision = 87% recall = 83% recall = 80%  2 nd proposition: 70% precision  Simplest patterns (i.e. adverb without modification in finite sentences with NPs of less than 5 words) are used in 83% of all cases  Among them, V-Adv-O patterns used in 48% of all cases  Patterns are mutually exclusive (no conflicts) 16

Limits of the system  Number of syntactic configurations that can be recognized: e.g. embedded that -clauses, ditransitive VPs  Words belonging to more than one category: e.g. check , use , purchase , being , etc.  Adverb with more than one semantic type: e.g. He looked at her sadly . (= manner) Sadly, they were arguing about money again . (= act-related)  Requires mostly grammatical input, but :  Intermediate users/learners: overall sentence structure is usually mastered  Patterns use categories, which allows for a margin of error in the input text 17

Strengths of the system  Directly portable to other romance languages (Italian, Spanish)  Architecture is adaptable to any kind of L1 , with specific alterations  Creation of new patterns and new feedback messages is easy and can be done by linguists or teachers with little to no programming skills  Design of specific feedback for each error type is simplified 18

4. 4. Cor orrec ecti tive e fee eedb dbac ack Heift (2004), Heift and Schulze (2007): metalinguistic feedback combined with highlighting has positive effects on learning Garnier (2011): review of feedback messages included in commercial/research grammar checkers; found to be often inconsistent and not adaptable to users  Five key steps: 1. Error marking 2. Error diagnosis 3. Metalinguistic feedback 4. Remediation 5. Illustration 19

We have tested separately all the parameters 1. Error marking In this sentence, the manner adverb 2. Error diagnosis [ separately ] is placed between the verb [ tested ] and the object [ all the parameters ]. In English, manner adverbs are not generally 3. Metalinguistic FB found between a verb and a following object. There are 2 ways to correct this sentence: 4. Remediation -By placing the adverb after the object; -By placing the adverb between the subject and the verb. Examples: 5. Illustration - She unwrapped the present carefully - She carefully unwrapped the present. 20

 Can be adapted to the profile of the user:  Learner : looking for help with her/his learning  steps 1, 2, 3, 4, 5 without the correction , in order to elicit self-correction  Curious : looking for more information on the error or the construction in general  steps 1, 2, 3, 4, 5  Careful : wants to check why the segment was flagged as an error  steps 1, 2, 5  Rushed : only wants to see the error and its correction(s)  step 1 21

Implementation of feedback messages  Profile is selected in advance by the user of the system  Feedback is displayed on an html page when user clicks on the error segment  Specific messages for each pattern + include the actual adverb, NP or verb used in the text 22