Automatic Text Analysis Tools Facilitating Text Selection in Reading Assessment Aylin Ünaldı FOAI 11
• Thank you FOAI for bringing together the assessment professionals for more than 10 years • Mehtap İnce , Esin Çağlayan , Berna Akpınar Aslan • Medipol University • Language Assessment Issues in Turkey A sister email/discussion list: lait@list.boun.edu.tr 2
Common Issues • Contextual validity in reading assessment • Concept of textual complexity • How automatic tools are used in determining textual complexity • How the information from automatic tools can guide us 3
TEST SPECIFICATIONS – cognitive operations that tasks are expected to trigger • e.g. reading across sentences to elicit implied meaning – task characteristics • e.g. short-answer – response method • written response: two-three words at most – input characteristics/textual features 4
Context Validity (Weir, 2005) What are the contextual features we need to take into consideration when we judge the suitability of texts for assessment purposes? How do we set the levels of complexity in texts targeted at different proficiency levels? 5
Complexity Taxonomy (Bulte & Housen, 2012) «… complexity refers to a property or quality of a phenomenon or entity in terms of (1) the number and the nature of the discrete components that the entity consists of, and (2) the number and the nature of the relationships between the constituent 6 components.»
‘Something wicked comes this way’ ‘ 7
Inventory of linguistic complexity measures (Bulte & Housen, 2012) • A. GRAMMATICAL COMPLEXITY Number of di erent verb forms • • a. Syntactic iv. Subsentential (Clausal + Variety of past tense forms Phrasal) • i. Overall • 18. Mean length of clause • • Mean length of T-unit ii. Derivational • 19. S-nodes/clause • • Mean length of c-unit Measure of affixation • v. Clausal • Mean length of turn • Syntactic arguments / clause • • Mean length of AS-unit B. LEXICAL COMPLEXITY • vi. Phrasal • • Mean length of utterance a. Diversity • Dependents / (noun, verb) • • S-nodes / T-unit Number of word types phrase • • S-nodes / AS-unit TTR • vii. Other ( ± syntactic • • ii. Sentential – Coordination Mean segmental TTR sophistication) • • Coordinated clauses / clauses Guiraud Index • 20. Frequency of passive forms • • iii. Sentential – Subordination (Word types)2 / words 21. Frequency of infinitival – • Clauses / AS-unit phrases D – Clauses / c-unit • • 22.Frequency of conjoined b. Density – Clauses/T-unit clauses • Lexical words / Function words – Dependent clauses/clause • 23. Frequency of Wh-clauses • Lexical words / Total words – Number of Subordinate 24. Frequency of imperatives • c. Sophistication clauses 25. Frequency of auxiliaries • – Subordinate clauses/clauses 40. Less frequent words / Total 26. Frequency of comparatives – words Subordinate clauses / 27. Frequency of conditionals dependent clauses • b. Morphological – Subordinate clauses/T-unit • i. Infectional – Relative clauses/T-unit • Frequency of tensed forms – Verb phrases/T-unit • Frequency of modals 8
Syntactic complexity measures (Lu, 2010) • Length of production 1. Mean length of clause (MLC) 2. Mean length of sentence (MLS) 3. Mean length of T-unit (MLT) • Sentence complexity 4. Mean number of clauses per sentence (C/S) • Subordination 5. Mean number of clauses per T-unit (C/T) 6. Mean number of complex T-units per T-unit (CT/T) 7. Mean number of dependent clauses per clause (DC/C) 8. Mean number of dependent clauses per T-unit (DC/T) • Coordination 9. Mean number of coordinate phrases per clause (CP/C) 10. Mean number of coordinate phrases per T-unit (CP/T) 11. Mean number of T-units per sentence (T/S) • Particular grammatical structures 12. Mean number of complex nominals per clause (CN/C) 13. Mean number of complex nominals per T-unit (CN/T) 14. Mean number of verb phrases per T-unit (VP/T) 9
Cohmetrix indices of the two texts: McNamara, Graesser, McCarthy and Cai (2004) http://tool.cohmetrix.com/ • Number Label Label V2.x Text Text2 Full description • 1 DESPC READNP 1 1 Paragraph count, number of paragraphs • 2 DESSC READNS 1 11 8 Sentence count, number of sentences • 3 DESWC READNW 191 206 Word count, number of words • 4 DESPL READAPL 11 8 Paragraph length, number of sentences in a paragraph, mean • 5 descriptives DESPLd 0 0 Paragraph length, number of sentences in a paragraph, standard deviation • 6 DESSL READASL 17.364 25.75 Sentence length, number of words, mean • 7 DESSLd n/a 9.470 8.246 Sentence length, number of words, standard deviation • 8 DESWLsyREADASW 1.571 1.718 Word length, number of syllables, mean • 9 DESWLsyd 0.810 1.040 Word length, number of syllables, standard deviation • 10 DESWLlt 4.780 5.505 Word length, number of letters, mean • 11 DESWLltd 2.560 2.921 Word length, number of letters, standard deviation • 12 PCNARz n/a -1.154 -1.042 Text Easability PC Narrativity, z score • 13 PCNARp n/a 12.510 14.920 Text Easability PC Narrativity, percentile • 14 PCSYNz n/a 0.779 -0.334 Text Easability PC Syntactic simplicity, z score • 15 PCSYNp n/a 77.940 37.070 Text Easability PC Syntactic simplicity, percentile • 16 PCCNCz n/a -0.649 1.697 Text Easability PC Word concreteness, z score • 17 PCCNCp n/a 26.110 95.450 Text Easability PC Word concreteness, percentile • 18 PCREFz n/a -1.872 -0.416 Text Easability PC Referential cohesion, z score Text Easibility • 19 PCREFp n/a 3.070 34.090 Text Easability PC Referential cohesion, percentile • 20 PCDCz n/a -0.015 -0.036 Text Easability PC Deep cohesion, z score • 21 PCDCp n/a 49.600 48.800 Text Easability PC Deep cohesion, percentile • 22 PCVERBz n/a -0.279 -2.506 Text Easability PC Verb cohesion, z score • 23 PCVERBp 38.970 0.620 Text Easability PC Verb cohesion, percentile • 24 PCCONNz -3.123 -4.501 Text Easability PC Connectivity, z score • 25 PCCONNp 0.090 0 Text Easability PC Connectivity, percentile • 26 PCTEMPz -2.841 0.952 Text Easability PC Temporality, z score • 27 PCTEMPp 0.230 82.890 Text Easability PC Temporality, percentile 10
Recommend
More recommend
