Selection of irrelevant navigation paths during digital reading: - PowerPoint PPT Presentation

Selection of irrelevant navigation paths during digital reading: Underlying cognitive processes Ladislao Salmerón ERI Lectura, University of Valencia, Spain ProcessDataConf2019

Overview • Navigation paths – Indicators – Interpretation – Context dependencies • Study • Conclusions ProcessDataConf2019

Navigation paths · Major difference between print and digital reading is the presence of hyperlinks that create non-sequential page structures (OECD, 2011, 2015) · Strong association between quality of students’ navigation and digital reading … … at country (OECD, 2015) and at individual (e.g. Naumann & Salmerón, 2016) level. … even after accounting for performance in print reading. In digital reading, process data is already (partially) accounted in performance scores. ProcessDataConf2019

Navigation paths (OECD, 2015) ProcessDataConf2019

Navigation paths · “ Savvy digital readers are expected to navigate across pages that contain information relevant for the question to be answered, or the task to be solved ” · Process indicators of navigation quality (Naumann, 2015) 1. Number of relevant pages visited 2. Total number of pages visited 3. Derived indicators (% relevant pages) ProcessDataConf2019

A page is Relevant if information for the response must be: Navigation paths -located in p. 4 -related p. 1 to p.4 -related 1-2-3-4 1 4 2 3 A page is irrelevant if it doesn’t contain information useful to answer the question, or is not in the path guiding to a relevant one ProcessDataConf2019

Navigation paths · Cognitive processes underlying navigation: a) To relevant pages > inferential process · Print reading comprehension (Hahnel, Goldhammer, Kröhne, & Naumann, 2017, 2018; Naumann, Richter, Christmann, & Groeben, 2008; Naumann & Salmerón, 2016; OECD, 2011, 2015; Salmerón & García, 2011) · Progression across high-school years (Keil & Kominsky, 2013; Salmerón, García & Vidal-Abarca, 2018) ProcessDataConf2019

Navigation paths · Cognitive processes underlying navigation: b) To irrelevant pages: · Generally interpreted as ‘ Lostness ’ (Shapiro & Niederhauser, 2004) · Its isolated effect has been less studied in the past ProcessDataConf2019

Navigation paths · Scenarios in the Digital reading assessment in PISA include highly differentiated sections. ProcessDataConf2019

Navigation paths · Can visits to irrelevant pages also represent students’ attempts to capture a complete view of the theme ? - Self-regulated vs. cue dependent navigation (Balcytiene, 1999) · This aspect should be more obvious in digital texts that are highly topically interconnected, such as Wikipedia texts, than in scenarios with highly differentiated sections, such as web portals. ProcessDataConf2019

Goal · Test if visits to irrelevant pages differ across digital contexts (less / more interconnected): -Relationship with print reading skills -Relationship with digital reading scores -Potential mediation between the former ProcessDataConf2019

Study Participants Grade 7th 154 555 students (12-16 years old) 8th 141 9th 143 · 5 schools (2 rural, 3 urban) 10th 137 · 40 intact classes Exclusion criteria: learning disability, curricular adaptation In class participation. Two sessions of 50’ ProcessDataConf2019

Method • Students’ print comprehension skills (reading literacy) were assessed using a standardized paper and pencil test (Llorens et al., 2011). • Texts: – a textbook excerpt about global warming, – an article from a popular science magazine about the language of the bees, – a pamphlet about the postural ergonomics of work chairs, – two letters to the editor about the pros and cons of nuclear energy, – a newspaper diagram about traffic accidents. • Each unit included 3-5 questions covering the different aspects suggested by the PISA framework • Cronbach’s alpha = .80. ProcessDataConf2019

Method · Two web reading scenarios were designed following the PISA (OECD, 2009) scheme . · Participants were given one question at a time, and were told to use the information from the web pages provided to answer the question. · Questions were constructed following three aspects (OECD, 2009): access and retrieve, integrate and interprete, and reflect and evaluate. · Heterogeneous relationship between question and relevant hyperlinks : word match, paraphrases, irrelevant word matches (Cerdán, Gilabert, & Vidal-Abarca, 2011) ProcessDataConf2019

Method · Web portal scenario: -Portal dedicated to young people -Divided into five categories (sports, courses, technology, nature and health), with three subcategories each. -12 questions required that participants clicked on at least two (maximum seven) links to access relevant information to answer the question, and involved both textual and graphic information, such as tables or maps. ProcessDataConf2019

Method Example question from Web portal scenario #19. You would like to learn how to sing, so you decide to enroll in a course offered by the Youth Portal. Search the address for the center. a) St. Del Padre Vitoria b) St. Mayor 25 c) St. De Donoso Cortés d) St. Goya ProcessDataConf2019

ProcessDataConf2019

Method · Wikipedia scenario: -Main document on the topic of the French Revolution -Included 10 subsections and 24 embedded hyperlinks to additional Wikipedia pages. - 6 questions required that participants clicked on at least one (maximum two) embedded hyperlink(s) to access relevant information to answer the question. ProcessDataConf2019

Method Example question from Wikipedia scenario #13. Why the policy of the Antient Regime was against the new social ideas from the French Revolution? Because … a) … contrary to the Ancient Regime, the new society valued only daily life problems. b) …social ideas from the Ancient Regime were based on the power of the army while new ideas were based on reason. c)… in the new regime the philosophers governed society and in the old regime monarchs and the Church ruled. d) … the new social ideas were based on reason, and the old in divine law. ProcessDataConf2019

ProcessDataConf2019

Method • Log-files indicators: – Number of unique pages visited – Relevant and irrelevant to the task – By scenario (Wikipedia and Web portal) ProcessDataConf2019

Results Descriptive scores for navigation indicators, averaged by question Min Max Mean SD Asymmet Skewness ry Unique Relevant 0 1.67 .37 .33 .41 -.74 visits Wikipedia Unique Irrelevant 0 .33 .06 .10 1.28 .72 visits Wikipedia Unique Relevant 0 3.33 2.0 .74 -1.17 .63 visits Web Portal Unique Irrelevant 0 .33 .10 .40 .39 -.59 visits Web Portal ProcessDataConf2019

Results Web portal scenario: Pearson correlations r = .45* r = .53* Relevant visits Print reading Digital reading comprehension comprehension r = -.13* Irrelevant visits r = -.15* ProcessDataConf2019

Results Web portal scenario Unique relevant visits Unique irrelevant visits Digital Reading comprehension Print Reading comprehension Visits Visits ProcessDataConf2019

Results Wikipedia scenario: Pearson correlations r = .40* r = .61* Relevant visits Print reading Digital reading comprehension comprehension r = .31* Irrelevant visits r = .15* ProcessDataConf2019

Results Wikipedia scenario Unique relevant visits Unique irrelevant visits Digital Reading comprehension Print Reading comprehension Visits Visits ProcessDataConf2019

Results Web portal scenario: Mediation analysis (Bootstraping, Preacher & Hayes (2008)) R 2 adj = .43* .42* Relevant visits .45* Print reading c ’= .34* Digital reading comprehension comprehension -.15* -.17* Irrelevant visits Indirect effects : -relevant visits: .18 [.14, .24] -irrelevant visits: .03 [.01, .05] ProcessDataConf2019

Results Wikipedia scenario: Mediation analysis (Bootstraping, Preacher & Hayes (2008)) R 2 adj = .44* .56* Relevant visits .40* Print reading c ’= .22* Digital reading comprehension comprehension .15* -.02, n.s. Irrelevant visits Indirect effects : -relevant visits: .22 [.17, .27] -irrelevant visits: -.01 [-.02, .01] ProcessDataConf2019

Conclusions • Irrelevant visits in the Web portal scenario, with highly differenciated topic structure: – Negative correlations with print reading comprehension and digital reading performance – Indirect effect of print reading comprehension to digital reading performance. ProcessDataConf2019

Conclusions • Irrelevant visits in the Wikipedia scenario, with highly interconnected topic structure: – Positive correlations with print reading comprehension and digital reading performance – It adds no variance to digital reading performance. ProcessDataConf2019

Conclusions • Underlying cognitive processes? – In loosely interconnected texts: Distraction? (must be independent of inferential processes, accounted for by relevant visits effects). – In highly interconnected texts: Byproduct of inferential processes (exploration, curiosity, Balcytiene, 1999 ) • Need to complement log-files with qualitative data (verbal protocols, interviews) ProcessDataConf2019