The Effect of Socio-economic Status on the Think-Aloud Quality in - PowerPoint PPT Presentation

The Effect of Socio-economic Status on the Think-Aloud Quality in Children Mila Sugovic PhD; Anh-Thu Ton MA; Cecilia Teal MS, MSW; Michael Plotkin

2 Background  Cognitive interviewing (Tourangeau, et al., 2000) is one of the most effective methods of eliciting in- depth, multi-faceted feedback from the target populations.  Think aloud techniques have been widely studied however, little research has involved survey pretesting, in particular with children.  Young children typically find the think-aloud method to be difficult (van Someren et al, 1994).  Student’s socio-economic status (SES) may affect their ability to accurately recall information (Ginet et al., 2014) but it is unclear whether the quality of verbal reporting produced by children also differs across SES.

3 Objective and Measures  Objective:  To evaluate whether students’ socio -economic status (SES) affects the quality of the think-aloud produced by grade 4 students.  Measures:  Response Length: number of words used by students during the think-aloud.  Response Relevance: overall relevance of reported think-aloud to questiontopic on a 3- point Likert Scale (1 – Not at all, 2 – Somewhat, 3 – Relevant).  Reported Issues: average number of issues identified by the student.

4 Methods 20 Students Low SES (≤$50K) High SES (>$50K) N=13 N=7 Introduction Introduction 1. Instructions 1. Instructions 2. Demonstration 2. Demonstration 3. Practice 3. Practice Administer Questions & Administer Questions & Think-Aloud Think-Aloud Debriefing Debriefing

5 Task 6 computer-based Science NAEP Assessment Questions (2009)  Question Content: 2 Physical Science, 2 Earth and Space Science, 2 Life Science  Question Difficulty: 2 Easy, 2 Medium, 2 Hard  Question Type: 3 Multiple Choice, 3 Short Constructed Response (SCR)

6 Sample Demographics  Total of 20 fourth grade students completed the cognitive interview.  The sample consisted of a mix of SES, race/ethnicity and gender. Table 1 : Race/Ethnicity by SES Table 2 : Gender by SES High SES Low SES Total High SES Low SES Total Black or African 3 7 10 Female 1 3 4 American 4 6 10 Male 4 2 6 White or Caucasian Total 7 13 20 Hispanic or Latino(a) 0 8 8 2 0 2 Asian Total 7 13 20

7 Results – Response Length by SES 137 136 140 119 120 110 108 Number of Reported Words 104 100 98 96 94 100 85 81 80 High SES 60 Low SES 40 20 Q1 Q2 Q3 Q4 Q5 Q6 Question Number

8 Results – Relevance of Verbal Reports 3.0 3.0 2.8 2.8 2.8 2.6 2.6 2.6 2.6 2.5 2.5 2.4 2.4 Relevance Rating 2.0 High SES Low SES 1.0 .0 Q1 Q2 Q3 Q4 Q5 Q6 Question Number

9 Results – Issues Identified .70 .62 Number of Identified Issues .60 .50 .43 .41 .39 .40 High SES .33 Low SES .30 .24 .20 .15 .14 .10 .10 .10 .00 .00 .00 Q1 Q2 Q3 Q4 Q5 Q6 Question Number

10 Conclusion  Results appear to indicate a modest trend showing that High SES students produced comparably more words than Low SES students.  There is no difference in verbal report relevance between High and Low SES students. Students across both SES groups generally provided relevant responses during the think-alouds.  Results suggest that there is no difference in the number of issues identified across the two SES groups of grade 4 students. Results may be affected by the overall low number of issues identified between the two groups across all questions.  Overall, findings suggest that the quality of information gathered from think-aloud interviewing is not considerably affected by SES among grade 4 students.

11 Limitations and Future Research  Limitations include:  Small sample size (n = 20)  Imbalance in the number of students across the two SES groups(7 High SES, 13 Low SES)  Already revised and tested Science NAEP Questions administered in 2009.  Newly developed NAEP assessments are more interactive and engaging.  To what extent does item engagement promote and interfere with student’s ability to remain on topic?  Does item engagement affect the overall quality of student’s think -aloud?

Anh-Thu Ton, MA Research Analyst tona@eurekafacts.com Certified to the International DBE & MBE certified nationally & in Standard for Market, Opinion and Small Business - US SBA SDB Certified Social Research (ISO 20252) CA , FL, IL, MA, MD, NJ, NY, PA, TX, VA