Validating a semiadaptive Korean placement test AATK 2014, Boston - PowerPoint PPT Presentation

Validating a semiadaptive Korean placement test AATK 2014, Boston University June 21, 2014 Sun-Young Shin & Hyo Sang Lee Indiana University

Overview • Background - Context for the study - Placement testing; Computerized (semi-)adaptive test - Validity - Outstanding issues • Methods - instruments: IU online semiadaptive Korean placement test & TOPIK - data analysis • Results • Discussion AATK 2 0 1 4 , Boston University #2

Background Context for the study Methods Placement test / CAT/ Validity Results & Discussion Outstanding issues Background • Context for the study  Placement decision in a language program P lacing students at the appropriate levels is important to ensure that course curriculum and materials are well targeted to their learning needs (Green, 2012)  Common placement tools (Brown, Hudson, & Clark, 2004): (i) a proficiency test (institutional) – a M/C & cloze test, & an essay; (ii) an oral interview; (iii) self-placement AATK 2 0 1 4 , Boston University #3

Background Context for the study Methods Placement test / CAT/ Validity Results & Discussion Outstanding issues Background • Context of the problem  Limitation of using a proficiency test for placement in a Korean language program in the U.S. - lack of alignment with the content of courses onto which students are placed (Green & Weir, 2004) - providing little information for high- and low- level of students due to the majority of test items of medium level of difficulty AATK 2 0 1 4 , Boston University #4

Background Context for the study Methods Placement test / CAT/ Validity Results & Discussion Outstanding issues Background  Larger standard error of measurements (SEMs) often obtained at the high end or the low end of the scale particularly problematic in Korean language programs in the US where student populations are often polarized between high (heritage Korean learners) and low-end (non-heritage Korean learners) proficiency levels (Sohn & Shin, 2007 ) AATK 2 0 1 4 , Boston University #5

Background Context for the study Methods Placement test / CAT / Validity Results & Discussion Outstanding issues Background  a computer adaptive language test (CALT) has been recommended to address such problems because its items can be tailored to test takers’ ability levels in a shorter, quicker test, and it also allows for the use of more innovative item types (Chalhoub-Deville & Deville, 1999) AATK 2 0 1 4 , Boston University #6

Background Context for the study Methods Placement test / CAT / Validity Results & Discussion Outstanding issues Background  Limitations of CALT (Meunier, 1994) - large item bank needs to be constructed and piloted - technically challenging to come up with the CALT algorithm - a questionable content/construct validity AATK 2 0 1 4 , Boston University #7

Background Context for the study Methods Placement test / CAT / Validity Results & Discussion Outstanding issues Background  Recommendations made for using semi-adaptive language test (Ockey, 2009) or the testlet model (Wainer & Kiely, 1987) under which a bundle of items are arranged linearly based on test takers’ response to them to overcome such limitations of CALT  To date, however, little studies have been conducted regarding development and validation of an online semi-adaptive foreign language test AATK 2 0 1 4 , Boston University #8

Background Context for the study Methods Placement test / CAT/ Validity Results & Discussion Outstanding issues Background • Validity  Appropriateness of inferences and uses that we make based on test scores (Messick, 1989)  Discussed in terms of different sources of evidence for validity (Bachman, 1990)  Content validity  Concurrent validity  Construct validity AATK 2 0 1 4 , Boston University #9

Background Context for the study Methods Placement test / CAT/ Validity Results & Discussion Outstanding issues Outstanding issues  How can we go about developing an online semi- adaptive Korean placement test?  What are the validity evidence of an online semi- adaptive Korean placement test? AATK 2 0 1 4 , Boston University #10

Background Instruments Methods Data Results & Discussion Data analysis Developing an online semi- adaptive placement test  Why online & computerized?  Mixture of audios, videos, graphics, and texts  Mixture of language skills in a single test item  Semi-adaptive AATK 2 0 1 4 , Boston University #11

Background Instruments Methods Data Results & Discussion Data analysis Characteristics of the test  Why online and computerized? - Any time, any place, year around - Self-assessment, no proctoring needed - Automatic grading - Instant result notification - Incorporating multi-media, multi-modality material - Ease of Data subtraction and analysis AATK 2 0 1 4 , Boston University #12

Background Instruments Methods Data Results & Discussion Data analysis Characteristics of the test  Diverse stimuli: audios, videos, graphics and texts - Reflection of natural language input environments - Measuring receptive and production skills simultaneously - Contextualized and content-oriented as opposed to structure (vocabulary, grammar)-oriented AATK 2 0 1 4 , Boston University #13

Background Instruments Methods Data Results & Discussion Data analysis Characteristics of the test  Mixture of language skills in a single test item - Both questions and options can be in audio or text (Text–test, audio-text, text-audio, audio-audio) - Integrating language skills AATK 2 0 1 4 , Boston University #14

Background Instruments Methods Data Results & Discussion Data analysis Characteristics of the test  Why adaptive? - A single test: All students have to finish - Level-specific tests: an a priori appropriate level cannot be identified AATK 2 0 1 4 , Boston University #15

Background Instruments Methods Data Results & Discussion Data analysis Characteristics of the test  Semi-adaptive: Compromising ideal and practicality under the circumstances; Impractical due to limited resources: requires large amount of test items and sophisticated and complex computer programing - Page-by-page instead of item-by-item: each page has 3 or more question items - Cutoff lines: 1-10 (57 items), 11-22 (50 items), 51-63 (66 items), 64-69 (39 items) - The order of pages reflects the progression of the course material - One needs to get 60% of the items in a given page to move forward - Stopped when failing to get 60% three times and not having gotten 70% of the total items completed at the time AATK 2 0 1 4 , Boston University #16

Background Instruments Methods Data Results & Discussion Data analysis Methods • Data - 112 students enrolled in K101 & K102 at a large Midwestern university who took both an online semi-adaptive Korean placement test (KPT) and Test of Proficiency in Korean (TOPIK) - 336 test takers who took an online semi-adaptive Korean placement test • Data analysis - Data were analyzed for Pearson’s correlation coefficients, Agreement index, and independent sample t-tests using SPSS 21 (2012) AATK 2 0 1 4 , Boston University #17

Background Reliability Methods Validity Results & Discussion Discussion Results • Reliability  Item discrimination & B-index (item bundles – pages) were calculated: - average item discriminations: .31 (reasonably good) - some low discriminatory items (6% of the total items), which need to be revised, have been found - B-index results ( M = .39, ranging from .05 to .79) AATK 2 0 1 4 , Boston University #18

Background Reliability Methods Validity Results & Discussion Discussion Results • Content validity – KPT has a direct relationship to the course contents • Concurrent validity – KPT vs. TOPIK  Correlation coefficient: .75 (item level); .89 (page level)  Agreement index: .71 AATK 2 0 1 4 , Boston University #19

Background Reliability Methods Validity Results & Discussion Discussion Results • Construct validity - differential group differences in performance on a test (experimental approach, Brown, 2005)  Those who passed on TOPIK vs. those who did not on TOPIK scored statistically significantly differently on KPT ( t = 7.64, p < .00) AATK 2 0 1 4 , Boston University #20

Background Reliability Methods Validity Results & Discussion Discussion Results Construct validity – internal evidence Do the testlet (item bundles) for each level differ in the intended difficulty? Levels Pages (#s) Questions (#s) Mean IFs K101 10 57 .72 (.13) K102 12 50 .43 (.09) K201 13 66 .22 (.04) K202 6 39 .20 (.01) F (3,37)=81.01, p <.00 Yes, but except for K201 vs. K202 AATK 2 0 1 4 , Boston University #21

Background Reliability Methods Validity Results & Discussion Discussion Discussion • An online semi-adaptive placement test can be developed and implemented for Korean language programs in a relatively reliable and valid manner • Note that this is costly and time-consuming project, so the cost-benefit analysis should be conducted in advance • Classification errors for placement based on KPT is minimal but some false-negative errors were found • Some items still need to be revised to better the test AATK 2 0 1 4 , Boston University #22