ONLINE (PHASE 2) Kijoo Ko Korean Language Program, UC Berkeley - - PowerPoint PPT Presentation

▶
online
SMART_READER_LITE
LIVE PREVIEW

ONLINE (PHASE 2) Kijoo Ko Korean Language Program, UC Berkeley - - PowerPoint PPT Presentation

Dec 28, 2023 156 likes •545 views

ONLINE (PHASE 2) Kijoo Ko Korean Language Program, UC Berkeley Korean Placement Test To assign students to the right level in the Korean language program (Brown, 1996; Rapi & Miconi, 2014) Computer Adaptive Language Test (CALT)

slide-1
SLIDE 1

ONLINE

(PHASE 2)

Kijoo Ko Korean Language Program, UC Berkeley

slide-2
SLIDE 2

▪ Korean Placement Test

▪ To assign students to the right level in the Korean language program

(Brown, 1996; Rapi & Miconi, 2014)

▪ Computer Adaptive Language Test (CALT)

▪ Computer-tailored test to the student’s level

(Meunier, 1994; Long et als., 2018)

slide-3
SLIDE 3

▪ Efficiency

▪ Developers: can easily upload/update test items ▪ Students: can take the test at any place/time, can get the result immediately ▪ Teachers: can save time for making/administrating/grading tests, can analyze data easily

▪ Authenticity

▪ Can use multimodal inputs

▪ Flexibility

▪ Individualized test ▪ Self-paced, less boring/frustrating

(Meunier, 1994; Long et als., 2018)

slide-4
SLIDE 4

▪ Levels

▪ 5 levels: Elementary, Intermediate, Advanced, 4th, 5th year Korean

▪ Tracks

▪ Heritage track: K1AX/BX, K10AX/BX, K100AX/BX ▪ Non-Heritage track: K1A/B, K10A/B, K100A/B

▪ Students

▪ 300-350 per semester

▪ Instructors

▪ 9 full-time lecturers (Spring, 2019), 1 tutor

slide-5
SLIDE 5

Current:

Time-consuming, labor-intensive test managed by individual KLP instructors

Goal:

To provide a reliable and practical measure of students’ level of Korean as

  • ne factor in assessing the appropriate course for a student.
slide-6
SLIDE 6

(PHASE 1)

▪ When

▪ Fall, 2017

▪ Who

▪ Kijoo Ko & Seung-Eun Chang

▪ How

▪ Support by CKS, BLC (BOLT)

▪ What

▪ Developed Item Set #1: 450 test items, 9 levels, 50 questions for each level ▪ Administered 4 Norming tests: Entrance & Exit tests in Spring, 2018 & Fall 2018

slide-7
SLIDE 7

(PHASE 2)

▪ When

▪ Spring, 2019

▪ Who

▪ Kijoo Ko

▪ How

▪ Support by BLC

▪ What

▪ Analyzed the norming test results ▪ Revised Item Set #1, Developed Item Set #2

slide-8
SLIDE 8

▪ Literature review ▪ Chinese & Japanese placement tests at UCB ▪ Korean placement tests at other universities ▪ Other official Korean tests ▪ Analysis of the norming test results ▪ Revision for Items Set #1 ▪ Addition for Items Set #2

(PHASE 2)

slide-9
SLIDE 9

▪ Why

▪ Do students really know the test item when it is taught? ▪ If not, we need to revise (question or level or distractor) or delete

▪ Who

▪ 598 students taking a Korean class at UCB in 2018

▪ When

▪ Entrance test: beginning of the semester ▪ Exit test: end of the semester

▪ What

▪ Level X: previous-level contents ▪ Level Y: next-level contents

slide-10
SLIDE 10

▪ Item Facility (IF) analysis

▪ IF score: (# of students answered correctly)/(# of students answered) ▪ 0.25 would be the result for guessing

▪ Item Discrimination (ID) analysis

▪ ID score: Difference between the IF score of students below level

and the IF score of students above the level of the question

(Brown, 1996; Lee, 2000) For example, Group –L IF score 0.3 Group +L IF score 0.8

ID score 0.5 ID score < 0.2 : not good 0.2 < ID score < 0.4 : ok ID score > 0.4 : great

slide-11
SLIDE 11

Q QL SL0 SL1 ID score 5145 2 0.57 0.53

  • 0.04

5008 2 0.93 0.9

  • 0.03

5010 2 0.93 0.91

  • 0.02

5146 2 0.64 0.64 5150 2 0.69 0.7 0.01 5011 2 0.86 0.88 0.02 5089 2 0.79 0.82 0.03 5009 2 0.86 0.89 0.03 5007 2 0.86 0.91 0.05 5141 2 0.57 0.67 0.1 5005 2 0.5 0.61 0.11 5087 2 0.69 0.8 0.11 5148 2 0.67 0.79 0.12 5003 2 0.5 0.63 0.13 5086 2 0.69 0.82 0.13 5090 2 0.71 0.84 0.13 5006 2 0.6 0.74 0.14 5085 2 0.71 0.85 0.14 5088 2 0.77 0.93 0.16 5091 2 0.57 0.73 0.16 5197 2 0.71 0.87 0.16 5204 2 0.69 0.85 0.16 5194 2 0.57 0.75 0.18 5199 2 0.54 0.73 0.19 5151 2 0.57 0.76 0.19 5143 2 0.62 0.83 0.21 5140 2 0.69 0.91 0.22 5136 2 0.54 0.77 0.23 5203 2 0.5 0.73 0.23 5138 2 0.21 0.44 0.23 5084 2 0.64 0.89 0.25 5117 2 0.21 0.46 0.25 5200 2 0.57 0.82 0.25 5202 2 0.5 0.75 0.25 5149 2 0.21 0.48 0.27 5198 2 0.38 0.65 0.27 5139 2 0.64 0.92 0.28 5144 2 0.46 0.75 0.29 5134 2 0.46 0.76 0.3 5205 2 0.46 0.78 0.32 5142 2 0.21 0.53 0.32 5116 2 0.54 0.87 0.33 5196 2 0.54 0.89 0.35 5201 2 0.4 0.75 0.35 5133 2 0.54 0.91 0.37 5002 2 0.23 0.63 0.4 5147 2 0.15 0.58 0.43 5135 2 0.38 0.88 0.5 5115 2 0.07 0.6 0.53 5195 2 0.21 0.8 0.59 Q QL SL1 SL2 ID score 5118 3 0.62 0.5

  • 0.12

5098 3 0.91 0.83

  • 0.08

5015 3 0.89 0.88

  • 0.01

5014 3 0.83 0.85 0.02 5016 3 0.76 0.79 0.03 5221 3 0.25 0.28 0.03 5222 3 0.24 0.27 0.03 5164 3 0.37 0.41 0.04 5017 3 0.75 0.83 0.08 5099 3 0.72 0.8 0.08 5154 3 0.57 0.65 0.08 5092 3 0.77 0.87 0.1 5206 3 0.47 0.57 0.1 5207 3 0.29 0.39 0.1 5093 3 0.79 0.9 0.11 5120 3 0.41 0.52 0.11 5012 3 0.78 0.9 0.12 5218 3 0.33 0.46 0.13 5208 3 0.42 0.55 0.13 5161 3 0.15 0.29 0.14 5018 3 0.79 0.93 0.14 5166 3 0.18 0.33 0.15 5220 3 0.68 0.85 0.17 5013 3 0.7 0.87 0.17 5159 3 0.34 0.52 0.18 5097 3 0.72 0.9 0.18 5213 3 0.41 0.61 0.2 5094 3 0.73 0.93 0.2 5223 3 0.1 0.31 0.21 5155 3 0.24 0.46 0.22 5153 3 0.58 0.83 0.25 5156 3 0.3 0.55 0.25 5158 3 0.33 0.59 0.26 5163 3 0.4 0.66 0.26 5121 3 0.48 0.79 0.31 5160 3 0.5 0.82 0.32 5122 3 0.22 0.55 0.33 5095 3 0.34 0.68 0.34 5096 3 0.42 0.79 0.37 5119 3 0.31 0.68 0.37 5217 3 0.42 0.79 0.37 5209 3 0.33 0.71 0.38 5165 3 0.25 0.63 0.38 5219 3 0.35 0.73 0.38 5157 3 0.3 0.69 0.39 5214 3 0.38 0.77 0.39 5211 3 0.32 0.74 0.42 5152 3 0.51 0.93 0.42 5162 3 0.22 0.73 0.51 5212 3 0.31 0.88 0.57 Q QL SL2 SL3 ID score 5175 4 0.34 0.24

  • 0.1

5102 4 0.24 0.2

  • 0.04

5106 4 0.8 0.77

  • 0.03

5224 4 0.56 0.59 0.03 5168 4 0.17 0.2 0.03 5103 4 0.87 0.9 0.03 5025 4 0.33 0.37 0.04 5026 4 0.86 0.9 0.04 5125 4 0.65 0.71 0.06 5101 4 0.89 0.96 0.07 5024 4 0.78 0.86 0.08 5180 4 0.7 0.79 0.09 5123 4 0.39 0.49 0.1 5173 4 0.36 0.46 0.1 5237 4 0.52 0.64 0.12 5100 4 0.43 0.55 0.12 5019 4 0.85 0.98 0.13 5022 4 0.69 0.82 0.13 5169 4 0.26 0.39 0.13 5104 4 0.75 0.91 0.16 5228 4 0.29 0.45 0.16 5124 4 0.21 0.38 0.17 5235 4 0.6 0.77 0.17 5225 4 0.32 0.5 0.18 5236 4 0.18 0.36 0.18 5105 4 0.75 0.94 0.19 5176 4 0.19 0.4 0.21 5020 4 0.69 0.92 0.23 5177 4 0.53 0.77 0.24 5179 4 0.2 0.44 0.24 5167 4 0.18 0.43 0.25 5170 4 0.46 0.74 0.28 5021 4 0.64 0.94 0.3 5232 4 0.53 0.83 0.3 5227 4 0.24 0.56 0.32 5238 4 0.35 0.68 0.33 5174 4 0.55 0.89 0.34 5126 4 0.52 0.87 0.35 5231 4 0.32 0.67 0.35 5023 4 0.6 0.96 0.36 5127 4 0.36 0.73 0.37 5181 4 0.42 0.79 0.37 5178 4 0.09 0.47 0.38 5229 4 0.14 0.52 0.38 5233 4 0.37 0.77 0.4 5171 4 0.38 0.8 0.42 5230 4 0.18 0.63 0.45 5234 4 0.29 0.77 0.48 5226 4 0.23 0.77 0.54 5172 4 0.26 0.85 0.59 Q QL SL3 SL4 ID score 5183 5 0.2

  • 0.2

5188 5 0.38 0.25

  • 0.13

5114 5 0.94 0.83

  • 0.11

5189 5 0.51 0.42

  • 0.09

5254 5 0.63 0.58

  • 0.05

5131 5 0.41 0.36

  • 0.05

5182 5 0.32 0.29

  • 0.03

5111 5 0.94 0.92

  • 0.02

5184 5 0.41 0.42 0.01 5028 5 0.54 0.58 0.04 5029 5 0.88 0.92 0.04 5108 5 0.96 1 0.04 5032 5 0.69 0.77 0.08 5033 5 0.84 0.92 0.08 5192 5 0.64 0.73 0.09 5027 5 0.9 1 0.1 5110 5 0.72 0.83 0.11 5239 5 0.72 0.83 0.11 5250 5 0.39 0.5 0.11 5186 5 0.73 0.85 0.12 5107 5 0.87 1 0.13 5130 5 0.69 0.82 0.13 5251 5 0.54 0.67 0.13 5255 5 0.54 0.67 0.13 5256 5 0.6 0.73 0.13 5253 5 0.85 1 0.15 5244 5 0.42 0.58 0.16 5243 5 0.57 0.75 0.18 5112 5 0.72 0.92 0.2 5187 5 0.71 0.92 0.21 5132 5 0.11 0.33 0.22 5193 5 0.57 0.83 0.26 5030 5 0.5 0.77 0.27 5245 5 0.4 0.67 0.27 5109 5 0.64 0.92 0.28 5249 5 0.46 0.75 0.29 5129 5 0.38 0.67 0.29 5128 5 0.28 0.58 0.3 5190 5 0.62 0.92 0.3 5248 5 0.53 0.85 0.32 5247 5 0.43 0.75 0.32 5242 5 0.59 0.92 0.33 5240 5 0.32 0.67 0.35 5031 5 0.63 1 0.37 5191 5 0.38 0.75 0.37 5185 5 0.52 0.92 0.4 5241 5 0.5 0.92 0.42 5246 5 0.3 0.77 0.47 5113 5 0.26 0.75 0.49 5252 5 0.43 0.92 0.49 Q QL SL0 SL1 ID score 5004 1 0.95 0.93

  • 0.03

5061 1 0.84 0.85 0.01 5042 1 0.55 0.57 0.02 5047 1 0.91 0.93 0.02 5057 1 0.89 0.92 0.03 5060 1 0.89 0.92 0.04 5073 1 0.81 0.86 0.04 5038 1 0.95 1.00 0.05 5041 1 0.88 0.93 0.05 5037 1 0.94 1.00 0.06 5039 1 0.86 0.93 0.06 5062 1 0.86 0.92 0.06 5049 1 0.93 1.00 0.07 5035 1 0.93 1.00 0.07 5036 1 0.93 1.00 0.07 5048 1 0.92 1.00 0.08 5065 1 0.92 1.00 0.08 5081 1 0.70 0.79 0.09 5059 1 0.91 1.00 0.09 5034 1 0.91 1.00 0.09 5078 1 0.83 0.93 0.10 5072 1 0.89 1.00 0.11 5044 1 0.76 0.87 0.11 5040 1 0.88 1.00 0.12 5058 1 0.78 0.92 0.14 5043 1 0.86 1.00 0.14 5079 1 0.86 1.00 0.14 5075 1 0.75 0.92 0.17 5053 1 0.61 0.79 0.17 5056 1 0.75 0.93 0.18 5063 1 0.73 0.92 0.19 5077 1 0.62 0.85 0.22 5080 1 0.77 1.00 0.23 5052 1 0.68 0.93 0.25 5070 1 0.68 0.93 0.25 5055 1 0.70 1.00 0.30 5050 1 0.54 0.85 0.31 5082 1 0.61 0.92 0.31 5074 1 0.53 0.85 0.32 5068 1 0.59 0.93 0.34 5067 1 0.50 0.85 0.35 5045 1 0.50 0.86 0.36 5064 1 0.56 0.93 0.37 5046 1 0.62 1.00 0.38 5051 1 0.54 0.92 0.38 5076 1 0.48 0.92 0.44 5066 1 0.47 0.92 0.45 5069 1 0.55 1.00 0.45 5054 1 0.47 0.93 0.46 5071 1 0.44 0.92 0.49

Levels 1-5

slide-12
SLIDE 12

Q QL SL4 SL5 ID score 5257 6 0.6 0.2

  • 0.4

5430 6 0.38 0.15

  • 0.23

5286 6 0.15 0.05

  • 0.1

5280 6 0.26 0.18

  • 0.08

5260 6 0.15 0.1

  • 0.05

5402 6 0.45 0.41

  • 0.04

5298 6 0.14 0.1

  • 0.04

5259 6 0.63 0.6

  • 0.03

5404 6 0.75 0.73

  • 0.02

5422 6 0.82 0.8

  • 0.02

5289 6 0.8 0.8 5388 6 0.48 0.48 5423 6 0.91 0.95 0.04 5401 6 0.48 0.52 0.04 5294 6 0.95 1 0.05 5384 6 0.48 0.53 0.05 5287 6 0.67 0.75 0.08 5429 6 0.62 0.7 0.08 5258 6 0.76 0.85 0.09 5299 6 0.79 0.9 0.11 5302 6 0.83 0.95 0.12 5426 6 0.68 0.8 0.12 5288 6 0.55 0.68 0.13 5282 6 0.28 0.42 0.14 5261 6 0.1 0.25 0.15 5403 6 0.85 1 0.15 5386 6 0.33 0.5 0.17 5300 6 0.73 0.9 0.17 5296 6 0.52 0.7 0.18 5387 6 0.76 0.95 0.19 5292 6 0.74 0.95 0.21 5421 6 0.64 0.85 0.21 5291 6 0.47 0.68 0.21 5281 6 0.42 0.65 0.23 5301 6 0.71 0.95 0.24 5285 6 0.37 0.62 0.25 5290 6 0.6 0.85 0.25 5284 6 0.23 0.5 0.27 5428 6 0.45 0.75 0.3 5427 6 0.64 0.95 0.31 5425 6 0.36 0.7 0.34 5424 6 0.32 0.7 0.38 5400 6 0.57 0.95 0.38 5283 6 0.35 0.74 0.39 5385 6 0.29 0.68 0.39 5278 6 0.56 1 0.44 5295 6 0.28 0.75 0.47 5279 6 0.45 0.95 0.5 5297 6 0.35 0.89 0.54 5293 6 0.2 0.81 0.61 Q QL SL5 SL6 ID score 5319 7 0.65 0.5

  • 0.15

5393 7 0.6 0.5

  • 0.1

5316 7 0.71 0.63

  • 0.08

5314 7 0.57 0.5

  • 0.07

5264 7 0.68 0.63

  • 0.05

5327 7 0.64 0.63

  • 0.01

5265 7 0.74 0.75 0.01 5437 7 0.85 0.88 0.03 5305 7 0.85 0.89 0.04 5405 7 0.84 0.89 0.05 5431 7 0.7 0.75 0.05 5434 7 0.82 0.88 0.06 5303 7 0.68 0.75 0.07 5306 7 0.67 0.75 0.08 5436 7 0.55 0.63 0.08 5439 7 0.65 0.75 0.1 5312 7 0.76 0.88 0.12 5409 7 0.63 0.75 0.12 5266 7 0.37 0.5 0.13 5318 7 0.85 1 0.15 5390 7 0.52 0.67 0.15 5406 7 0.63 0.78 0.15 5433 7 0.85 1 0.15 5307 7 0.58 0.75 0.17 5392 7 0.71 0.88 0.17 5311 7 0.45 0.63 0.18 5324 7 0.81 1 0.19 5320 7 0.7 0.89 0.19 5263 7 0.55 0.75 0.2 5326 7 0.68 0.88 0.2 5438 7 0.55 0.75 0.2 5440 7 0.8 1 0.2 5308 7 0.58 0.78 0.2 5391 7 0.53 0.75 0.22 5317 7 0.41 0.63 0.22 5262 7 0.4 0.63 0.23 5435 7 0.27 0.5 0.23 5389 7 0.63 0.88 0.25 5410 7 0.63 0.88 0.25 5304 7 0.47 0.75 0.28 5315 7 0.6 0.88 0.28 5310 7 0.45 0.75 0.3 5323 7 0.7 1 0.3 5432 7 0.7 1 0.3 5313 7 0.3 0.63 0.33 5321 7 0.53 0.88 0.35 5322 7 0.6 1 0.4 5309 7 0.05 0.5 0.45 5408 7 0.48 1 0.52 5325 7 0.26 1 0.74 Q QL SL6 SL7 ID score 5343 8 0.89 0.70

  • 0.19

5333 8 0.22 0.05

  • 0.17

5413 8 0.64 0.50

  • 0.14

5414 8 0.78 0.65

  • 0.13

5267 8 0.4 0.30

  • 0.10

5339 8 0.5 0.40

  • 0.10

5444 8 0.7 0.60

  • 0.10

5449 8 0.89 0.81

  • 0.08

5446 8 0.82 0.75

  • 0.07

5411 8 1 0.95

  • 0.05

5448 8 1 0.95

  • 0.05

5341 8 0.78 0.75

  • 0.03

5442 8 0.78 0.75

  • 0.03

5354 8 0.22 0.20

  • 0.02

5412 8 0.82 0.80

  • 0.02

5332 8 0.91 0.90

  • 0.01

5447 8 0.91 0.90

  • 0.01

5445 8 0.6 0.60 0.00 5450 8 0.67 0.67 0.00 5394 8 0.22 0.25 0.03 5271 8 0.11 0.15 0.04 5395 8 0.36 0.40 0.04 5335 8 0.33 0.38 0.05 5348 8 0.78 0.85 0.07 5349 8 0.67 0.75 0.08 5441 8 0.89 1.00 0.11 5351 8 0.78 0.90 0.12 5415 8 0.78 0.90 0.12 5443 8 0.78 0.90 0.12 5336 8 0.67 0.80 0.13 5355 8 0.7 0.85 0.15 5330 8 0.44 0.60 0.16 5342 8 0.44 0.60 0.16 5345 8 0.78 0.95 0.17 5340 8 0.56 0.76 0.20 5328 8 0.7 0.90 0.20 5331 8 0.6 0.80 0.20 5353 8 0.6 0.80 0.20 5329 8 0.6 0.85 0.25 5337 8 0.5 0.75 0.25 5269 8 0.44 0.70 0.26 5270 8 0.3 0.60 0.30 5346 8 0.44 0.75 0.31 5344 8 0.44 0.76 0.32 5347 8 0.67 1.00 0.33 5334 8 0.5 0.85 0.35 5268 8 0.44 0.81 0.37 5350 8 0.33 0.80 0.47 5396 8 0.4 0.90 0.50 5352 8 0.33 0.85 0.52 Q QL SL8 SL9 ID score 5375 9 0.60 0.36

  • 0.24

5272 9 0.60 0.43

  • 0.17

5381 9 0.80 0.64

  • 0.16

5398 9 0.55 0.43

  • 0.12

5416 9 0.57 0.5

  • 0.07

5380 9 0.85 0.79

  • 0.06

5420 9 0.85 0.79

  • 0.06

5361 9 0.55 0.5

  • 0.05

5370 9 0.40 0.36

  • 0.04

5452 9 0.90 0.86

  • 0.04

5357 9 0.95 0.93

  • 0.02

5453 9 0.95 0.93

  • 0.02

5417 9 0.86 0.86 0.00 5275 9 0.14 0.15 0.01 5418 9 0.55 0.57 0.02 5372 9 0.90 0.93 0.03 5451 9 0.90 0.93 0.03 5374 9 0.10 0.14 0.04 5359 9 0.45 0.5 0.05 5356 9 0.30 0.36 0.06 5379 9 0.15 0.21 0.06 5378 9 0.14 0.21 0.07 5274 9 0.57 0.64 0.07 5360 9 0.85 0.93 0.08 5376 9 0.85 0.93 0.08 5458 9 0.71 0.79 0.08 5366 9 0.55 0.64 0.09 5382 9 0.90 1 0.10 5383 9 0.90 1 0.10 5367 9 0.75 0.86 0.11 5373 9 0.10 0.21 0.11 5364 9 0.30 0.43 0.13 5371 9 0.20 0.36 0.16 5363 9 0.40 0.57 0.17 5419 9 0.40 0.57 0.17 5377 9 0.76 0.93 0.17 5273 9 0.10 0.29 0.19 5457 9 0.43 0.64 0.21 5369 9 0.57 0.79 0.22 5399 9 0.29 0.54 0.25 5456 9 0.67 0.93 0.26 5455 9 0.65 0.93 0.28 5460 9 0.50 0.79 0.29 5358 9 0.55 0.85 0.30 5397 9 0.60 0.93 0.33 5365 9 0.30 0.64 0.34 5368 9 0.65 1 0.35 5454 9 0.50 0.93 0.43 5362 9 0.50 1 0.50 5459 9 0.35 0.86 0.51

Levels 6-9

slide-13
SLIDE 13

▪ Check out IF scores, ID score ▪ Investigate to see why ▪ Make a decision

▪ Keep the item ▪ Revise the item

  • the difficulty level
  • the question
  • the answer
  • the distractor(s)

▪ Delete the item

slide-14
SLIDE 14

ID score Result

  • 0.2

revise

  • 0.13

revise

  • 0.11

revise

  • 0.09

revise

  • 0.05

revise

  • 0.05

keep

  • 0.03

keep

  • 0.02

revise 0.01 revise 0.04 revise 0.04 revise 0.04 revise 0.08 keep 0.08 revise 0.09 keep 0.1 revise 0.11 revise 0.11 revise 0.11 keep 0.12 keep 0.13 revise 0.13 revise 0.13 revise 0.13 keep 0.13 revise 0.15 revise 0.16 revise 0.18 revise 0.2 revise 0.21 revise 0.22 revise 0.26 keep 0.27 revise 0.27 revise 0.28 revise 0.29 revise 0.29 keep 0.3 revise 0.3 revise 0.32 keep 0.32 keep 0.33 revise 0.35 keep 0.37 keep 0.37 keep 0.4 revise 0.42 keep 0.47 keep 0.49 keep 0.49 keep ID score Result

  • 0.1

revise

  • 0.04

revise

  • 0.03

revise 0.03 revise 0.03 revise 0.03 revise 0.04 revise 0.04 revise 0.06 revise 0.07 revise 0.08 revise 0.09 revise 0.1 revise 0.1 keep 0.12 revise 0.12 revise 0.13 revise 0.13 revise 0.13 revise 0.16 revise 0.16 revise 0.17 keep 0.17 revise 0.18 revise 0.18 revise 0.19 revise 0.21 revise 0.23 revise 0.24 keep 0.24 keep 0.25 keep 0.28 keep 0.3 revise 0.3 keep 0.32 keep 0.33 keep 0.34 keep 0.35 keep 0.35 keep 0.36 revise 0.37 keep 0.37 keep 0.38 keep 0.38 keep 0.4 keep 0.42 keep 0.45 keep 0.48 keep 0.54 keep 0.59 keep ID score Result

  • 0.12

revise

  • 0.08

revise

  • 0.01

revise 0.02 revise 0.03 revise 0.03 revise 0.03 revise 0.04 revise 0.08 revise 0.08 revise 0.08 revise 0.1 revise 0.1 revise 0.1 keep 0.11 revise 0.11 revise 0.12 revise 0.13 keep 0.13 revise 0.14 revise 0.14 revise 0.15 revise 0.17 revise 0.17 revise 0.18 keep 0.18 revise 0.2 keep 0.2 keep 0.21 revise 0.22 keep 0.25 keep 0.25 keep 0.26 keep 0.26 keep 0.31 keep 0.32 keep 0.33 keep 0.34 keep 0.37 keep 0.37 keep 0.37 keep 0.38 keep 0.38 keep 0.38 keep 0.39 keep 0.39 keep 0.42 keep 0.42 keep 0.51 keep 0.57 keep ID score Result

  • 0.04

keep

  • 0.03

revise

  • 0.02

revise revise 0.01 revise 0.02 revise 0.03 revise 0.03 revise 0.05 revise 0.1 keep 0.11 revise 0.11 revise 0.12 keep 0.13 keep 0.13 revise 0.13 revise 0.14 keep 0.14 revise 0.16 revise 0.16 keep 0.16 revise 0.16 keep 0.18 keep 0.19 keep 0.19 keep 0.21 keep 0.22 revise 0.23 revise 0.23 revise 0.23 keep 0.25 revise 0.25 keep 0.25 keep 0.25 keep 0.27 keep 0.27 keep 0.28 keep 0.29 keep 0.3 keep 0.32 keep 0.32 keep 0.33 keep 0.35 keep 0.35 keep 0.37 keep 0.4 keep 0.43 keep 0.5 keep 0.53 keep 0.59 keep ID score Result

  • 0.03

revise 0.01 revise 0.02 revise 0.02 keep 0.03 keep 0.04 revise 0.04 keep 0.05 revise 0.05 revise 0.06 revise 0.06 revise 0.06 revise 0.07 keep 0.07 revise 0.07 revise 0.08 keep 0.08 revise 0.09 revise 0.09 revise 0.09 revise 0.10 revise 0.11 revise 0.11 revise 0.12 keep 0.14 revise 0.14 revise 0.14 revise 0.17 revise 0.17 keep 0.18 revise 0.19 revise 0.22 keep 0.23 revise 0.25 revise 0.25 revise 0.30 keep 0.31 keep 0.31 revise 0.32 keep 0.34 keep 0.35 keep 0.36 keep 0.37 keep 0.38 keep 0.38 keep 0.44 keep 0.45 keep 0.45 keep 0.46 keep 0.49 keep

Levels 1-5

keep revise delete

slide-15
SLIDE 15

ID score Result

  • 0.24

revise

  • 0.17

revise

  • 0.16

revise

  • 0.12

revise

  • 0.07

delete

  • 0.06

revise

  • 0.06

delete

  • 0.05

revise

  • 0.04

keep

  • 0.04

revise

  • 0.02

delete

  • 0.02

delete 0.00 delete 0.01 revise 0.02 delete 0.03 revise 0.03 revise 0.04 revise 0.05 revise 0.06 keep 0.06 revise 0.07 revise 0.07 revise 0.08 revise 0.08 revise 0.08 revise 0.09 keep 0.10 revise 0.10 revise 0.11 revise 0.11 revise 0.13 revise 0.16 revise 0.17 delete 0.17 delete 0.17 revise 0.19 revise 0.21 revise 0.22 revise 0.25 keep 0.26 revise 0.28 keep 0.29 revise 0.30 revise 0.33 keep 0.34 keep 0.35 keep 0.43 keep 0.50 keep 0.51 keep ID score Result

  • 0.19

revise

  • 0.17

revise

  • 0.14

delete

  • 0.13

delete

  • 0.10

revise

  • 0.10

revise

  • 0.10

delete

  • 0.08

delete

  • 0.07

delete

  • 0.05

delete

  • 0.05

delete

  • 0.03

delete

  • 0.03

delete

  • 0.02

revise

  • 0.02

delete

  • 0.01

delete

  • 0.01

delete 0.00 delete 0.00 delete 0.03 revise 0.04 revise 0.04 delete 0.05 delete 0.07 revise 0.08 revise 0.11 delete 0.12 revise 0.12 delete 0.12 delete 0.13 revise 0.15 revise 0.16 revise 0.16 revise 0.17 revise 0.20 revise 0.20 revise 0.20 revise 0.20 revise 0.25 revise 0.25 revise 0.26 keep 0.30 revise 0.31 revise 0.32 revise 0.33 revise 0.35 keep 0.37 revise 0.47 keep 0.50 keep 0.52 keep ID score Result

  • 0.15

delete

  • 0.1

keep

  • 0.08

keep

  • 0.07

keep

  • 0.05

delete

  • 0.01

keep 0.01 revise 0.03 revise 0.04 delete 0.05 revise 0.05 delete 0.06 revise 0.07 delete 0.08 delete 0.08 revise 0.1 revise 0.12 delete 0.12 revise 0.13 revise 0.15 delete 0.15 revise 0.15 revise 0.15 delete 0.17 revise 0.17 revise 0.18 delete 0.19 revise 0.19 revise 0.2 revise 0.2 revise 0.2 revise 0.2 delete 0.2 revise 0.22 revise 0.22 delete 0.23 keep 0.23 keep 0.25 keep 0.25 revise 0.28 revise 0.28 keep 0.3 revise 0.3 keep 0.3 delete 0.33 delete 0.35 revise 0.4 keep 0.45 delete 0.52 revise 0.74 keep ID score Result

  • 0.4

revise

  • 0.17

revise

  • 0.1

revise

  • 0.08

revise

  • 0.05

revise

  • 0.04

revise

  • 0.04

revise

  • 0.03

keep

  • 0.02

revise

  • 0.02

revise keep delete 0.04 revise 0.04 revise 0.05 revise 0.05 revise 0.08 delete 0.08 keep 0.09 keep 0.11 revise 0.12 revise 0.12 revise 0.13 delete 0.14 revise 0.15 revise 0.15 revise 0.17 revise 0.17 revise 0.18 keep 0.19 revise 0.21 revise 0.21 revise 0.21 keep 0.23 revise 0.24 revise 0.25 keep 0.25 keep 0.27 keep 0.3 revise 0.31 revise 0.34 keep 0.38 keep 0.38 revise 0.39 keep 0.39 keep 0.44 keep 0.47 keep 0.5 keep 0.54 keep 0.61 keep

Levels 6-9

keep revise delete

slide-16
SLIDE 16

Keep Revise Delete

slide-17
SLIDE 17

[TIP 1]

It takes time. Be patient and plan well in advance.

  • Online template -> Item development -> Norming test -> Analysis -> Revision ->

Add new items ….. (+ Tech issues)

  • Depending on the time when administered -> May not test the “true” level ->

Incorrect data

slide-18
SLIDE 18

쌀 살 잘 찰

  • ID score: 0.46 -> Keep the item
  • Period: 1st - 3rd weeks -> Actual ID score should be larger

Listen to the Korean syllable and choose the corresponding Korean alphabet letter.

(ㅆ: tensed) (ㅅ: regular) (ㅈ: voiced) (ㅊ: aspirated)

  • Make an overall plan to consider timing issues.
slide-19
SLIDE 19

[TIP 2]

Check out various possibilities to understand the data.

  • Some data show obvious results, others do not.
  • Data can give you a clue on how to revise the test item.
slide-20
SLIDE 20

K1AX heritage (L1) K1BX heritage (L2)

  • 1. 누구 (whose)
  • 2. 누가 (who, as subject)
  • 3. 뭐 (what)
  • 4. 어떻게 (how)

(Whose dictionary is this?)

Choose the answer that best fits into the blank.

  • Original ID score: -0.04
  • H-group-only ID score: 0.35 -> Keep the item
  • Some data show obvious results, others do not.
slide-21
SLIDE 21
  • One reading passage, two questions
  • Both problematic ID scores, but different IF scores patterns
  • Revise the items to be easier for low IF scores (5102) and harder for high IF scores (5103)

(5102) 지난 -> 이번 to make the ambiguous word to be clearer (5103) Which statement correctly describes Steve’s boardinghouse? (1) The rent is relatively cheap. (2) His room is small but a common area is spacious. (3) It provides all meals at good prices. (4) The floor is a bit cold in winter.

스티브는 지난 1월에 한국에 왔습니다. 스티브는 그동안 학교 기숙사에서 살다가 첫 여름 방학이 돼서 지난 주말에 학교 앞 하숙집으로 옮겼습니다. 하숙집은 방은 작지만 주인 아주머니가 아주 친절하시고 하숙비도 싼 편입니다. 또한, 하숙집은 아침과 저녁을 주기 때문에 점심만 학교에서 사 먹으면 됩니다. 스티브의 하숙방은 온돌방입니다. 온돌방은 바닥이 따뜻해서 추운 겨울에도 바닥에서 잘 수 있습니다. 스티브는 처음에는 온돌방이 불편했지만, 이제는 미국에 돌아가면 온돌방이 그리울 것 같습니다.

Read the passage and choose the correct answer.

Q Item L3 IF score L4 IF score ID score 5102 4R3 0.24 0.2

  • 0.04

5103 4R4 0.87 0.9 0.03

  • Data can give you a clue how to revise the item.
slide-22
SLIDE 22

[TIP 3]

The format of test items may bring different results.

  • Try to use multiple-choices type, not True/False type.
  • Consider whether you will give listening files once or multiple times.
  • Try not to use negative formats for instructions.
slide-23
SLIDE 23
  • High IF scores for True/False type question format
  • Revise the item to be more difficult (-> multiple choices)

[Reading] Read the passage and fill in the blanks “T” true or “F” false.

Q Item L2 IF score L3 IF score ID score 5097 3R6 0.72 0.9 0.18 5098 3R7 0.91 0.83

  • 0.08

5099 3R8 0.72 0.8 0.08

  • Try to use multiple-choice type, not T/F type.
slide-24
SLIDE 24
  • High IF scores, low ID scores
  • Listening item issues: repeated twice, slow/unnatural, presented as a chunk
  • Revise the item: twice -> once, make them more difficult

Listen to the conversation and fill in the blanks. Q L2 IF score L3 IF score ID score 5014 0.89 0.88

  • 0.01

5015 0.83 0.85 0.02 5016 0.76 0.79 0.03

  • 1. The two people are talking _______.

(1) at the airport (2) on the airplane (3) at a café (4) at school

  • 2. Mark is waiting for _______.

(1) his younger brother (2) his friend (3) his younger sister (4) his mom

  • 3. Minji’s friend is arriving at _______.

(1) 4:00 pm (2) 3:50 pm (3) 4:50 pm (4) 3:00 pm

3L3-L5

  • Consider whether you will give listening files once or more.
slide-25
SLIDE 25
  • Low IF scores for negative question format
  • Revise the item to be easier
  • Instruction -> underline/bold/color/highlight?
  • Item -> positive question format

Choose the answer that does NOT fit into the blank.

Q Item L2 IF score L3 IF score ID score 5221 3G5 0.25 0.28 0.03 5222 3G10 0.24 0.27 0.03 5164 3V13 0.37 0.41 0.04 5161 3V10 0.15 0.29 0.14

  • Try not to use negative formats for instructions.
slide-26
SLIDE 26
  • Negative ID scores for negative question format
  • Heritage students tend to have more mistakes(?)
  • Revise the item to be easier
  • Instruction -> underline/bold/color/highlight?
  • Item -> positive question format

Choose the answer that does NOT fit into the blank.

(I met Steve _______ home.)

  • 1. 오러 (in order to come)
  • 2. 오다가 (on my way)
  • 3. 오는 길에 (on my way)
  • 4. 오면서 (on my way)

Q Item Group L3 IF score L4 IF score ID score H+NH 0.34 0.24

  • 0.1

H 0.67 0.31

  • 0.36

5175 4V9

slide-27
SLIDE 27
  • Negative question formats: high IF scores with instructions in English and Korean.

(1) 친구와 내기를 했으니까 이왕이면 이겼어. (2) 가격을 확인하고 나니 먹고픈 생각이 싹 사라졌다. (3) 면접용 정장도 사랴 머리도 하랴 돈이 한두 푼 드는 게 아냐. (4) 그렇게 무리해서 일해 봤댔자 다 헛수고일지도 몰라. (1) 곰곰이 생각해 보니 그 말도 그다지 일리가 있긴 해. (2) 괜히 애매하게 끼어들었다간 큰 코 다치는 수가 있어. (3) 행여나 마감 시간에 늦을까 봐 서둘러 접수를 했다. (4) 막상 일을 그만 접으려니 서운한 감이 없지 않다.

Choose the INCORRECT answer. 다음 중 바르지 않은 문장을 고르시오. Q Item L8 IF score L9 IF score ID score 5381 9V25 0.80 0.64

  • 0.16

Q Item L8 IF score L9 IF score ID score 5380 9V24 0.85 0.79

  • 0.06
slide-28
SLIDE 28

[TIP 4]

It takes effort. Be patient and don’t give up.

  • Choose right reference materials for making test items.
  • Compare test items within and across levels, if necessary.
slide-29
SLIDE 29
  • Compare the similar category items to check the level and revise them accordingly.
  • Delete the items & add new ones (based on 100B materials)

쇠귀에 경 읽기 (In one ear, out the other) 배보다 배꼽이 더 크다 (tail wagging the dog) 수박 겉 핥기 (Scratch the surface) 호랑이도 제 말 하면 온다 (Speak of the devil) 발 없는 말이 천 리를 간다 (Rumors travel far and fast) 남의 떡이 더 커 보인다 (The grass is always greener on the other side of the fence) Q Item SL6 SL7 ID score 5311 7V9 0.45 0.63 0.18 5312 7V10 0.76 0.88 0.12 5313 7V11 0.3 0.63 0.33

  • Choose right reference materials for making test items.
  • Compare items within and across levels.
slide-30
SLIDE 30
  • Compare the similar category items to check the level and revise them accordingly.
  • Delete the items & add new ones (based on K100B & BX materials)

학수고대, 주객전도, 애지중지, 새옹지마, 다다익선, 자업자득, 설상가상, 대기만성, 불가사의 Q Item SL6 SL7 ID score 5308 7V6 0.58 0.78 0.2 5309 7V7 0.05 0.5 0.45 5310 7V8 0.45 0.75 0.3 고진감래, 동병상련, 금시초문, 이구동성, 구사일생, 금상첨화, 동고동락

slide-31
SLIDE 31

[TIP 5]

The subjects’ backgrounds may produce different results.

  • H and NH students showed different results for many test items.
  • Do we need separate placement tests for these two groups?
slide-32
SLIDE 32

콩 (ㅋ: aspirated) 공 (ㄱ: plain) 꽁 (ㄲ: tensed) 통 (ㅌ: aspirated)

  • Non-heritage students have difficulties for learning ㄱ/ㅋ/ㄲ consonants.
  • Pedagogical implication for H vs NH groups
  • ID score: 0.17 -> Keep the item
  • Period: 1st - 3rd weeks -> Actual ID score should be larger

Listen to the Korean syllable and choose the corresponding Korean alphabet letter.

79% 61%

  • H and NH students showed different results for many test items.
slide-33
SLIDE 33

K1B non-heritage (L2) K1BX heritage (L2)

Choose the correct spelling of the word to be inserted in the given blank.

  • 1. 계세요 (to be)
  • 2. 게세요
  • 3. 게새요
  • 4. 계새요

(Where do your parents live?) (In Seoul)

slide-34
SLIDE 34

Q Item Group L4 IF score L5 IF score ID score H 0.52

  • 0.52

NH 0.18 0.67 0.49 5250 5G12

  • Common mistakes by native speakers (ex) 좋은가요/좋나요
  • > Heritage students may have a hard time due to non-standard input
  • Pedagogical implication for H vs NH groups
  • Separate placement tests for H vs NH groups?

(Is it the case that Dongsoo _____ a lot? So, it is the case that he could not ___?)

  • 1. 아픈가요, 오나요 (sick, come)
  • 2. 아픈가요, 오는가요
  • 3. 아프나요, 오는가요
  • 4. 아프나요, 오나요

Choose the answer that best fits into the blank.

slide-35
SLIDE 35

[TIP 6]

Consider why and how you will use the test.

  • When and how will you administer the online placement test?
  • How will you use the test results for actually placing students to each level?
  • Is the test useful for students and teachers? If not, how can it be improved?
slide-36
SLIDE 36

▪ Developed the Online Korean Placement Test using BOLT. ▪ Will be available for the Korean Language Program at UCB, starting from Fall, 2019. ▪ Should be and will be improved. ▪ Thanks to BLC!!! ▪ Q & A

slide-37
SLIDE 37

Bachman, L., & Palmer, A. (1996). Language testing in practice: Designing and developing useful language

  • tests. Oxford, UK: Oxford University Press.

Brown, J. (1996). Testing in language programs. Prentice Hall Regents. Lee, Y. (2000). Designing a Korean Language Placement Test. The Korean Language in America, 4, 181-210. Long, A., Shin, S., Geeslin, K., & Willis, E. (2018). Does the test work? Evaluating a web-based language placement test. Language Learning & Technology, 22 (1), 84-94. Meunier, L. (1994). Computer adaptive language tests (CALT) offer a great potential for functional testing. Yet, why don’t they? CALICO Journal, 11 (4), 23-39. Plakans, L. (2018). Then and Now: Themes in Language Assessment Research. Language Education & Assessment, 1 (1), 3-8. Rapi, L. & Miconi, H. (2014). Theoretical and practical problems in designing tests for foreign language

  • programs. Euroopean Scientific Journal, 10 (29), 156-164.

Roever, C. (2001). Web-based language testing. Language Learning & Technology, 5 (2), 84-94.