In Engineering Classes, How to Assign Possible Scenarios Partial - PowerPoint PPT Presentation

Need to Assign Partial . . . How Partial Credit is . . . For Engineering . . . Analysis of the Problem In Engineering Classes, How to Assign Possible Scenarios Partial Credit: From Current Subjective Estimating Expected . . . Practice to Exact Formulas (Based on Estimating Expected . . . Computational Intelligence Ideas) So How Should We . . . Commonsense . . . Joe Lorkowski 1 , Vladik Kreinovich 1 Home Page and Olga Kosheleva 2 Title Page 1 Department of Computer Science 2 Department of Teacher Education ◭◭ ◮◮ University of Texas at El Paso ◭ ◮ El Paso, TX 79968, USA lorkowski@computer.org, vladik@utep.edu, Page 1 of 23 olgak@utep.edu Go Back Full Screen Close Quit

Need to Assign Partial . . . How Partial Credit is . . . 1. Need to Assign Partial Credit For Engineering . . . • If on a test, a problem is solved correctly, then the Analysis of the Problem student gets full credit. Possible Scenarios Estimating Expected . . . • If the student did not solve this problem at all, the Estimating Expected . . . student gets no credit for this problem. So How Should We . . . • In many cases: Commonsense . . . Home Page – the student correctly performed some steps that leads to the solution, Title Page – but, due to missing steps, still did not get the so- ◭◭ ◮◮ lution. ◭ ◮ • The student is usually given partial credit for this prob- Page 2 of 23 lem. Go Back • The more steps the student performed, the more par- Full Screen tial credit this student gets. Close Quit

Need to Assign Partial . . . How Partial Credit is . . . 2. How Partial Credit is Usually Assigned For Engineering . . . • Usually, partial credit is assigned in a very straightfor- Analysis of the Problem ward way: Possible Scenarios Estimating Expected . . . – if the solution to a problem j requires n j steps, Estimating Expected . . . – and k j < n j of these steps have been correctly per- So How Should We . . . formed, Commonsense . . . – then we assign the fraction k j of the full grade. Home Page n j Title Page • For example: ◭◭ ◮◮ – if a 10-step problem is worth 100 points ◭ ◮ – and a student performed 9 steps correctly, Page 3 of 23 – then this student gets 90 points for this problem. Go Back Full Screen Close Quit

Need to Assign Partial . . . How Partial Credit is . . . 3. For Engineering Education, This Usual Prac- For Engineering . . . tice is Sometimes a Problem Analysis of the Problem • Sometimes, our objective is simply to check intellectual Possible Scenarios progress of a student. Estimating Expected . . . Estimating Expected . . . • In this case, the usual practice of assigning partial credit makes perfect sense. So How Should We . . . Commonsense . . . • However, in engineering education, we want to check Home Page how well a student can handle real engineering tasks. Title Page • Let us consider the case when: ◭◭ ◮◮ – in each of the 10 problems on the test, ◭ ◮ – the student performed 9 stages out of 10. Page 4 of 23 • Then, none of the answers is correct, but the student Go Back gets 10 · 9 = 90 points: “excellent”. Full Screen • This student gets as many points as another students who correctly solved 9 problems. Close Quit

Need to Assign Partial . . . How Partial Credit is . . . 4. Analysis of the Problem For Engineering . . . • Let us imagine that this student has graduated and is Analysis of the Problem working for an engineering company. Possible Scenarios Estimating Expected . . . • To gauge the student’s skills, let’s estimate the benefit Estimating Expected . . . that he/she brings to this company. So How Should We . . . • Let us start with considering a single problem j whose Commonsense . . . solution consists of n j stages. Home Page • Let us assume that the newly hired student can cor- Title Page rectly perform k j out of these n j stages. ◭◭ ◮◮ • This is not a test, this is a real engineering problem, it ◭ ◮ needs to be solved. Page 5 of 23 • So someone else must help to solve the remaining n j − k j Go Back stages. Full Screen Close Quit

Need to Assign Partial . . . How Partial Credit is . . . 5. Possible Scenarios For Engineering . . . • Sometimes, in the company, there are other specialists Analysis of the Problem who can perform the remaining n j − k j stages. Possible Scenarios Estimating Expected . . . • In this case, the internal cost of this additional (un- Estimating Expected . . . planned) help is proportional to the number of stages. So How Should We . . . • Otherwise, we need to hire outside help. Commonsense . . . Home Page • In such a situation, the main part of the cost is usually the hiring itself: Title Page – the cost of bringing in the consultant is usually ◭◭ ◮◮ much higher than ◭ ◮ – the specific cost of the consultant performing the Page 6 of 23 corresponding tasks. Go Back • Thus, in the first approximation, the cost of using a Full Screen consultant does not depend on the number of stages. Close Quit

Need to Assign Partial . . . How Partial Credit is . . . 6. Estimating Expected Loss to a Company: Case For Engineering . . . of a Single Problem Analysis of the Problem • Let c i ( i for “in-house”) denote the cost of performing Possible Scenarios one stage in-house. Estimating Expected . . . Estimating Expected . . . • The resulting in-house cost is equal to c i · ( n j − k j ). So How Should We . . . • Let c h denote the cost of hiring a consultant. Commonsense . . . • Let p denote the probability that there is an in-house Home Page specialist who can perform a given stage. Title Page • It is reasonable to assume that different stages are in- ◭◭ ◮◮ dependent. ◭ ◮ • So the probability that we can find inside help for all Page 7 of 23 n j − k j stages is equal to p n j − k j . Go Back • Thus, the expected loss is equal to Full Screen c i · p n j − k j · ( n j − k j ) + c h · 1 − p n j − k j � � . Close Quit

Need to Assign Partial . . . How Partial Credit is . . . 7. Estimating Expected Loss to a Company: Case For Engineering . . . of Several Problems Analysis of the Problem • Several problems on a test usually simulate different Possible Scenarios engineering situations that may occur in real life. Estimating Expected . . . Estimating Expected . . . • We can use the above formula to estimate the loss So How Should We . . . caused by each of the problems: Commonsense . . . c i · p n j − k j · ( n j − k j ) + c h · � 1 − p n j − k j � . Home Page Title Page • The overall expected loss L can be then computed as the sum of the costs corresponding to all J problems: ◭◭ ◮◮ J ◭ ◮ � c i · p n j − k j · ( n j − k j ) + c h · 1 − p n j − k j �� � � L = . Page 8 of 23 j =1 Go Back Full Screen Close Quit

Need to Assign Partial . . . How Partial Credit is . . . 8. So How Should We Assign Partial Credit For Engineering . . . • Usually, the grade g is counted in such as way that: Analysis of the Problem Possible Scenarios – complete knowledge corresponds to 100 points, and Estimating Expected . . . – a complete lack of knowledge – to 0 points. Estimating Expected . . . • Let L be the loss caused by the student’s lack of skills. So How Should We . . . • Then, we should take g = 100 − c · L for some c . Commonsense . . . Home Page • The complete lack of knowledge is k j = 0 for all j : Title Page J � ( c i · p n j · n + c h · (1 − p n j )) . L = ◭◭ ◮◮ j =1 ◭ ◮ • We want to select c so that 100 − c · L = 100 = 0, thus: Page 9 of 23 J � c i · p n j − k j · ( n j − k j ) + c h · � 1 − p n j − k j �� � Go Back j =1 g = 100 − 100 · . Full Screen J � ( c i · p n j · n j + c h · (1 − p n j )) Close j =1 Quit

Need to Assign Partial . . . How Partial Credit is . . . 9. How to Assign Partial Credit: the Resulting For Engineering . . . Formula Analysis of the Problem • If we divide numerator and denominator by c h , we get Possible Scenarios = c i def a simpler formula in terms of c ′ . Estimating Expected . . . i c h Estimating Expected . . . • A student who, for each j -th problems out of J , per- So How Should We . . . formed k j out of n j stages, get the grade Commonsense . . . Home Page J � � 1 − p n j − k j �� � i · p n j − k j · ( n j − k j ) + c ′ Title Page j =1 g = 100 − 100 · . J ◭◭ ◮◮ � ( c i · p n j · n j + c h · (1 − p n j )) j =1 ◭ ◮ • Here c ′ i is the ratio of an in-house cost of performing a Page 10 of 23 stage to the cost of hiring an outside consultant. Go Back Full Screen Close Quit