Rubrics Outcomes By the end of this segment, you will be able to: 1. - PDF document

Rubrics Outcomes By the end of this segment, you will be able to: 1. Describe the purposes and uses of rubrics 2. Recognize different types of rubrics 3. Evaluate rubrics for impact on student learning 4. Understand reliable application of rubrics in assessment 1 37

results, -+ 1) Design Assessment: S) ln~rpret Goals, out comes, identify, and implement ev i dence, criteria and revisions to pedagogv. standards (ex. r ubrics) curriculum, programs, crite:ria or outcomes. \ Using I Rubrics 4) Co llect, rev'. iew 2) Publi cl y share outcomes, and analyze evidence of criteria and standards student learning 3) Provide ' int en ti ona l l earn i ng experiences (curric lA um & pedagogy) Rubrics - What are they and why use them? A rubr ic is a scoring guide: a list or chart that describes criteria used to evaluate or grade student work . 1 suskie, 20091 Rubrics contain a set of criteria specifying the characteristics of a learning outcome and the levels of achievement for each characteristic. !L evv, 20121 There is no single way to write or format rubrics - they can be created and adapted for the circumstances and situations of your courses and programs. 38

can rubrics be used to evaluate? Performances or behaviors: Written or visual student work: 0 Presentati on 0 Teamwork 0 Papers 0 Journals 0 Role plays 0 Artwork 0 Performances 0 Portfolios Rubric Strengths Compl ex products or behaviors ca n be exam· in ed eff iciently and effectivel y. Developing a rubric hel ps to precisely define facul ty expectations. St ude nt appreciate clari ty in expectations fo r their wo rk and/o r behaviors. Rubrics can serve a variety of purposes: Provide formative feedback to students Grade student work Conduct assessment at the program level Rubrics a re criterion-referenced rather than norm-referenced. 1 39

Rubrics and Learning Outcomes Learning outcomes describe what s tu dents will do to de monstrate their learning The r ubr ic desc ribes: The expected properti es of that demonstrati on (c r it eri a) The possible l eve ls of ach ie vement/perfor ma n ce (s tandar ds) Types of Rubrics Holistic Describe how one global, ho listic judgment is made; provides one score fo r a product or behavior. Checklist and rat ing scales are types of holistic rubr i cs. Anal yt ic Involves a ser i es of judgments, each assessing a characte ri stic of the product being evaluated; pr ov ides separ ate, hol i stic scor ing of specified character istics of a product or behavior.

Typical Four-Point Rubric Levels 1. Below Expectat ions 2. Needs Improvement 3. Meets Expectations 4. Exceeds Expectations Example: Holistic Rubric for Assessing Student Essays Standards Criteria The essay has at least one se r io us weakn ess . It may unfocused, underdeveloped, or Inadequate rambling. Problems with the use of language seriously in terfere with the reader's abil ity to understand what is being communicated. De ve lo ping competence The essay may be somewhat unfo cused, underdeveloped. or rambling, but it does have some coher ence . Problems wit h the use of language occasionally interfere with the reader's ab ility to understand what is being co m mu ni ca ted. Acce ptable The essay is gene ra lly focused and contains some development of i de a, but the d isc usstOn mav be simplistic or re petitive. Th e la nguage l acks syntactic comple )( ity and may contain occas ional g ra mmatical errors. but the reader is able to understand what is bei ng communi ca ted. Soph istk at ed The essay is focused and clear ly organi zed, and it shows depth of development. The language is prec ise and shows svntaaic variety. a nd i deas are clearly communk:at ed to the reader. AU en , M. J. {2004). Assessing Academ ic Programs in Hi gtier Educat ion . San Francisco, CA: An ker . Pa ge 139 141

Examp l e: Ana lytic Rub ri c fo r Pee r Assessment of Team Project Members Below Expectation Exceptional Good Proj ect Co ntribu ti ons Made few substantive Con tri buted a "fai r Con tribu te d contributions to t he sh are" of substance to considerable su bstan ce team's final pr od uct the team's fi nal product to t he team's fi na l pr od uct Leadership Rarely or never Accepted a " fa ir shar e" Ro ut in ely provi ded exe rcised leadership of leadership excellent leadership responsibilities Collaboration Undermined gr oup Res pected others' Res pected ot he rs' di scuss ions or ofte n op inions and opinions and m ade fa iled to pa rt icipate co n trib uted to t he major contrib uti ons to group's discu ss ion the grou p's dis cu ss i on Al l en , M . J. (2004). Assessi ng Academic Programs in H'igher Educa ti on. San F rancisco, CA : Anker . Page 139 Reliability in the Use of Rubrics In ter -Rater Rel iability Correlation Betwee n Pa ired Readers Discrepancy Index 1 42

Preparing for Assessment with Rubrics Collect a range of samples ,of student work for use in assessment and remove identifying information Develop and pilot test the rubric Select exemplars of weak, acceptable, and strong student work Reviewing Application of the Rubric Have reviewers apply the rubric and develop a shared understanding of the criteria and standards. Piloting the rubric will reduce the likelihood of discrepancy before scoring - this is calibrating the rubric. Discuss reasons for the assignment of scores. What are the similarities and differences? Is a shared agreement possible? For discrepancies, ask a th ird rater to score. Revise the rubric as necessary to clarify. 1 43

Resu lts ... Summari ze how frequently each level of performance was observed by reviewers. Example: number of student papers that were determined to be: 0 Inadequate 0 Developi ng Competence 0 Acceptable 0 Sophisticated Dra wing Conclusions Demonst ra ti on of learning outcomes: 0 Are you satisfied 0 H ow do you know? 0 If not, what might you do? To draw conclusions about student success, there needs to be a desired st andard {or l evel of compe t ency, or benchmark). 0 Example: 80% of st udent papers will be at the acceptable or sophisticated level. 1 44

Drawing Conclusions - Next Steps How useful is the rubric? 0 Does it work well? ~Could it be improved? 0 How? Data Collection: 0 Did it work well? ~Could it be improved? 0 How? Developing and maintaining the instrument Identify or develop the rubric and shape it to the contours of the assignment. Env ision possible results. P ilot the rubric by applying it to example work. Share the rubric with students to understand how they interpret it. Revise .and continue to refine the rubric to increase inter-rater reliability and usefulness to students. 1 45