youth pqa external methods? What might create an obstacle to - PDF document

Before we Assess Check your bias!  What triggers your judgment?  What styles of instruction do you tend to like or not like? What about youth pqa external methods?  What might create an obstacle to assessor reliability your being able to be objective? training Remember, the goal is to collect Training Tips Slides objective data ! Trust the tool to do the assessment. David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training The Role of an External Assessor Overall Program Quality  You are here to be a reliable rater.  This means your scores provide a touchstone or point of reference for self-assessment scores.  Being reliable means you have a more complete understanding of the language and intent of each item in the tool.  Set aside your biases. Recognize what you are and are not scoring — see next slide.  Score the tool as defined and written. David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training Tips for Completing and Scoring the PQA Quality Continuum for an Individual Item  Use the Observation Guide when you observe; it lists every item and will help trigger your memory, as well as remind you to ask follow up questions when needed.  Always try to see multiple items in every interaction and cross-reference constantly.  Look to a preponderance of evidence but favor higher scores; give programs the benefit of the doubt . Remember: NO 2 ’ s or 4 ’ s.  Fill in the front pages! All the site data is necessary: YOU are part of our growing sample! David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training 1

More Tips on Scoring the PQA Tips for These Scales  Only ask follow-up questions for items with  Nourishment : Water fountain in hallway OK. question(s) in the evidence column. Ask the  Emergency Preparedness: Only score additional safety questions as they are printed. equipment if safety equipment is necessary for activity.  Anecdotal evidence needs to be entered for every  Emergency Preparedness: Only score outdoor program row in order to score. space if outdoor space is present and utilized.  Emergency Preparedness: If session is completely  Score an X if specified items or scales are not outdoors, mark X for supervised indoor entrances. to be scored . (See notes for which items/scales  Emergency Preparedness: Note, if the program is outside, this is allowed — i.e., conflict scale if no conflict the emergency procedures and fire extinguisher may be present.) inside — score 3 if it is accessible.  Read each level of the item carefully. Use  Emergency Preparedness: Doors locked from outside do handbook and Frequently Confusing Items sheet! not need to be monitored in the same ways as a main door. David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training Tips for Emotional Safety Tips for Warm Welcome  “ Warm and Respectful ” and “ Positive Body  This scale is one of the most potentially subjective. Language ” are potentially subjective items.  Collect as much evidence as possible and assess  Look for evidence of disrespect, sarcasm, insults, the overall climate . dismissive hand gestures, etc.  Observe both youth and staff in order to collect  A wide variety of styles can still be assessed as a “ 5. ” evidence of:  Item 1: Opening “ hurrah ” does not count as a  Interactions between youth and staff. greeting. With a greeting, youth presence is  Interactions among youth. acknowledged, as individuals or a group.  Individual youth behavior. David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training Tips for Session Flow Tips for Active Engagement  “ Active Learning ” is not necessarily “ hands-on ” learning.  Item 2: Materials simply need to be in the room and  Item 2: “ Talking about what they are doing ” must be a easily accessible for them to be “ ready. ” structured opportunity, not something that just happens.  They are NOT ready if staff or youth have to leave the  Item 3: Abstract and concrete should be complementary. room to get them.  Pay attention to the purpose of the workshop or activity you are observing.  Item 4: To assess clear explanations and  Item 4: Tangible products are things you can see, feel or appropriate time, observe how youth react to staff: hear. Simply learning a skill is not a tangible product. A  What do they do after instructions are given? performance is presenting or demonstrating a skill or product to an audience.  What do they say or do to indicate whether they feel rushed, bored, or the pace is just right?  Item 5: Use evidence from activities observed rather than activities that are planned. David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training 2

Tips for Skill Building Tips for Encouragement  Most of this scale relies on a specific skill or set of  Item 1: Evaluative language is not “ bad ” and can be (in fact it is often) present in a session that is assessed with a “ 5. ” skills being addressed in the session.  Listen for staff to follow-up an evaluative statement with specific,  Item 1: What youth are going to “ do ” is not non-evaluative language. necessarily the same as what they will “ learn. ”  Item 2: Take notes on all of the questions you hear.  Note how staff frames learning objective. Determine whether or not they were open-ended when you go to score the tool.  Items 2 & 5: Observing youth behavior will give you  Item 3: Look for staff to be an active part of what happens in clues about whether or not they are being the session. challenged to improve and/or struggling.  This can take on a lot of different forms (e.g., giving instructions,  Note staff responses to wrong answers and errors. working alongside, circulating, etc.).  Observe staff interactions with youth to assess the scale. David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training Tips for Reframing Conflict Tips for Belonging  When a conflict occurs, pay close attention to how  Item 1: Look for structured opportunities. staff reacts.  Item 2: Carefully observe youth interactions with  Observe as much of the interaction between staff each other. and youth as possible.  Observe complete interactions whenever possible.  Item 3 can be a difficult item for programs to achieve a high assessment score. David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training Collaboration Tips for Collaboration  Item 1: Look for opportunities for youth to work with  These are 3 ways to promote collaboration. other youth.  Often they work together, encouraging deeper forms of true  Item 2: “ Interdependent Roles ” simply means that collaboration. the activity wouldn ’ t work individually.  A program may have any combination of these elements.  Doesn ’ t always have to be explicitly assigned.  Item 3: “ Shared goals ” are mutual or joint goals. David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training 3