Alumni Survey: Three conceptualizations to alumni research Alberto F. Cabrera Professor & WISCAPE Sr. Researcher E-mail: cabrera@education.wisc.edu Métodos de análisis de David J. Weerts la inserción laboral de los universitarios Project Director& WISCAPE Research Associate E-mail: dweerts@education.wisc.edu & Universidad de León Bradford J. Zulick 9- 11 de Junio de 2003 Doctoral Student & Graduate Research Assistant E-mail: bzulick@wisc.edu University of Wisconsin- Madison 1
Topics 1. Approaches to alumni assessment – Conceptualizations – Policy questions – Methodological considerations 2. Common methodological issues 3. Conclusions & recommendations 2
3
Growth of Alumni Surveys in the United States Alumni Surveys 1979 – present 15% 15% 70% Outcome Engagement/Competency Giving 4
I. Outcomes Approach 1) Oldest Approach – Used since 1937 (Pace, 1979) 2) Assumption – Institutional quality and effectiveness can be appraised by what alumni have accomplished 3) Policy Questions 1) How satisfied are graduates with their employment? 2) How satisfied are the graduates with the degree granting institution? 3) To what extent are graduates fully participating in civic activities? 5
Why stress job outcomes? Most perspective students rate getting a good job as important in selecting an institution 6
Top 6 reasons when selecting a college Top 6 reasons when selecting a college (281,064 students at 421 baccalaureate colleges & universities) 1. Academic Reputation (56%) 2. Graduates getting good jobs (51%) 3. Size of the institution (35%) 4. Financial Assistance (33%) 5. Admission to top graduate schools (30%) 6. School’s social activities (28%) Source: HERI Fall 2001 Survey 7
Outcomes Approach 4) Forces – Accreditation – State Mandates – Social Justice Issues (Shape of the River) 5) Measures – Job Satisfaction – Satisfaction with the Institution – Income – Occupational Status – Participation in civic activities/organizations 8
9
Civic Engagement 10
II. Engagement & Competencies Approach • Assumption – Student engagement in college and attainment of competencies while attending college are the best predictors of alumni success. • General advice – Align assessment efforts with what research shows matters in a student’s development, learning and attainment of competencies (e.g. Pascarella, 2002; Kuh 2001; Roberson, Carnes & Vice, 2002). 11
Policy Questions 1. What are the competencies (i.e., outcomes, abilities & values) that college education should foster most? 2. To what extent are alumni engaged with faculty/staff/peers while attending college? 3. To what extent colleges and universities engage students on those learning activities most prone to produce critical competencies? 4. To what extent graduates apply those competencies in the job or in graduate school? 12
Measures of Engagement • Active & collaborative learning • Interactions with faculty & staff • Educational experiences • Campus climate • Enriching educational experiences • Classroom practices 13
Measures of competencies ACT College Outcomes Survey • Gains in problem solving & group skills • Critical thinking • Practical Competence • Cognitive Complexity • Knowledge Acquisition & Application • Preparation for career 14
The Extent to Which 35 Abilities and Skills Were Enhanced by the Undergraduate Program at Albany (N=1775 Alumni) Average Response (Scale of 1 to 5) 3.9 Exercise personal responsibility 70 8 3.9 Maintain openness to new ideas 8 68 3.8 8 66 Think analytically & logically Function independently 9 3.8 66 Learn how to learn 64 17 3.8 63 Exercise self-discipline 11 3.8 59 3.7 10 Acquire new skills & knowledge 3.5 53 Adapt to social situations 17 50 Listen effectively 3.5 15 50 Understand myself 3.4 17 49 3.4 Exercise problem solving skills 16 47 Write effectively 3.4 17 43 18 Possess clear goals 3.3 43 23 3.3 Speak effectively 43 Carry out research 26 3.2 41 3.2 27 Relate to people of different races 41 3.2 29 Know literature of recognized merit 0 20 40 60 80 100 100 80 60 40 20 0 Percent responding 1 or 2(low) Percent responding 4 or 5(high) 15 on a 5 point scale on a 5 point scale Source: Volkwein (2002)
The Extent to Which 35 Abilities and Skills Were Enhanced by the Undergraduate Program at Albany (N=1775 Alumni) Average Response (Scale of 1 to 5) Evaluate and choose among alternatives 3.2 11 39 3.1 Understand cultural differences 34 39 3.1 27 38 Be effective team member 33 3.0 37 Produce scholarship 3.0 Analyze literature critically 35 37 3.0 36 37 Understand scientific findings 3.1 Cope with conflict 28 36 2.9 36 34 Appreciate artistic expression 2.9 37 34 Use statistics Place problems in historical perspective 3.0 35 32 2.8 41 32 Understand math concepts 2.9 37 31 Take moral & ethical action 2.8 42 31 Employ scientific methods 38 30 2.9 Lead and supervise others 2.7 30 44 Solve math problems 2.7 29 45 Analyze scientific data 2.4 27 58 Use foreign language skills 2.2 19 61 Utilize computing skills 100 80 60 40 20 0 0 20 40 60 80 100 Percent responding 1 or 2(low) Percent responding 4 or 5(high) 16 on a 5 point scale on a 5 point scale Source: Volkwein (2002)
Approaches in identifying competencies • Focus groups with experts • Content Validation approach 1. Define a universe of competencies. Do a comprehensive job analyses of those jobs graduates are most likely to hold. Single out those skills, knowledge and abilities each task involves. Then, identify those competencies that can be taught in college from those that are learned on the job. Pay particular attention to those contexts where those competencies are most likely to be effective. 2. Develop and adopt assessment measures. Make certain that your assessment tool reflects the competencies that enable graduates succeed on the job under a variety of relevant contexts 3. Validate assessment measures with experts. Pilot test the instrument with graduates, job holders, employers and other experts. 17
Competencies & Skills for Business Program Competency dimension Skill group Leadership influence Vision & strategic agility, action oriented, costumer focus, building teamwork & political savvy Implementation Planning, organizing, time management, processes management, delegating, directing others Problem solving & decision Collecting information, analyzing root causes, making developing alternatives, making effective decisions Communication Written & oral communication, presentation skills, informing others Interpersonal skills Approachability, composure, interpersonal savvy, negotiating, managing peer relationships, conflict management Coaching & facilitating Coaching others, developing others, motivating others Integrity & character Honesty & trust, ethics & social responsibility Source: Roberson, M. T., Carners, L. W. & Vice, J. P. (2002). Defining and measuring student competencies: A content validation approach for Business program outcome assessment. Delta Phi Epsilon Journal , 18 XLIV (1), p. 17.
Forces pushing for adoption of engagement & competency approach 1. Private funding organizations 2. Accreditation 3. State regulations 19
Good teaching practices as an alternative to rankings (Chronicle, October 22, 1999) “ Unless we develop measures of quality where colleges can provide evidence of their contribution to student, then this whole system [of ranking colleges] turns on resources and reputation, and reinforces the elitism of higher education Russell Edgerton. Pew Charitable Fund 20
How regional & professional accreditors are changing • • Middle States Association of Colleges & Schools: Middle States Association of Colleges & Schools: – Revising standards to focus on student learning and other outcomes such as the quality of student services. – 1996 Middle States: Framework for Outcomes Assessment: • “The ultimate goal of outcomes assessment is the improvement of teaching and learning ” • • New England Association of Schools & Colleges: New England Association of Schools & Colleges: – Developing better ways for colleges to measure student learning. • North Central Association of Colleges & Schools : • North Central Association of Colleges & Schools – Drafted revisions in eligibility requirements to require evidence of student achievement and institutional effectiveness. Changes must still be voted on by the group. Revisions of standards to begin this fall. Based on The Chronicle of Higher Education (July 7, 00) A29 21
Recommend
More recommend
