Using Open-Ended Needs Assessment Data for Improvement Charlie Goss, Analytics Specialist Office of Institutional Effectiveness: Liberty University csgoss@liberty.edu 1
Participants will be able to… • Develop an open-ended needs assessment for the purpose of gathering actionable feedback. • Analyze the responses using an established method. • Package the analysis results in a way that facilitates end-user review as well as development of improvements. 2
Session Outline • Development of Needs Assessment • Survey Launch • Content Analysis • Dissemination and Improvements • Activity 3
What is a needs assessment? Sample definition: • – “a systematic, rational process for collecting and analyzing information to describe the needs of a population” 1 Often done in education, health care, and counseling • Examining the gap between “is” and “ought to be” • Eliciting current and future needs • Quantitative or open-ended • A robust indirect measure; no rubrics involved • 1-New York State OASAS, “OASAS Prevention Program – Needs Assessment” (October 2015), 4 https://www.oasas.ny.gov/prevention/needs/needsassessment.cfm
Liberty’s Needs Assessment Goal: obtain list of needs to be addressed and facilitate • improvement plans Ask students directly what improvements they want • No “positive” information sought; only opportunities • Specific feedback not confined to preformatted options • Collection of stated needs (known and new) • Built to be acted upon, not just to show adequacy and • satisfaction 5
Open-Ended Surveys Pros: Cons: Rich, nuanced data Is a labor-intensive process, • • not fully automated Respondent can be more • specific than selecting Takes time to analyze • standard response options Can be difficult to organize like “Strongly Agree” • into themes (meaningful May get unexpected chunks) • answers and gain new Must separate the wheat • insights from the chaff (lots of Reading the review vs. just useless responses) • looking at the number of stars 6
Building the Needs Assessment • 13 questions developed; each student received 1. – List up to three services you would like to see added to the university. – List up to three problems that you most frequently encounter in your courses. • Question types: Problems (4), Facilities (2), Services (3), Student Development (2), Future Needs (1), Skills (1) • Branded as a one-question survey to increase response rate • Sent to all Fall 2014 residential and online students (about 1,000-5,000 per question) 7
Backstage Survey Settings • Number of characters limited to 50 – Good : “not knowing what kind of notes to take” (38) – Too Long : “I think students should be allowed to have one skip per week. Also, there should be fewer limits on dining options on campus. I am always starving in between classes.” (165) • Three separate response fields; only one required – Mary Smith response 1: “Internet is slow” – Mary Smith response 2: “parking is limited” – Mary Smith response 3: “I want more dining options” • Easier analysis, one theme per response 8
Timeline and Response Rate Timeline • Launched 9/15/14 (Monday) • Closed 9/23/14 (Tuesday) • Elapsed time: 9 days Response Rate Resident students 12.6% (1,721 / 13,611) Online students 10.2% (3,893 / 38,350) TOTAL 10.8% (5,614 / 51,961) 9
Response Metadata • Total Responses: 13,811 • Responses per Question: between 300 and 1,600 • Average Response Length: 32 characters – E.g., “getting in touch with an advisor” • Average Responses per Student: 2.7 out of 3 10
Overview of Analysis Process 1. All 13 response sets split by delivery mode into 23 files 2. Two reviewers for increased reliability and efficiency 3. Incorporating themes across all files 4. Stacking of all 23 files with responses and themes 5. Sub-Themes developed 6. Results used for residential and online action plans Total Time: 3 weeks at 8 man-hours per day 11
Software Toolkit SPSS Text Analytics Excel *Readying the *First-pass *Used for detailed raw data to be automated word reading of all analyzed search responses *Stacking all *Themes manually themes together assigned at the end *Final spreadsheet 12
Transposing Raw Data Stud. ID Student’s First Response Student’s Second Response Student’s Third Response Finding work study 05 Post office accessibility Nothing else really. opportunities. Not being allowed to drive The bus system making me late 06 Strict curfew hours my car on campus for classes One row per response allowed each response to receive its own theme later. Stud. ID Resp. Num. Responses 05 1 Finding work study opportunities. 05 2 Post office accessibility 05 3 Nothing else really. Not being allowed to drive my car on 06 1 campus 06 2 The bus system making me late for classes 06 3 Strict curfew hours 13
Content Analysis • Perusing all responses, searching for common themes • Themes=repeating needs and ideas grouped together into an organized whole (e.g., Advising or Financial Aid) • Fake examples below: Response Theme More spaces in parking garage Parking Docs Diner has a lot of people Dining Dorm internet speed is lacking Internet 14
Text Analytics: Themes and Keywords Advising - Text Match: Course Assignments - Text Match: adviser discussion board advise reading advising quiz advisor Course Resources - Text Match: Blackboard - Text Match: course schedule blackboard course chart black board syllabus bb study guide Books - Text Match: course preparation text text book mbs Any responses with a text match were book assigned to the corresponding theme. False hits were okay at this stage. 15
First-pass Detailed look and Analysis Sample word search theme selection (automatic) (manual) Individual Responses First Theme Final Theme ASIST-everything about it IT streamline the portal login too many IT list/links ipad compatibility with liberty and Blackboard Phones / Devices blackboard Online outreach Communication I am an online student so cannot think of Irrelevant anything Automated calling. Too many calls for same Fewer Outreaches to thing Student Constant FA emails with same information Fin Aid Matters Finance More Courses / Opening a program for Architecture. Programs Tutoring for online students its there but Tutoring Tutoring where? everything else is great! Irrelevant Online advising. Advising Advising Mobile blackboard Blackboard Phones / Devices Online interaction and timeliness More Community Connecting with Others 16 Text selection Books Bookstore
Two-Reviewer Theme Consolidation Reviewer 1 Reviewer 2 Final Theme Example Tasks Theme Theme in Both (Residential) (Online) Response Sets Convert similar Quicker Student Quicker themes to the same Service Services Service theme Assign new themes Answers to Answers to one reviewer may Tests Tests have overseen Note specific themes Parking for each delivery Parking (residential mode only) 17
Forward-Theme Assignment • Applied themes forward that had been assigned in earlier analyses to ensure they were not forgotten later Q1 Res Q7 Res Q7 Res Financial Aid Financial Aid Financial Aid Parking Parking Parking Dormitories Dormitories Dormitories Faculty Faculty Faculty Academics Academics • Ensured consistency of themes across all 23 analyses 18
Reverse-Theme Assignment • Assigned themes found in later analyses to earlier response sets Q7 Res Q1 Res Q1 Res Financial Aid Financial Aid Finance Parking Parking Parking Bookstore Bookstore Books Books Books Faculty Faculty Faculty • Ensured consistency of themes across all 23 analyses 19
Development of Sub-Themes Task Before After Themes with many Skills - Personal responses broken Skills - Personal Ethics (7) down into sub- (299) Goal-Setting (6) themes Organizing Skills (20) Themes with few responses Student Affairs Student subsumed as sub- Student Housing (41) Housing (41) themes within new Student Care (18) themes 20
Stacking of Sub-Themes • From 23 themed files to 1 large file Stacked File Athletics Bookstore Q1 Res Q2 Res Bus Issues Bus Issues Dormitories Disc. Board Parking Parking Internet Exercise Dormitories Facilities Athletics Exercise … Facilities Internet Parking Q13 Res Q13 Onl Tests Tests Internet Textbooks Disc. Board Textbooks Website Website Disc. Board Bookstore Writ. Ctr. Writ. Ctr. 21
Descriptive Statistics • Minor cleanup of themes and sub-themes – E.g., ‘Registar’ changed to ‘Registrar’ to match others • Frequencies and crosstabs on themed data using SPSS • Responses and themes overall, by delivery mode • Some sub-themes not included in final spreadsheet – Irrelevant (did not answer question sensibly – “n/a,” etc.) – Sensitive (mentioned specific professor) – Duplicate (submitted same answer two or three times) 22
Snapshot of Findings • Themes: 32 • Sub-Themes: 232 • Pertinent Responses: 12,172 • Example of Theme and Sub-Themes: Response Theme Sub-Theme Bus routes remaining the same Transportation Bus Issues Traffic being slow. Transportation Traffic Delays Stoplights at main intersections Transportation Transportation - Other 23
Recommend
More recommend
