CMSC 20370/30370 Winter 2020 Evaluation – Quantitative Methods Case Study: Accessibility Jan 22, 2020
Quiz Time (5-7 minutes). Quiz on Sound Awareness for Deaf and Hard of Hearing Users Principles of Good Design
Administrivia • IA2 due on Friday • IA2 rubric on Piazza and course website • Tentative schedule for group proposal presentations will be posted on Friday • More information on presentation format provided on Friday – Everyone is expected to attend each session – Everyone is expected to present in at least part of the presentation
Today’s Agenda • Evaluating your design/prototype/system – Usability testing – Inspection methods – Qualitative techniques
USER-CENTERED DESIGN DESIGN/PROTOTYPE IMPLEMENT USER NEEDS EVALUATE
Case Study: Sound Awareness • Deaf and hard of hearing users • Based on interviews with 12 DHH users in Study 1 • In Study 2, conducted Wizard of Oz lab study with 10 DHH users and three initial sound awareness prototypes • Recommendations for sound awareness for DHH – E.g. integrate into daily routines – Shared space – Uncertainty – Form factors • Limitations
From Dhruv Jain’s website
From Dhruv Jain’s website
What home sound awareness do DHH users desire ?
How do DHH users react to sound awareness prototypes?
These questions require empirical evaluation – testing with real users
We could run experiments!
Case Study: Sound Awareness Setup • Used faculty and student lounge to look like a home (had kitchen area, bathroom, dining room, lounge area, and windows to outside) • 1 hour session • Background questionnaire • Initial prototype demos • Thematic scenarios • Semi-structured interview
From Dhruv Jain’s website
From Dhruv Jain’s website
Initial demo • 3 sets of everyday actions – Starts microwave and does dishes – Knocks on door, opens, greets, sits down, door closes – Makes coffee, pours liquid, bird chirs
Thematic scenarios • Bathroom scenario (privacy) • Babysitter scenario (activity tracking) • Movie scenario (information overload) • Each designed to gather feedback on specific aspect for design • Also looked at uncertainty using mockups
Post-study • Semi-structured interview • Data analysis – thematic coding • Design implications and discussion
From Lazar, Research Methods in HCI
What does that mean? • In HCI, lab studies are not always strictly experiments – E.g. the case study is a lab study but not a controlled experiment – In these cases, there is no null hypothesis or alternative hypothesis – Instead it is exploratory – Still useful for inclusive technology – why?
In Class Exercise • How would we turn this into a lab experiment assuming we had 2 systems – An existing sound awareness prototype – Our sound awareness prototype
Hypothesis • Precise problem statement that can be tested with empirical investigation • Our prototype will improve sound awareness for DHH users in the home
Control? • If you’re claiming your system improves over the current state of the art, you have to benchmark against it • Or benchmark against having no intervention to show your system makes a difference • Control could be: no sound awareness condition
Sample Size? • In HCI, people run sometimes statistical tests with very small sample sizes • However, statistical power increases with sample size • Use calculator • Depends on resource, no of conditions e.g. Control, existing system, prototype – Three conditions, 10 users each, 30 users total
Within or Between Subjects? • Within – each user experiences all conditions – Each user tries control, existing system, and sound awareness system, • Between – each user in only 1 condition, compared against different users in other conditions – Users either try control, existing system, or sound awareness • Pros and Cons?
Randomization? • Avoid Priming Effects – Exposure to one stimulus influences response to another stimulus • Avoid Practice Effects – Might perform better in within subjects if by third condition you are more familiar with the system • Assign to different conditions
Dependent and Independent Variables? • Dependent variable is what you’re measuring – how accurately can users locate or identify sounds or become aware of sounds • Independent variable is what you are alternating e.g. type of system
Confounding factors? • Factors influencing dependent and independent variables – Age, DHH spectrum, environment is not like familiar home
Designing Experiments • Hypothesis • Control • Sample Size • Within Subjects vs Between Subjects • Randomization – Avoid Priming Effects – Avoid Practice Effects – Assign to different conditions • Dependent + Independent variables • Confounding factors
Hawthorne Effect • Participants may behave differently in lab- based experiments due to being observed or rewards for participation • “Hawthorne Effect” • Landsberger 1958 – study of workers caused improvement in worker productivity that slumped when study ended • Assumed to be because of motivation from being observed and interest in them
HCI and Stats 101 Significance testing – checking is observed difference occurring by • chance? Compare means of 2 groups • – T-tests, ANOVA (between or within group tests or both) Identify relationships • – Correlation (2 variables) – Regression (1 dependent variable and multiple independent variables) Non-parametric measures for categorical or ordinal data etc • – CHI-squared – relationship between variables – Mann-Whitney – between groups – Wilcoxon signed-rank – within groups – Kruskal-Wallis one way ANOVA – three or more sets of data – Friedmans two ANOVA
Experiments: Pros and Cons?
Other ways to evaluate systems empirically • Many HCI studies use Amazon Mechanical Turk or Prolific – Not as well suited for many underserved or marginalized users – May require more information about context of use/user in context – may not be target users
Case Study: HomeSound @ CHI 2020
Summary • In HCI, we use both controlled and laboratory studies • Need to consider study design and goals carefully • Need to use right sample size and statistical analysis
Coming up next class • Project team discussions • Come to class – Will share tentative presentation schedule and information about presentation format – Ensure that your group checks in with one of the TAs on your project progress – TAs have a short checkpoint form – Q&A with TAs • Turn in IA 2
Get in touch: Office hours: Fridays 2-4pm (Sign up in advance) or by appointment JCL 355 Email: marshini@uchicago.edu
Recommend
More recommend
