Considering Qualitative Evaluation University of Calgary: Sheelagh Carpendale, Alice Thudt, Jo Vermeulen, Jagoda Walny University of St. Andrews: Uta Hinrichs Cork Institute of Technology: Trevor Hogan Tableau: Melanie Tory
Overview Talk 1: Setting the stage: qualitative Talk 2: Observation as a skill Activity 1: Practicing observation Activity 2: Interviewing 1 Activity 3: Interviewing 2 Talk 3: Interviewing as a skill Activity 4: Interviewing 3 Panel: open discussion Talk 4: In summary – next step analysis
Choosing an evaluative approach start with a question relate the question to existing research in contrast in agreement as extension or variation choose a methodology
Goals of Empirical Research PRECISION REALISM GENERALIZABILITY McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)
Goals of Empirical Research PRECISION REALISM GENERALIZABILITY results can apply to other people or situations McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)
Goals of Empirical Research PRECISION REALISM control of factors that were not studied GENERALIZABILITY McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)
Goals of Empirical Research PRECISION REALISM context of study is like the context of use GENERALIZABILITY McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)
obtrusive unobtrusive PRECISION lab experimental experiment simulation abstract field judgment study experiment concrete sample survey field study REALISM formal computer theory simulation GENERALIZABILITY McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)
today focus is on realism and qualitative data gathering methods
Albert Einstein ‘ Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted ’
Qualitative Research • study aspects that can not necessarily be counted • rich in-depth understanding • considers the interplay among factors • helps investigate open-ended questions • can provide insights beyond the expected • can generate design criteria and more specific hypotheses Lofland, J., Lofland, L.: Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Wadsworth Publishing Company, CA, USA (1995) Seidman, I.: Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences. Teachers’ College Press, New York (1998)
Qualitative Data • primary methods for gathering data: – observations – interviews • types of data records: – field notes – journal entries – artefacts (e.g. sketches, diagrams, annotations) – audio & video tapes – screen captures – computer logs – … Lofland, J., Lofland, L.: Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Wadsworth Publishing Company, CA, USA (1995) Seidman, I.: Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences. Teachers’ College Press, New York (1998)
Data Collection as a Skill • qualitative data collection requires skill and sensitivity • observation and interviewing skills can develop with practice and can be learnt • follows rigorous methods Lofland, J., Lofland, L.: Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Wadsworth Publishing Company, CA, USA (1995) Seidman, I.: Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences. Teachers’ College Press, New York (1998)
Qualitative Methodologies • primarily qualitative methodologies (focus of this tutorial) • mixed methodologies : qualitative methods in conjunction with quantitative methods • heuristic inspection methodologies (not discussed here)
Qualitative Methods as Primary • develop a rich in-depth understanding • used at any time in the development life cycle – finished visualization or prototype assessed e.g. via field studies – during the design process – formative studies to inform the initial design • variety of methods to choose from – in-situ observations – immersive observations – laboratory observational studies – contextual interviews – …
Mixed Methods • Qualitative methods can be used in conjunction with quantitative methods to: – provide context for quantitative results – gather participants opinions and preferences – raise new questions – aid the interpretation of quantitative results • Common mixed methods approaches: – experimenter observations – think aloud protocol – collecting opinions (e.g. through interviews or Likert scales) Gorard, S.: Combining Methods in Educational Research. McGraw-Hill, New York (2004)
Characteristics of Qualitative Methods • involved and rigorous process that requires practice – gathering data and rich note taking – transcription – in-depth analysis Strauss, A.L., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, London (1998) Auerbach, C.: Qualitative Data: An Introduction to Coding and Analysis. University Press, New York (2003)
Characteristics of Qualitative Methods • sample sizes can differ from quantitative experiments – not concerned with making statistically significant statements – focussed on ‘saturation’ – can be determined during the study U. Hinrichs, S. Forlini, and B. Moynihan. Speculative Practices: Utilizing InfoVis to Explore Untapped Literary Collections. IEEE Transactions on Visualizations and Computer Graphics, 22(1), 2016. C. Neustaedter and P. Sengers. Autobiographical Design in HCI Research: Designing and Learning through Use-It-Yourself. In Proc. of DIS, pages 514–523, 2012. Strauss, A.L., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, London (1998) Auerbach, C.: Qualitative Data: An Introduction to Coding and Analysis. University Press, New York (2003)
Characteristics of Qualitative Methods • subjective perspective as an asset – awareness and explicit discussion of one’s own ‘lens’ Strauss, A.L., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, London (1998) Auerbach, C.: Qualitative Data: An Introduction to Coding and Analysis. University Press, New York (2003)
Qualitative Methods in Visualization Qualitative methods can help to enhance our understanding of: – Experiences beyond usability – Environments and work practices – Visual data analysis and reasoning – Communication through visualization – Collaborative visual analysis Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2012). Empirical studies in information visualization: Seven scenarios. IEEE Transactions on Visualization and Computer Graphics , 18 (9), 1520-1536.
Summary • carefully choose method to fit the research goal • consider trade-offs between realism , generalizability and precision • borrow and adapt methods from other fields • conduct research with rigor specific to the chosen method • publish sufficient details • make claims appropriate to the chosen method
Recommend
More recommend
