Collecting information from people Kristian Wiklund Photo CC-BY-2.0 Richard Riley Kristian Wiklund | 20134-03-05 | Page 1 http://www.flickr.com/photos/rileyroxx/169900126/
Outline › What is a survey? › Different ways of collecting data from people – Questionnaires, (Qualitative) Interviews, Focus Groups › Break with discussion › Design: Survey design and How to not ask questions – Total survey quality – Sampling and Frames – Non-sampling validity threats (valid for all designs above) › Self-reporting, scales, order, … Kristian Wiklund | 20134-03-05 | Page 2
What is a survey? › A survey concerns a set of objects comprising a population Describe the population Population – The goal of the survey is to describe the population in terms of one or more measurable properties – To do this, a sampling frame is Sampling needed, from which a sample of the population is selected to be Frame included in the survey › The survey makes observations on the sample Sample – The observed measurements are used to make inferences on the population [Stat100] Kristian Wiklund | 20134-03-05 | Page 3
Why Perform a survey? › Create a profile of a group as whole – The design of the survey makes it possible to generalize the answers of a selected few to the group › Types of research – Exploratory - ‘What is going on here?' – Descriptive - ‘What are the characteristics of the population?’ – Explanatory – ‘Why is something happening?’ – Predictive – ‘What is the likelihood of something happening?’ Kristian Wiklund | 20134-03-05 | Page 4
Data Collection Modes Focus Questionnaires Groups Semi-structured: Structured: A base set of Free form: Following a questions and Having a nice questionnaire clarification as chat needed All interviews conducted in exactly the same way, no explanations, no clarifications. More options “What does it mean for interpretation to you?” Flexibility Self-administered • What are the questionnaires reactions of the respondent? “Discourse Analysis” Kristian Wiklund | 20134-03-05 | Page 5
Questionnaires CC-BY-NC Ikhlasul Amal http://www.flickr.com/photos/21372148@N00/2443194039 Kristian Wiklund | 20134-03-05 | Page 6
The Problem with Questionnaires… “Let’s put together a questionnaire and send to some people and check…” › Ideal situation: – Researcher writes a number of questions, distributes the survey on Twitter, drinks some coffee and writes a couple of journal papers waiting for the data to drop in – Hordes of willing respondents answer the questions – Researcher analyses the data with Excel (it basically analyses itself), writes a paper with some nice bar charts, sends the paper to ICSE – Paper accepted, nice trip to Florence, wine, best paper award Kristian Wiklund | 20134-03-05 | Page 7
The Problem with Questionnaires… In reality: – Researcher intent is encoded into questions › Was that done right? – Researcher finds subjects and convinces them to answer › Was the selection done right? – Questions transmit researcher intent to subject › Were they interpreted correctly? – Subject decides to answer and formulates an answer › Is the answer reliable? – Researcher encodes the answers into something that can be analyzed › Was that done right? – Data is analyzed › Was that done right? Kristian Wiklund | 20134-03-05 | Page 8
Conducting Interviews
Main challenges › Access to subjects › Performing the interviews without introducing bias › Reliable collection of data CC-BY-NC-SA Gabe McIntyre http://www.flickr.com/photos/38366783@N00/2617316249 Kristian Wiklund | 20134-03-05 | Page 10
Validity › The same validity issues as with questionnaire design applies › “The key to successful – ...and will be described in detail interviewing is learning how to later probe effectively… › In addition to this, there is a clear › ...that is, to stimulate an risk for interviewer bias informant to produce more Information… › Useful skills › ... without injecting yourself so – Active listening much into the interaction that you only get a reflection of › http://www.babblingengineer.com/com yourself in the data .” munication/how-to-improve-your- active-listening/ – Coaching [Weiss2000] Kristian Wiklund | 20134-03-05 | Page 11
http://www.faculty.londondeanery.ac.uk/e-learning/appraisal/skilful-questioning-and-active-listening Allow people to answer in their own terms Use non-leading, open, questions to get started, probe for details later Let the subject exhaust one question before moving on Have enough time allocated Kristian Wiklund | 20134-03-05 | Page 12
Recording › I strongly suggest that the interviews are recorded – Informed consent › Multiple reasons: – You will not capture everything in your notes › Not even if someone else takes the notes – Self-evaluation › Did you introduce bias? › Was the design followed? › Personal quirks (“uuuhhhmm…”) http://www.flickr.com/photos/labanex/8668665270/ – Capture “how” and not only “what” Kristian Wiklund | 20134-03-05 | Page 13
Recording › Tooling: – Additional computer works okay, separate microphone recommended, but I have had success with the built in microphone › Transcription – Get software support for transcription › I use Express Scribe on a Mac – Transcription takes time, 2-10 times the interview time › And you will likely feel silly for most of the time listening to yourself http://www.flickr.com/photos/strandarchives/9273941774/in/photostream/ Kristian Wiklund | 20134-03-05 | Page 14
Focus Groups
› “A focus group is a carefully planned discussion designed to obtain perceptions on a defined area of interest in a permissive, non-threatening environment” › ...or what most people in industry would call a “workshop” – Which makes this a very marketable skill. CC-BY-SA-NC 2.0 Some rights reserved by Nebraska Library Commission [Fisk2005] Kristian Wiklund | 20134-03-05 | Page 16
When to use › Use for: › Don’t use for: – Qualitative information – Quantitative information – Insights into new area – Confidential information – Preparation for a larger, – Situations that may go more formal, study out of control › Emotionally charged discussions › Participants with an agenda of their own Kristian Wiklund | 20134-03-05 | Page 17
Design Items › Objectives and questions – as important here as for interviews and questionnaires › Agenda for the focus group meeting – Introduction, questions, summary › Facilitator – neutral person with sufficient domain knowledge and capability to move the work forward – not necessarily the researcher › Note-taker/Assistant facilitator › Group member selection – 6 to 12 people, without dependencies such as manager/employee › Location – “Not the office” Kristian Wiklund | 20134-03-05 | Page 18
How to run the meeting › Basically the same skills and risks as for › Parkinson’s Law of Triviality: interviews › “Briefly stated, it means that the time spent on any item of the agenda will › Group bias risk: be in inverse proportion to the sum – A group will easily bike- involved.” [Parkinson1957] shed on issues that are not really relevant for the research Kristian Wiklund | 20134-03-05 | Page 19
http://www.faculty.londondeanery.ac.uk/e-learning/appraisal/skilful-questioning-and-active-listening › For each question, iterate: – Open question – Deeper follow up probes to keep the ball rolling – Summary of the findings by the moderator For more information: http://www.hse.gov.uk/stress/standards/pdfs/focusgroups.pdf and http://www.cse.lehigh.edu/~glennb/mm/FocusGroups.htm Kristian Wiklund | 20134-03-05 | Page 20
Coffee Break + Discussion › Self-organize a group and discuss: – Is there anything in your research area that can be answered by › A survey › Interviews › Focus groups – Why? Why not? – What are the main challenges in conducting this type of study in your area? Kristian Wiklund | 20134-03-05 | Page 21
Survey and Questionnaire design
The Survey Challenge › The main problems with surveys is that the complexity commonly is under-estimated. – It is commonly perceived to be trivial to do a survey. – In reality, there are limitless ways to go wrong. › A survey is a fixed design, which means that we have to live with our mistakes. › Hence, it is very important to be aware of the potential problems. Kristian Wiklund | 20134-03-05 | Page 23
What is survey research? › Survey research is a branch of statistics – This is evident when the theoretical background is studied › Statistical concepts such as sampling theory is well developed, while psychological concepts such as how to ask questions is still being developed – For a very long time it was assumed that if a survey was designed correctly with respect to sampling, it would be correct. – Now it is known that there are a lot of factors influencing the success of the survey › A survey should be designed to minimize the total survey error (Biemer2011) Kristian Wiklund | 20134-03-05 | Page 24
Recommend
More recommend
