Methodological concerns emerging from matched sample design of disabled people living in Hungary by Bori Simonovits, PhD Quality of Life and Costs of Living and Services of Disabled People in Various Residential Arrangements in Hungary (VP/2013/013/0057)
Structure of the presentation: • Background and aim of the research project on disabled people living in Hungary • Why they are hard-to-survey? • Research design • Sampling: matched sampling design » criteria for matching • Methodological challenges and limitations of the research • Feedback on the results to the target group
Background of the project: • Objectives: (1) to examine and compare the quality of life of adults with disabilities living in various residential arrangements ; (2) to analyse the cost- effectiveness of the three residential arrangements in terms of social care provision. • Target groups: (1) adults with intellectual disabilities, (2) adults with autism (ASD) (3) adults with profound intellectual and multiple disabilities (4) adults with psychosocial disabilities (schizophrenia) • Residential arrangements : (1) larger residential institutions, (2) smaller group homes and (3) private households • Time frame : March 2015 – January 2016 • Commissioned by the Equal Opportunities of Persons with Disabilities Non-profit Ltd. (FSZK) and Hand in Hand Foundation (Kézenfogva Alapítvány)
Disabled people as hard-to-survey target groups: • Based on the typology of Tourangeau (2014 ) on hard-to-survey groups: – Hard-to-sample , due to the lack of proper sampling frame and low prevalence in the population (e.g. the prevalence of autism is around 0,6-1%, schizophrenia approx. 1%) – Hard-to-identify: esp. people with autism, due to the lack of diagnosis in case of older generations (they have other primary diagnosis, mainly intellectual disabilities) and false negative cases in case of psychosocial disabilities and motivated misreporting of schizophrenia – Hard-to-contact: esp. those who live at institutions (due to gatekeepers ) and to certain extent family homes (could not be contacted or explicitly refused ) – Hard-to-persuade : esp. people with psychosocial disabilities and people with autism (high refusal rate » response bias) – Hard-to interview: all target groups, but esp. people with profound intellectual and multiple disabilities » proxies were helped them to translate the questions;
Research design: “linked questionnaires”: • A self-report “client questionnaire”: – the respondent’s subjective well-being in eight quality of life domains and some basic socio-demographic characteristics and personal opinion; – Answered by the clients themselves or with the help of a „translator” carer upon request • A “carer questionnaire”: – answered by a carer (paid or unpaid/family carer) who knows the client and their everyday life well; – provides information on the living conditions, infrastructure and some aspects of objective quality of life of the individual and it repeats the questions on quality of life from the “client questionnaire”; • A “manager/head of household questionnaire”: – answered by someone who is familiar with the financial situation – budget, income, spending etc. – of the setting where the participant lives (i.e. institution, group home, or private household). – On the costs and infrastructure of the social provision and care of the client
Measuring Quality of life: based on (Schalock et al. 2002) • These QoL domains have been (1) emotional well-being validated both in terms of (2) material well-being internal structure and stability by (3) interpersonal relations various international studies (see e.g. Verdugo et al. 2005) and in (4) personal development different cultural contexts (5) physical well-being (Schalock et al. 2005). • In Hungary we developed a scale (6) self-determination of 16 item scale (2 items per (7) social inclusion domains) (8) rights • Combined quality of life score (0- 16)
Sampling and recruitment of the participants Stratified sample , but not representative (taking into account location and client profile of the institutions and group homes ); Various channels (day service providers, advocacy organisations) in case of recruiting private households ; serious selection bias in all residential arrangements and all target groups; different case mix (by age, gender, degree of disability) in the different residential arrangements In order to eliminate bias matching was carried out , after the data collection
The aim of the research » matching criteria • To examine the quality of life outcomes and cost-effectiveness across different residential arrangements, the criteria of matching varied by groups; • The basis of the matching: – Intellectual disabilities and autism sub-samples: on the basis of adaptive behaviour (i.e. the collection of practical and social skills used in everyday life) that is considered a key determinant of quality of life for people with intellectual disabilities (Ericsson- Mansell 1996). – The sub-sample of people with psychosocial disabilities was matched on the basis of their age . (due to the lack of sufficient information on the time of the first diagnosis of their schizophrenia)
Composition of the achieved sample (N=359) Profound Intellectual intellectual and Psychosocial Autism Total disabilities multiple disability disabilities prior to matching total 130 82 112 138 462 after matching Institution 33 - 31 35 99 Group home 33 30 31 35 130 Private 33 30 31 35 130 household Total 99 60 93 105 359
Methodological limitations and concerns 1. Internal validity concerns stemming from: • Lacking proper validation of the measurement tool; • The use of different groups as proxy respondents in the different residential settings; 2. Selection bias throughout the sampling and recruitment process; 3. External validity concerns , resulting from the non representative ( matched) sample ;
1. Internal validity concerns # 1: • The quality of life question block was not validated properly (due mainly to time constrains…) » Therefore scores on individual quality of life items and the combined quality of life scores are not comparable across target groups. • We focused on certain quality of life areas that are relatively easy to measure by survey tools disregards others , that are equally important for quality of life (e.g. the quality of interpersonal relationships was not measured). » We were only able to measure certain aspects of QoL.
1. Internal validity concerns #2: different types of respondents • Clients: individual variations can occur in the ability to self-report , particularly among those with more severe cognitive limitations ; • Proxy respondents (if the clients were not able to answer the questions): – different groups as proxy respondents in the different residential settings (i.e. paid carers and family carers) and we could not take into account any potential differences between their responses; – paid carers as proxies tend to rate quality of life higher than family carers, esp. in those cases when paid carers know the individual less well (Rand and Caiels 2015); » In general: Social desirability bias (i.e. respondents to answer questions in a manner that will be viewed favourably by others ) might be a serious methodological concern in our study
2. Selection bias: Although the study did not aim to use a representative sample , we must take into account that institutions and group homes that agreed to take part were systematically and significantly different from those who did not want to take part in research. Similarly, it is unclear to what extent families that were recruited via parents’ associations and advocacy organisations represent “typical” families . » these kinds of selection bias occurs in any types of surveys …
3. External validity concerns Low number of observations Case-by-case matching was not possible, groups were created based on: » adaptive behaviour categories (e. g independent in less than 4 areas/independent in 5-9 areas of everyday life); » age groups categories (e. g 30-47 years old / 48-65 years old) in case of the people with psychosocial disability; Our matching procedure was far from optimal… External validity problems: International comparison? • 3 residential arrangements (HU) ↔ typically 2 arrangements are compared (institutions and group homes); • 4 target groups (HU) ↔ international research examples typically focus on 1 target group ;
Conclusion • Partly due to the research design (various residential arrangements and too many target groups ) • Partly due to the specific research goal » matched sample design –Numerous methodological compromise have been made throughout the implementation of the research; –Raising serious validity (both internal and external) and reliability ( mainly due to the interview situation itself and using proxies ) concerns emerged...
Finally… • We found important providing feedback about research finding to the target group. • How to present results to the target group..? » An easy-to-read summary was design to show the results of the empiric research...
Recommend
More recommend
