Evaluating Soul City Evaluating Soul City A multi level A multi - PowerPoint PPT Presentation

Evaluating Soul City Evaluating Soul City A multi level A multi level Communication for Social Communication for Social Change initiative Change initiative Shereen Usdin Shereen Usdin

Soul City Soul City A Sout h Af r i c an NGO wor k i ng i n par t ne r s hi p wi t h A Sout h Af r i c an NGO wor k i ng i n par t ne r s hi p wi t h l oc al NGOs i n 8 SADC c ount r i e s t o de v e l op l oc al NGOs i n 8 SADC c ount r i e s t o de v e l op i ndi g e nous CF SC i nt e r v e nt i ons i n Sout he r n Af r i c a i ndi g e nous CF SC i nt e r v e nt i ons i n Sout he r n Af r i c a

We are complex beings… We are complex beings… “ We do not g r ow abs ol ut e l y , c hr onol og i c al l y . We g r ow “ We do not g r ow abs ol ut e l y , c hr onol og i c al l y . We g r ow s ome t i me s i n 1 di me ns i on and not i n anot he r ; s ome t i me s i n 1 di me ns i on and not i n anot he r ; une v e nl y . We g r ow par t i al l y . We ar e r e l at i v e . We ar e une v e nl y . We g r ow par t i al l y . We ar e r e l at i v e . We ar e mat ur e i n one r e al m, c hi l di s h i n anot he r . T he pas t , mat ur e i n one r e al m, c hi l di s h i n anot he r . T he pas t , pr e s e nt and f ut ur e mi ng l e and pul l us bac k war d, pr e s e nt and f ut ur e mi ng l e and pul l us bac k war d, f or war d, or f i x us i n t he pr e s e nt . We ar e made up of f or war d, or f i x us i n t he pr e s e nt . We ar e made up of l ay e r s , c e l l s , c ons t e l l at i ons . ” l ay e r s , c e l l s , c ons t e l l at i ons . ” Anai sNi n* Anai sNi n* *F r om D. Re e l e r , ‘ A t he or y of Soc i al Chang e ” , CDRA *F r om D. Re e l e r , ‘ A t he or y of Soc i al Chang e ” , CDRA

The Soul City Model of Social The Soul City Model of Social Change Change Socio-economic + political national environment • Healthy public policy • Re-orientate services a i d e M s s a M n o i t a s i l i b o M Individual a l i c o S • Knowledge • Attitudes/Beliefs/ Community /Risk perception • Dialogue/debate • Skills • Social norms • Self-efficacy • Diffusion of innovations • Support for policy action • Community efficacy • Subjective norms • Community action • Link to services • Social networks • Social cohesion A d v o c a c y

Experimental designs Experimental designs • Gr oup 1 g e t pi l l and c ont r ol Gr p2, a pl ac e bo • Gr oup 1 g e t pi l l and c ont r ol Gr p2, a pl ac e bo • I nv e s t i g at e c aus e - and- e f f e c t r e l at i ons hi ps • I nv e s t i g at e c aus e - and- e f f e c t r e l at i ons hi ps • At t r i but e e f f e c t t o t he i nt e r v e nt i on • At t r i but e e f f e c t t o t he i nt e r v e nt i on

So Soc i al al Chang hang e i s s c c o ompl e e x x and and s s o o Soc So i al al Chang hang e i s s c c o ompl e e x x and and s s o o CF CF SC i nt nt e e r r v v e e nt i ons ons ne e e e d d mo mor e e CF CF SC i nt nt e e r r v v e e nt i ons ons ne e e e d d mo mor e e c c o ompl e e x x , nuanc nuanc e d e e v v al uat uat i on on c c o ompl e e x x , nuanc nuanc e d e e v v al uat uat i on on de de s i g g n ns de de s i g g n ns

Challenges: noisy confounders Challenges: noisy confounders As exposure to Soul City increases … Can we say that Soul Condom us e i nc r e as e d City causes increased condom use? ????? ????? People exposed to Soul City are also likely to be exposed to other HIV interventions.

More Challenges More Challenges • Caus al i t y i s di f f i c ul t t o pr ov e be c aus e al mos t • Caus al i t y i s di f f i c ul t t o pr ov e be c aus e al mos t impossible to get a pristine control group impossible to get a pristine control group • Chang e i s not uni di r e c t i onal and pe opl e may we l l • Chang e i s not uni di r e c t i onal and pe opl e may we l l r e l aps e r e l aps e • Mai nt ai ni ng pos i t i v e be hav i our al s o c ount s • Mai nt ai ni ng pos i t i v e be hav i our al s o c ount s • Chang e oc c ur s i n a c ompl e x e nv i r onme nt ( f or e x ampl e a • Chang e oc c ur s i n a c ompl e x e nv i r onme nt ( f or e x ampl e a pr e s i de nt of a c ount r y s ay i ng t hat HI V doe s not c aus e pr e s i de nt of a c ount r y s ay i ng t hat HI V doe s not c aus e AI DS –i t i s har d t o me as ur e t he ne g at i v e i mpac t of t hi s ) AI DS –i t i s har d t o me as ur e t he ne g at i v e i mpac t of t hi s )

Things to consider… Things to consider… • Chang e may not happe n ov e r ni g ht and • Chang e may not happe n ov e r ni g ht and e v al uat i ons may be t oo e ar l y t o s e e r e s ul t s e v al uat i ons may be t oo e ar l y t o s e e r e s ul t s • Of t e n ne e d t o me as ur e i nt e r me di at e i ndi c at or s • Of t e n ne e d t o me as ur e i nt e r me di at e i ndi c at or s ( di al og ue /i nt e nt i on…) whi c h may pr e di c t ( di al og ue /i nt e nt i on…) whi c h may pr e di c t s uc c e s s s uc c e s s • Unl e s s y ou t e s t , i t i s har d t o l i nk di r e c t l y t o • Unl e s s y ou t e s t , i t i s har d t o l i nk di r e c t l y t o c hang e i n HI V i nc i de nc e /pr e v al e nc e c hang e i n HI V i nc i de nc e /pr e v al e nc e

Study Design Study Design 5. Media Monitoring and Analysis 2. Sentinel Site Studies 3.National 4. Partnership Qualitative Impact Study Assessment 1. National Quant Community Survey Individual 6. Institutional / Organisational Impact Study Society 7. Cost Effectiveness Study

Evidence of impact attributed Evidence of impact attributed to the intervention to the intervention • Qualitative evidence: Qualitative evidence: • Qualitative evidence: Qualitative evidence: – Direct mention / direct attribution – Direct mention / direct attribution – Consistency – Consistency – Data Triangulation – Data Triangulation • Quantitative evidence: Quantitative evidence: • Quantitative evidence: Quantitative evidence: – Simple descriptive analysis – Simple descriptive analysis – More complex inferential analysis (regression – More complex inferential analysis (regression analysis allowing us to make stronger associations analysis allowing us to make stronger associations – Dose response – Dose response

Evidence of impact attributed to Evidence of impact attributed to the intervention the intervention • Consistency of results Consistency of results • Consistency of results Consistency of results – Across evaluations of multiple SC interventions over time – Across evaluations of multiple SC interventions over time – Across studies within evaluations of one specific intervention – Across studies within evaluations of one specific intervention • Triangulation Triangulation • Triangulation Triangulation – Different Methods / designs – Different Methods / designs – Different Researchers for each study – Different Researchers for each study – Different sources of data (ngo’s; gov; civil leaders etc) – Different sources of data (ngo’s; gov; civil leaders etc)