More than preservation: Creating motivational designs and tailored - PowerPoint PPT Presentation

More than preservation: Creating motivational designs and tailored incentives in research data repositories Sebastian S. Feger, Sünje Dallmeier-Tiessen, Pamfilos Fokianos, Dinos Kousidis, Artemis Lavasa, Rokas Maciulaitis, Jan Okraska, Diego Rodriguez Rodriguez, Tibor Šimko, Anna Trzcinska, Ioannis Tsanaktsidis, Stephanie van de Sandt

About the Presenter 3 rd year PhD Student Human-Computer-Interaction (HCI) Focus on designing technology interaction that supports reproducible research practices 2

Motivation Feilden, Tom. "Most scientists ‘can’t replicate studies by their peers.’." BBC News (2017).

Baker, Monya. "1,500 scientists lift the lid on reproducibility.’." Nature (2016).

Research Preservation and Reproducibility Essential to: Demands: ● Verify findings ● Accessibility ● Compare results ● Rich documentation ● Enable future reuse ● Sharing Hindered by: ● Required effort ● Missing rewards 5

How can we design technology that supports research preservation and sharing? 6

Research Context Data-intensive scientific domain CERN is a key High Energy Physics (HEP) laboratory Some of the largest, distributed scientific collaborations Inventive and adaptive to new technologies (e.g. The World Wide Web) 7

Preservation Requirements Analysis resources with several Gigabytes / Terabytes Multiple types of information and files Datasets Code Meta-Data Environments Workflows Results x86 8

CERN Analysis Preservation (CAP) Community-tailored service Eases ○ Documentation ○ Preservation ○ Sharing Drafts are visible only to author(s) Published analyses are accessible in the collaboration Excerpt from the CMS preservation template on CERN Analysis Preservation

How can we design technology that supports research preservation and sharing? 10

Research Activities Reviewed Interview Gamification Workshop HEP Practices Study Prototypes Observation 11

Research Activities Reviewed Interview Gamification Workshop HEP Practices Study Prototypes Observation 12

Research Activities Reviewed Interview Gamification Workshop HEP Practices Study Prototypes Observation Feger et al, CHI 2019: Designing for Reproducibility: A Qualitative Study of Challenges and Opportunities in High Energy Physics 13

How can preservation technology create meaningful incentives to contribute? 14

Study 12 Semi-structured interviews with physicists Diverse experience levels Mean interview duration: 46 minutes (SD: 7.6) Key interview protocol parts ○ Current sharing and reuse practices ○ Repository exploration ○ Search needs 15

Motivation & Benefits “People may want to use information - but we need to get them to contribute information as well.” - P1, Postdoc “It's more motivating to start a new analysis , other than spending time encoding things…” - P9, Convener “ … if there is a way of getting an extra benefit out of this, while doing your proper preservation, that is good - that would totally work.” - P11, Convener 16

Design Implications Motivate and incentivize contributions Exploit platforms’ secondary usage forms ○ Support coping with uncertainty ○ Provide collaboration-stimulating mechanisms ○ Support structured designs 17

Interview Study Conclusion Tailored preservation technology can lower efforts Balance effort required with benefit offered Design of tailored services should consider tailored incentives Secondary usage forms ○ can provide meaningful benefits to contributors ○ transform researchers’ perceptions of preservation services Sebastian S. Feger, Sünje Dallmeier-Tiessen, Albrecht Schmidt, Paweł W. Woźniak

Research Activities Reviewed Interview Gamification Workshop HEP Practices Study Prototypes Observation 19

Research Activities Reviewed Interview Gamification Workshop HEP Practices Study Prototypes Observation Feger et al, CHI 2019: Gamification in Science: A Study of Requirements in the Context of Reproducible Research 20

Open Science Badges (Center for Open Science) impact practices (Kidwell et al, 2016) 21

Can gamification motivate scientists to document, preserve and share their research? 22

Simple Game Elements Design (SGED)

Simple Game Elements Design (SGED)

Rational-Informative Design (RID)

Rational-Informative Design (RID)

Evaluation 10 within-subjects evaluation sessions Diverse experience levels Participants o explored prototypes and commented (5.2 hours recorded) o responded to questionnaires evaluating the prototypes 27

Results 28

Gamification in Science “ We are so many people in the collaborations, of course. Especially if we want to continue in our field, we have to get some visibility somehow. ” – P6, PhD Student “And what I think what we largely want to do is get signals to people of which ones are like doing it best ‘practice-y’ way and which ones aren't.” – P8, Professor 29

Findings & Implications Scientific Challenges Breaking Points Applications Beyond Achievements 30

Reviewed Interview Gamification Workshop HEP Practices Study Prototypes Observation 31

Conclusion ● Secondary usage forms are expected to motivate contributions ● Gamification can stimulate contributions ● Researcher-centered design ○ enables meaningful new uses for researchers that are not part of traditional repository design thinking ○ reshapes user perception of repositories CERN Analysis Preservation cernanalysispreservation @ analysispreserv 32