Answering the Call of the Wild?: Thoughts on the Elusive Quest for Ecological Validity in Variability Modelling Seiede Reyhane Kamali, Shirin Kasaei, Roberto E. Lopez-Herrejon École de technologie supérieure, Université du Québec, Canada introduction 1 MODEVAR @ SPLC19
Ecological Validity – Form of External Validity } Poses the question: } Is a study, method, artifacts, setting an approximation of what happens in real-life? In the lab … In the wild … 2 MODEVAR @ SPLC19
In SPL, what is “real-life”? Industry } Limited access } Transparent access } Unfeasible reproducibility } Enables reproducibility 3 MODEVAR @ SPLC19
Study description } General question: } What is the impact of open source projects in SPL research? } Our paper focus: } Variability Modeling } Starting point: } José A. Galindo, David Benavides, Pablo Trinidad, Antonio Manuel Gutiérrez Fernández, and Antonio Ruiz-Cortés. 2019. Automated analysis of feature models: Quo vadis? Computing 101, 5 (2019), 387–433. } Corpus of 242 articles from 2017 to 2017 } Open source project – less restrictive interpretation: } Publicly available code repository (URL) } More than one author } No academic projects 4 MODEVAR @ SPLC19
First findings } Primary sources selected: 41 } Open source projects identified: 43 } Publications per year: } Median 5 5 MODEVAR @ SPLC19
Projects and primary sources Subversion H264 Gnumeric } Inconsistent or Gimp Elevator Xfig incomplete report of: Qemu Xterm Vim73 21 projects with } Version numbers Sendmail Libxml2 OpenVPN 1 primary source } Number of features Postgresql Parrot Gnuplot } Projects with more Cherokee x264 Gantt than one version JavaGeom JDK Buffer library OpenSSL reported: Ubuntu Apache http… SQLite LLVM } Eclipse (7) coreboot 9 projects with Freetz EmbToolkit } BusyBox & FraSCati (3) 2 primary sources FraSCAti BuildRoot Eclipse } Drupal & uClibc (2) ArgoUml ToyBox axTLS Drupal BerkeleyDB eCos with 13 Fiasco uClinux Linux rules with 23 uClibc but 1 version BusyBox and 43 versions FreeBSD eCos Linux 0 5 10 15 20 25 6 MODEVAR @ SPLC19
Recency – gap between versions analyzed and released Linux Months Versions Drupal BusyBox OpenSSL FreeBSD ToyBox BuildRoot axTLS Freetz Eclipse x264 coreboot Ubuntu JDK buffer library 0 20 40 60 80 100 120 140 160 180 200 3 years 7 MODEVAR @ SPLC19
Feature Models – What for? 30 Tooling 10 Maintenance and Evolution 2 Application Testing 4 Application Realisation 0 Application Design 7 Application Requirements Engineering 8 Domain Testing 20 Domain Realisation 1 Domain Design 33 Domain Requirements Engineering 0 5 10 15 20 25 30 35 8 MODEVAR @ SPLC19
Availability for replication 7 Undetermined 10 None 2 Partial 22 Full 0 5 10 15 20 25 9 MODEVAR @ SPLC19
Model Formats EvoFM 1 SPLConqueror 1 CIDE 1 FDL 1 Prolog 1 XML 1 BDD 1 CNF 1 Adhoc Metamodel 2 SXFM 2 FAMILIAR 2 FAMA 4 CDL 5 DIMACS 12 KConfig 12 0 2 4 6 8 10 12 10 MODEVAR @ SPLC19
Analysis (1) } Predominance of Linux and KConfig } Number of primary sources, number of versions, most recent, … } T ool ecosystem built around it } Aging and outdated datasets } Only Linux has been updated but still at a lower rate than releases } Artifacts beyond feature models and source code } Open sources projects are used for multiple purposes hence the need to study other types of artifacts à e.g. faults, test cases, test code, … 11 MODEVAR @ SPLC19
Analysis (2) } Replication as a pillar for empirical SPL research } Reproducibility should be the norm not the exception } Building bridges to open source communities } Linux has had enormous impact on SPL research, but how about the other way around? 12 MODEVAR @ SPLC19
Future Work } Expand the focus of our study on impact of open source projects in SPL research to: } Primary sources beyond Galindo et al.’s work } Other artifacts à related to testing activities } Other open source projects } Based on KConfig } Mining Repositories community at large } Open source communities at large 13 MODEVAR @ SPLC19
The End Any questions? Roberto Erick Lopez Herrejon École de technologie supérieure Université du Québec Montreal, Canada Roberto.lopez@etsmtl.ca Natural Sciences and Engineering Research Council of Canada 14 MODEVAR @ SPLC19
Recommend
More recommend
