Improving IR-based Traceability Recovery Using Smoothing Filters - PowerPoint PPT Presentation

Improving IR-based Traceability Recovery Using Smoothing Filters Andrea Massimiliano Rocco Annibale Sebastiano De Lucia Di Penta Oliveto Panichella Panichella

Software traceability “ The degree to which a relationship can be established between two products of a software development process ” [IEEE Glossary for Software Terminology] Source Use case Test case code Source Use case Test case code n Important for: Up-to-date n program comprehension traceability links n requirement tracing rarely exist → need to n impact analysis n software reuse recover them n …

IR-based traceability recovery Antoniol et al., 2002 (VSM+Probabilistic model) Marcus and Maletic, 2003 (LSI)

Traditional IR vs. IR applied to Software Engineering Traditional IR IR applied to SE n Deals with n We have sets of heterogeneous homogeneous documents for what documents for what concerns: concerns n Linguistic choices n Syntax, linguistic choices n Syntax n Examples: n Semantics n Use cases, test n We just live with that documents, design differences documents follow a common template and contain recurrent words

Problem n Different kinds of software artifacts require specific preprocessing Test case Change the date for a visit: C51 Version: 0 02 000 Use case Satisfies the request to modify a visit for a patient UcModVis Priority High .... Test description Input Select a visit: 26/09/2003 11:00 First visit Change: 03/10/2003 11:00 Oracle Invalid sequence: The system does not allow to change a booking Coverage Valid classes: CE1 CE8 CE14 CE19 CE21 Invalid classes: None

Problem n Different kinds of software artifacts require specific preprocessing Test case Change the date for a visit: C51 Version: 0 02 000 Use case Satisfies the request to modify a visit for a patient UcModVis Artifact-specific words do Priority High not bring useful information .... Test description Input Select a visit: 26/09/2003 11:00 First visit Change: 03/10/2003 11:00 Oracle Invalid sequence: The system does not allow to change a booking Coverage Valid classes: CE1 CE8 CE14 CE19 CE21 Invalid classes: None

A similar problem: image processing

Noisy images Pixels with peaks of low Pixels with peaks of color intensity high color intensity Noise

Reducing noise using smoothing filters Mean filter 1 g ( x , y ) f ( n , m ) ∑ = M f ( n , m ) S ∈

Image vs. traceability noise Traceability noise: Image noise: n Terms and linguistic n Pixels with high or patterns occurring in low color intensity many artifacts of a given n Pixels are position category dependent n Use cases, test cases.. n Artifacts (columns) are position independent d2 ¡ ¡ ¡d1 ¡ ¡ ¡ d1 ¡ ¡ ¡d2 ¡ ¡ ¡

Representing the noise Source Documents Target Documents s 1 s 2 s 3 … s k t 1 t 2 t 3 … t z s 1 s 2 s 3  s k t 1 t 2 t 3  t z ! $ v 1,1 v 1,2 v 1,3 v 1,1 v 1,2 v 1,3 v 1, k v 1, z # & word 1   # & v 2,1 v 2,2 v 2,3 v 2,1 v 2,2 v 2,3 v 2, k v 2, z word 2   # & # &                # &   word n v n , k v n , z # & v n ,1  v n ,2  v n ,3 v n ,1  v n ,2  v n ,3 # & " % Linguistic information strictly Linguistic information strictly belonging to source documents belonging to target documents Common Information Common Information for source documents For target documents

Representing the noise Source Documents Target Documents s 1 s 2 s 3 … s k t 1 t 2 t 3 … t z s 1 s 2 s 3  s k t 1 t 2 t 3  t z ! $ v 1,1 v 1,2 v 1,3 v 1,1 v 1,2 v 1,3 v 1, k v 1, z # & word 1   # & v 2,1 v 2,2 v 2,3 v 2,1 v 2,2 v 2,3 v 2, k v 2, z word 2   # & # &                # &   word n v n , k v n , z # & v n ,1  v n ,2  v n ,3 v n ,1  v n ,2  v n ,3 # & " % " % " % k m 1 1 $ ' $ ' ∑ v 1, j ∑ v 1, j k z $ ' $ ' j = 1 j = k + 1 $ ' $ ' k m 1 1 $ ' $ ' ∑ v 2, j ∑ v 2, j S= Mean ¡target ¡vector ¡ T= Mean ¡source ¡vector ¡ $ ' $ ' S = k z j = 1 j = k + 1 $ ' $ ' $  ' $  ' $ ' $ ' k m 1 1 $ ' $ ' ∑ v n , j ∑ v n , j Common Information Common Information k z $ ' $ ' j = 1 j = k + 1 # & # & for source documents for target documents The Mean vectors are like the continuous component of a signal…

Representing the noise Source Documents Target Documents s 1 s 2 s 3 … s k t 1 t 2 t 3 … t z s 1 s 2 s 3  s k t 1 t 2 t 3  t z ! $ v 1,1 v 1,2 v 1,3 v 1,1 v 1,2 v 1,3 v 1, k v 1, z # & word 1   # & v 2,1 v 2,2 v 2,3 v 2,1 v 2,2 v 2,3 v 2, k v 2, z word 2   # & # &                # &   word n v n , k v n , z # & v n ,1  v n ,2  v n ,3 v n ,1  v n ,2  v n ,3 # & " % - - S T (mean ¡source ¡vector) ¡ (mean ¡target ¡vector) ¡ Filtered ¡ Filtered ¡ Source ¡Set ¡ Target ¡Set ¡

Empirical Study n Goal: analyze the effect of smoothing filter n Purpose: investigating how the filter affects traceability recovery n Quality focus: traceability recovery performance n Perspective: n Researchers: evaluating the novel technique n Context: artifacts from two systems n EasyClinic and Pine

Context EasyClinic Pine Description Medical doctor office Text-based management email client Language Java C Files/Classes 37 31 KLOC 20 130 Documents 113 100 Language Italian English Artifacts Use cases Requirements Use cases Interaction diagrams Source code Test cases

Research Questions and Factors n RQ1: Does the smoothing filter improve the recovery performances of traceability recovery? n RQ2: How effective is the smoothing filter in filtering out non-relevant words, as compared to stop word removal? n Factors: n Use of filter: YES, NO n Technique: VSM, LSI n Artifact: Req., UC, Int. Diagrams, Code, TC n System: Easyclinic, Pine

Analysis Method – RQ1 n Performances evaluated by precision and recall: correct retrieved ∩ correct retrieved ∩ recall = precision = correct retrieved M2 M1 n We statistically compare the # of false positives of different 0 methods for each correct link 2 identified 2 n Wilcoxon Rank Sum test 3 n Cliff ’ s delta effect size

Analysis Method – RQ2 n We replace stop word filtering by one of the following treatments: Standard stop word removal 1. Manually customized stop word removal 2. Smoothing filter 3. Standard stop word removal + filter 4. Customized stop word removal + filter 5. n …and compare the performances

Results

EasyClinic: Use cases into source (VSM) [-60, -74]% of false positives for recall<80% Filtered Precision Not Filtered Recall

EasyClinic: Use cases into source (LSI) [-60, -77]% of false positives for recall<80% Filtered Precision Not Filtered Recall

EasyClinic: Test cases into source (LSI) Test cases are: § Short documents § Limited vocabulary § Mostly consistent with source code Precision Filtered Not Filtered Recall

Pine: Use cases into requirements (LSI) [-42, -62]% of false positives for recall<80% Precision Filtered Not Filtered Recall

Statistical Comparison Data set Traced VSM LSI Artifacts p-value Effect size p-value Effect size EasyClinic UC → Code <0.01 0.50 <0.01 0.50 ( large ) ( large ) Int. Diag. <0.01 0.52 <0.01 0.34 → Code ( large ) ( medium ) TC → Code 1.00 - 1.00 - ( negligible ) ( negligible ) Pine Req. → UC <0.01 0.58 <0.01 0.58 ( large ) ( large )

RQ2 – Summary of results Comparison EasyClinic Pine UC è è CC ID è è CC TC è è CC HLR è è UC Smoothing filter Standard list YES YES NO YES (small) (small) (large) (large) Smoothing filter Cust. list YES YES NO YES (small) (small) (large) (large) Standard list+ Cust. list YES YES NO YES Smoothing filter (large) (large) (medium) (large) Standard list+ Cist list NO - YES YES Smoothing filter + (small) (medium) (small) Smoothing filter

Link precision improvement Login Patient vs. Person Poor vocabulary overlap (10%)

Threats to validity n Construct validity n Mainly related to our oracle n Provided by developers and for EasyClinic also peer- reviewed n Internal validity n Improvements could be due to other reasons… n However, we compared different techniques (VSM, LSI) n The approach works well regardless of stop word removal, stemming and use of tf-idf n Conclusion validity n Conclusions based on proper (non-parametric) statistics n External validity n We considered systems with different characteristics and artifacts n … but further studies are desirable