Conclusion Representations Differences ◮ Mutation operations and Rep. Type Languages Search Space CLang tree C-family source ◮ Languages Cil tree C Int. Lang. LLVM vector LLVM SSA IR ◮ Requirements ASM vector x86 & ARM ◮ Expression as executables ELF vector x86 & MIPS ◮ Communication of mutations Neutral Networks and Automated Software Evolution Program Representation and Transformation 8
Outline Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 9
Software Mutational Robustness [Schulte et al. , GPEM 2013] percentage of mutants which are functional
Software Mutational Robustness Program Space Specification Test Suite Original Program Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 10
Software Mutational Robustness Program Space Specification Test Suite Original Program Neutral Mutant Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 10
Software Mutational Robustness Program Space Specification Test Suite Original Program Neutral Mutant Killed Mutant Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 10
Definition MutRB ( P , T , M ) P program test suite T M mutation operators MutRB ( P , T , M ) = |{ P ′ | m ∈ M . P ′ ← m ( P ) ∧ T ( P ′ ) }| |{ P ′ | m ∈ M . P ′ ← m ( P ) }| Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 11
Mutational Robustness by Test Suite Sorters Siemen’s 50 50 % MutRB 40 40 % MutRB 30 30 20 20 10 10 0 0 b i m q u n u s e p s s s t c b e r i c r c e p b r g k i n h d a a l t e t e c s e i o - - s t d e - s o o u s n o k l o - r r t e e r s t t o n r s t Systems Programs Cil ASM 100 80 % MutRB 60 40 20 0 bzip2 1.0.2 – ccrypt 1.2 – imagemagick 6.5.2 jansson 1.3 leukocyte lighttpd 1.4.15 nullhttpd 0.5.0 oggenc 1.0.1 – potion 40b5f03 redis 1.3.4 tiff 3.8.2 vyquon 335426d grep Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 12
Fitness Distribution by Representation 3500 CLANG CIL LLVM 3000 ASM ELF 2500 Number of Variants 2000 1500 1000 500 0 0 1 2 3 4 5 6 7 8 9 10 Fitness Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 13
Fitness Landscapes and Neutral Networks Program Space Specification Test Suite Original Program Neutral Mutant Killed Mutant Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14
Fitness Landscapes and Neutral Networks P r o g r a m S p a c e S p e c i fi c a t i o n T e s t S u i t e O r i g i n a l P r o g r a m Fitness N e u t r a l M u t a n t K i l l e d M u t a n t Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14
Fitness Landscapes and Neutral Networks P r o g r a m S p a c e S p e c i fi c a t i o n T O e r i s t g i S n a u i l t e P r o g r a m N e u t r a l M u t a n t Fitness K i l l e d M u t a n t Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14
Fitness Landscapes and Neutral Networks P r o g r a m S p a c e T e s t S u i t e Fitness Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14
Fitness Landscapes and Neutral Networks P r o g r a m S p a c e T e s t S u i t e Fitness Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14
Fitness Landscapes and Neutral Networks P r o g r a m S p a c e T e s t S u i t e Fitness Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14
Fitness Landscapes and Neutral Networks P r o g r a m S p a c e T e s t S u i t e Fitness Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 14
Span of Neutral Networks Insertion Sort Neutral Variants 230 21 20 220 19 % Neutral Variants 210 18 Avg. LOC 17 200 16 190 15 14 180 13 Avg. LOC % Neutral Variants 170 25012 0 50 100 150 200 Number of Applied Mutations Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 15
Span of Neutral Networks Insertion Sort Neutral Variants: size controlled 175 20 18 16 % Neutral Variants 174 14 Avg. LOC 12 10 173 8 6 Avg. LOC % Neutral Variants 172 2504 0 50 100 150 200 Number of Applied Mutations Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 15
Random Walks in Program Space Quick Sort Random Variants 1 0 . 35831 x Fraction of Variants which are Neutral 0.9 Fraction Found Neutral Experimentally 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 1 2 3 4 Number of Applied Mutations Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 16
Conclusion Software Neutral Networks are wide span large distances in program space thick consistent mutational robustness accessible automatically explorable Program Space Test Suite Fitness Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 17
Conclusion Software Neutral Networks are wide span large distances in program space thick consistent mutational robustness accessible automatically explorable Neutral Networks and Automated Software Evolution Mutational Robustness and Neutral Networks 17
Outline Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion Neutral Networks and Automated Software Evolution Application: Program Diversity 18
Automated Diversity Program Space Specification Test Suite Original Program Neutral Networks and Automated Software Evolution Application: Program Diversity 19
Automated Diversity Program Space Specification Test Suite Original Program Mutant Neutral Networks and Automated Software Evolution Application: Program Diversity 19
Automated Diversity Program Space Specification Test Suite Neutral Networks and Automated Software Evolution Application: Program Diversity 19
Automated Diversity Program Space Specification Test Suite Neutral Networks and Automated Software Evolution Application: Program Diversity 19
Automated Diversity Proactive Bug Repair Program Space Specification Test Suite Bug Neutral Networks and Automated Software Evolution Application: Program Diversity 19
Contribution Automated Diversity ◮ Neutral networks expose meaningful diversity [Baudry, ISSTA 2014] ◮ Diverse variants proactively fix defects ◮ Select small mutually diverse populations Neutral Networks and Automated Software Evolution Application: Program Diversity 20
Outline Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 21
Program Repair in Embedded Devices [Schulte et al. , ASPLOS 2013] ; ! b a Resource Constraints ( t f i n i ) ; ] a [ v t ◮ Small disks g n i r a ) ] * [ r , a ◮ Less memory c h g c r , a c g t r ◮ Slow processors a n i ( n i a i a m m ◮ Slow, costly comm. Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 22
Genprog Automatically Fix Bugs in C Software Strengths Effective Repaired 55/105 bugs for $8 each General Multiple classes of bugs and security defects Best Papers ICSE 2009, GECCO 2009, SBST 2009 Humies Gold 2009, Bronze 2012 Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 23
Genprog Automatically Fix Bugs in C Software Strengths Effective Repaired 55/105 bugs for $8 each General Multiple classes of bugs and security defects Best Papers ICSE 2009, GECCO 2009, SBST 2009 Humies Gold 2009, Bronze 2012 Requirements Source code ◮ Build tool chain ◮ Expensive fault localization ◮ Expensive fitness function (compilation, test execution) ◮ Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 23
Genprog: Assembler- and Binary-Level Automatically Fix Bugs in C Software Strengths Effective Repaired 55/105 bugs for $8 each General Multiple classes of bugs and security defects Best Papers ICSE 2009, GECCO 2009, SBST 2009 Humies Gold 2009, Bronze 2012 Requirements Source code ◮ Build tool chain ◮ Expensive fault localization ◮ Expensive fitness function (compilation, test execution) ◮ Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 23
Embedded Repair Benchmark Programs Program Program Description Bug atris graphical tetris game stack buffer exploit encryption utility segfault ccrypt deroff document processing segfault lexical analyzer generator segfault flex source code processing infinite loop indent dictionary lookup infinite loop look svr4 dictionary lookup infinite loop look ultrix merge merge sort duplicate inputs merge sort (in C++) duplicate inputs merge-cpp s3 sendmail utility buffer overflow duplicate text processing segfault uniq units metric conversion segfault embedded media player infinite loop zune Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 24
Contribution ASM and ELF representation ◮ Reduce requirements of automated repair ◮ Effective: fixing 12 (ASM) and 11 (ELF) of 13 ◮ 62% faster runtime ◮ 95% smaller disk footprint ◮ 86% less memory ◮ Reduce resources needed to perform repair ◮ New program repair search spaces Neutral Networks and Automated Software Evolution Application: Assembler- and Binary-Level Program Repair 25
Outline Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 26
NETGEAR Exploit [Schulte et al. , unpublished] 1. URL starting with BRS bypasses authentication 2. URL including unauth.cgi or securityquestions.cgi bypass authentication 3. unprotected page removes authentication for every page http://router/BRS_02_genieHelp.html Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 27
Common Vulnerability NETGEAR WNDR4700 D-Link DIR-100 Linksys WAG200G Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 28
NETGEAR Repair Overview QEMU QEMU Virtual Machines QEMU Environment Evaluate QEMU QEMU Threads Mutate Firmware binwalk Insert Select Binary Population Minimize Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 29
NETGEAR Repair Overview QEMU QEMU Virtual Machines QEMU Environment Evaluate QEMU QEMU Threads Mutate Firmware binwalk Insert Select Binary Population Minimize Repair Runtime evaluations ∼ 36 , 000 runtime 86.6 min. threads 32 Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 29
Contribution Genprog Requirements Non-stripped ELF file ◮ Regression test suite ◮ Fault localization ◮ Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 30
Contribution Genprog Requirements Non-stripped ELF file ◮ Regression test suite ◮ Fault localization ◮ Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 30
Contribution Genprog Requirements Non-stripped ELF file ◮ Regression test suite ◮ Fault localization ◮ Enables ◮ repair of black box executables ◮ repair in data sections ◮ patching proprietary software Neutral Networks and Automated Software Evolution Application: Patching Closed Source Executables 30
Outline Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 31
Optimization Problem [Schulte, Dorn et al. , ASPLOS 2014] Optimizing complex non-functional properties properties × hardware × environment properties memory, network, energy, etc. . . hardware architectures, processors, memory stack, etc. . . environment variables, load, etc. . . Every program transformation requires a-priori reasoning manual implementation guaranteed correctness Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 32
Genetic Optimization Algorithm (GOA) Post-compiler, test-driven, Genetic Optimization Algorithm Post-compiler source .s .s .exe GOA Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 33
Genetic Optimization Algorithm (GOA) Post-compiler, test-driven, Genetic Optimization Algorithm Test driven Use test cases to exercise program ◮ evaluate functionality ◮ measure runtime properties Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 33
Genetic Optimization Algorithm (GOA) Post-compiler, test-driven, Genetic Optimization Algorithm Algorithm Assembler Fitness Function Workload Fitness Population Minimize Profile Executable Mutate Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 33
Genetic Optimization Algorithm (GOA) Post-compiler, test-driven, Genetic Optimization Algorithm Algorithm Assembler Fitness Function Workload Fitness 2 10 pop 2 18 Population Minimize evals Profile time ∼ 16 h Executable Mutate Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 33
Benchmark Applications C/C++ ASM Program Lines of Code Description blackscholes 510 7,932 Finance modeling bodytrack 14,513 955,888 Human video tracking facesim no alternate inputs ferret 15,188 288,981 Image search engine fluidanimate 11,424 44,681 Fluid dynamics animation freqmine 2,710 104,722 Frequent itemset mining raytrace no testable output swaptions 1,649 61,134 Portfolio pricing vips 142,019 132,012 Image transformation x264 37,454 111,718 MPEG-4 video encoder total 225,467 1,707,068 Table: PARSEC benchmark applications. Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 34
Hardware Platforms AMD Server Intel Desktop Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 35
Results: Energy Reduction Energy Reduction 100% AMD Intel 80% 60% 40% 20% 0% b b f fl f s v x A e r w i 2 l o u e v a r p 6 d r i q a e c d s e 4 y m p r k a t a s t t n r i i g c n o a i h e m e n c o s k a l e t s e Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 36
Functionality on Withheld Tests Program AMD Intel blackscholes 100% 100% bodytrack 92% 100% ferret 100% 100% fluidanimate 6% 31% freqmine 100% 100% swaptions 100% 100% vips 100% 100% x264 27% 100% Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 37
Anecdotes Blackscholes ◮ 90% less energy ◮ removed redundant outer loop ◮ modified semantics Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 38
Anecdotes Blackscholes ◮ 90% less energy ◮ removed redundant outer loop ◮ modified semantics Swaptions ◮ 42% less energy ◮ improved branch prediction ◮ hardware specific Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 38
Anecdotes Blackscholes ◮ 90% less energy ◮ removed redundant outer loop ◮ modified semantics Swaptions ◮ 42% less energy ◮ improved branch prediction ◮ hardware specific Vips ◮ 20% less energy ◮ substitution of memory access for calculation ◮ resource trade-off Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 38
Contribution Genetic Optimization Algorithm 1. optimize complex runtime properties (energy) 2. leverages particulars of hardware, and environment 3. reveal compiler inefficiencies 4. find efficiencies, e.g., loop elimination 5. transformations presented as ASM diff Neutral Networks and Automated Software Evolution Application: Optimizing nonfunctional Program Properties 39
Outline Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion Neutral Networks and Automated Software Evolution Future Work 40
Challenges Interface ◮ Guarantee semantics preservation ◮ Communicate goals (fitness functions) ◮ Incorporate into development cycle Power ◮ Novel functionality ◮ Combine components Neutral Networks and Automated Software Evolution Future Work 41
Future Work Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr. ◮ Assembler diff chunks formally equivalent ◮ Static analysis to, ◮ find bugs ◮ measure code quality ◮ Invariant preservation e.g., Daikon Neutral Networks and Automated Software Evolution Future Work 42
Future Work Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr. “Smarter” mutation operations ◮ Repair templates and external code libraries ◮ Type-aware operations ◮ Learned mutation heuristics Neutral Networks and Automated Software Evolution Future Work 42
Future Work Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr. Homologous Crossover Heterologous Crossover between different ◮ compilation flags ◮ versions ◮ implementations Heterologous Crossover ◮ programs ∼ Neutral Networks and Automated Software Evolution Future Work 42
Future Work Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr. Evolution Threads Fuzz Tester e.g., Klee, S 2 E Fuzz Test Program p Passing all Failing Tests Program p Tournament Fitness Selection Evaluation Test Suite ◮ regression tests ◮ fuzz test 1 Population of ◮ fuzz test 2 Program Variants ◮ etc. Neutral Networks and Automated Software Evolution Future Work 42
Future Work Verification, Mutation, Crossover, Hardening, Fitness-Distance Corr. Augment tests to increase granularity ◮ Count unique values of program counter ◮ Real-valued oracle comparison (e.g., numdiff ) Neutral Networks and Automated Software Evolution Future Work 42
Outline Introduction Program Representation and Transformation Mutational Robustness and Neutral Networks Application: Program Diversity Application: Assembler- and Binary-Level Program Repair Application: Patching Closed Source Executables Application: Optimizing nonfunctional Program Properties Future Work Conclusion Neutral Networks and Automated Software Evolution Conclusion 43
Conclusion Extant software is 1. Mutationally robust with large neutral networks Neutral Networks and Automated Software Evolution Conclusion 44
Conclusion Extant software is 1. Mutationally robust with large neutral networks 2. A product of evolution Neutral Networks and Automated Software Evolution Conclusion 44
Conclusion Extant software is 1. Mutationally robust with large neutral networks 2. A product of evolution 3. Amenable to automated evolutionary improvement Neutral Networks and Automated Software Evolution Conclusion 44
Conclusion Extant software is 1. Mutationally robust with large neutral networks 2. A product of evolution 3. Amenable to automated evolutionary improvement Software evolution might facilitate the automation of software engineering Neutral Networks and Automated Software Evolution Conclusion 44
Thank You Eric Schulte email eschulte@cs.unm.edu homepage https://cs.unm.edu/~eschulte dissertation https://cs.unm.edu/~eschulte/dissertation Neutral Networks and Automated Software Evolution Conclusion 45
References Forrest, GECCO 2009 Stephanie Forrest, Westley Weimer, ThanhVu Nguyen and Claire Le Goues. A Genetic Programming Approach to Automatic Program Repair, in G ECCO, 2009 Baudry, ISSTA 2014 Benoit Baudry, Simon Allier, and Martin Monperrus. Tailored source code transformations to synthesize computationally diverse program variants. In I SSTA, 2014. Draghi, Nature 2010 J.A. Draghi, T.L. Parsons, G.P. Wagner, and J.B. Plotkin. Mutational robustness can facilitate adaptation. N ature, 463(7279):353–355, 2010. Le Goues, ICSE 2012 Claire Le Goues, Michael Dewey-Vogt, Stephanie Forrest, and Westley Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In I CSE, 2012. Nguyen, SBST 2009 ThanhVu Nguyen, Westley Weimer, Claire Le Goues and Stephanie Forrest. Extended Abstract: Using Execution Paths to Evolve Software Patches, in S BST, 2009. Schulte, GPEM 2013 Eric Schulte, Zachary. Fry, Ethan Fast, Westley Weimer, and Stephanie Forrest. Software mutational robustness. G enetic Programming and Evolvable Machines, pages 1–32, 2013. Schulte, ASPLOS 2013 Eric Schulte, Jonathan DiLorenzo, Westley Weimer, and Stephanie Forrest. Automated repair of binary and assembly programs for cooperating embedded devices. In A SPLOS, 2013. Schulte, ASPLOS 2014 Eric Schulte, Jonathan Dorn, Stephen Harding, Stephanie Forrest, and Westley Weimer. Post-compiler software optimization for reducing energy. In A SPLOS, 2014. Weimer, ICSE 2009 Westley Weimer, ThanhVu Nguyen, Claire Le Goues and Stephanie Forrest. Automatically Finding Patches Using Genetic Programming, in I CSE, 2009 Neutral Networks and Automated Software Evolution References 46
Backup Slides ◮ Program Space ◮ Benefits of ASM ◮ Program Expression ◮ Embedded Repair Space ◮ Many Bugs ◮ Profiling (GOA) ◮ LLVM SSA ◮ Minimization (GOA) ◮ Robustness and Adaptation Time ◮ Energy Model (GOA) ◮ Proactive Bug Repair ◮ Runtime and Energy Reduction (GOA) ◮ Fault Localization ◮ Physical Evolutionary ◮ Embedded Repair Results Computation ◮ Distributed Repair Neutral Networks and Automated Software Evolution Backup Slides 47
Program Space Back Constants: d=3, e=5 Total n i +1 = n i ( d − i )( e − 1) ( i + 1) Unique n z , i +1 = n z − 1 , i ( d − i )1 + n z , i ( d − i )( e − 2) i + 1 � − 1 � l � A i = n z , i z z ≤ i Step Total Neutral 0 1 � − 1 1 � l � r i � R = n z , i 2 z i ≤ d z ≤ i 3 Neutral Networks and Automated Software Evolution Backup Slides 48
Recommend
More recommend
