efficient malware detection using model checking
play

Efficient Malware Detection using Model-Checking Tayssir Touili - PowerPoint PPT Presentation

Efficient Malware Detection using Model-Checking Tayssir Touili LIPN, CNRS & Univ. Paris 13 Motivation: Malware Detection The number of new malware exceeds 75 million by the end of 2011, and is still increasing. The number of


  1. Efficient Malware Detection using Model-Checking Tayssir Touili LIPN, CNRS & Univ. Paris 13

  2. Motivation: Malware Detection • The number of new malware exceeds 75 million by the end of 2011, and is still increasing. • The number of malware that produced incidents in 2010 is more than 1.5 billion. • The worm MyDoom slowed down global internet access by 10% in 2004. • Authorities investigating the 2008 crash of Spanair flight 5022 have discovered a central computer system used to monitor technical problems in the aircraft was infected with malware

  3. Motivation: Malware Detection • The number of new malware exceeds 75 million by the end of 2011, and is still increasing. • The number of malware that produced incidents in 2010 is more than 1.5 billion. • The worm MyDoom slowed down global internet access by 10% in 2004. • Authorities investigating the 2008 crash of Spanair flight 5022 have discovered a Malware detection is central computer system used to monitor technical problems in the aircraft was infected with malware important!!

  4. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature

  5. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable

  6. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment

  7. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment o Checks program’s behavior only in a limited time interval

  8. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature Solution: o Easy to get around Check the behavior (not the syntax) of o New variants of viruses with the same behavior cannot be detected by the program without executing it these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment o Checks program’s behavior only in a limited time interval

  9. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature Solution: o Easy to get around Check the behavior (not the syntax) of o New variants of viruses with the same behavior cannot be detected by the program without executing it these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment o Checks program’s behavior only in a limited time interval Model Checking is a good candidate

  10. Goal: Model-checking for malware detection

  11. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ?

  12. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Model?

  13. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Model? Specification formalism?

  14. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Model? Specification formalism? Existing works: use finite automata to model the programs

  15. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Model? Specification formalism? Existing works: use finite automata to model the programs Stack?

  16. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system • Antiviruses determine malware by checking the calls to the operating systems. • Virus writers try to hide these calls.

  17. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system • Antiviruses determine malware by checking the calls to the operating systems. • Virus writers try to hide these calls. L0 : push L1 L0 : call f L’0: jmp f L1: … L1: … … … … … f : function f f : function f

  18. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system Important to analyse the program’s • Antiviruses determine malware by checking the calls stack to the operating systems. • Virus writers try to hide these calls. L0 : push L1 L0 : call f L’0: jmp f L1: … L1: … … … … … f : function f f : function f

  19. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system Important to analyse the program’s • Antiviruses determine malware by checking the calls stack to the operating systems. • Virus writers try to hide these calls. L0 : push L1 Solution: L0 : call f L’0: jmp f Use pushdown systems to model L1: … L1: … programs … … … … f : function f f : function f

  20. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Pushdown Specification Systems formalism?

  21. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Specification Pushdown Systems formalism?

  22. Specification of malicious behaviors? Example: fragment of email worm Avron Call the API GetModuleHandleA mov eax, 0 with 0 as parameter. push eax This returns the entry address of its call GetModuleHandleA own executable. Copy itself to other locations.

  23. Specification of malicious behaviors? Example: fragment of email worm Avron Call the API GetModuleHandleA mov eax, 0 with 0 as parameter. push eax This returns the entry address of its call GetModuleHandleA own executable. Copy itself to other locations. How to describe this specification?

  24. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 push eax call GetModuleHandleA In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA )

  25. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) EX p : there is a path where p holds at the next state

  26. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX call GetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX call GetModuleHandleA ) ˅ … .. all the other registers EX p : there is a path where p holds at the next state

  27. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA Huge! In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX call GetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX call GetModuleHandleA ) ˅ … .. all the other registers EX p : there is a path where p holds at the next state

  28. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 CTPL = CTL + push eax ∃ , variables + ∀ call GetModuleHandleA In CTL: mov(eax,0) ˄ EX ( push(eax) ˄ EX callGetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX callGetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX callGetModuleHandleA ) ˅ … .. all the other registers

  29. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 CTPL = CTL + push eax ∃ , variables + ∀ call GetModuleHandleA In CTL: mov(eax,0) ˄ EX ( push(eax) ˄ EX callGetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX callGetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX callGetModuleHandleA ) ˅ … .. all the other registers In CTPL: ᴲ r ( mov(r,0) ˄ EX ( push(r) ˄ EX call GetModuleHandleA ) )

  30. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 CTPL = CTL + push eax ∃ , variables + ∀ call GetModuleHandleA In CTL: CTPL cannot describe the stack: mov(eax,0) ˄ EX ( push(eax) ˄ EX callGetModuleHandleA ) ˅ needed for malicious behaviors mov(ebx,0) ˄ EX ( push(ebx) ˄ EX callGetModuleHandleA ) description ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX callGetModuleHandleA ) ˅ … .. all the other registers In CTPL: ᴲ r ( mov(r,0) ˄ EX ( push(r) ˄ EX call GetModuleHandleA ) )

Recommend


More recommend