effective and efficient malware detection at the end host
play

Effective and Efficient Malware Detection at the End Host Clemens - PowerPoint PPT Presentation

Aug 29, 2022 •16 likes •446 views

Secure Systems Lab Technical University Vienna Effective and Efficient Malware Detection at the End Host Clemens KOLBITSCH, Paolo MILANI COMPARETTI, Engin KIRDA, Christopher KRUEGEL, Xiaoyong ZHOU, XiaoFeng WANG ck@iseclab.org Secure Systems

  1. Secure Systems Lab Technical University Vienna Effective and Efficient Malware Detection at the End Host Clemens KOLBITSCH, Paolo MILANI COMPARETTI, Engin KIRDA, Christopher KRUEGEL, Xiaoyong ZHOU, XiaoFeng WANG ck@iseclab.org Secure Systems Lab [TU Vienna, Institute Eurecom Sophia Antipolis, UC Santa Barbara] Indiana University at Bloomington 1

  2. Motivation Secure Systems Lab Technical University Vienna Why do we propose yet another malware detection scheme (yamds)? • Binary signature based detection inherently ineffective – We all know the problems... – Arms-race, pretty much a lost battle • Network based approaches evadable – Systems scan for communication artifacts – Encryption / blending thwart detection 2

  3. Motivation Secure Systems Lab Technical University Vienna Why do we propose yet another malware detection scheme (yamds)? • Don't rely on artifacts of malware instances – Instead focus on generic patterns • Proposed solution: – Detection based on malware's behavior – Behavior is hard to obfuscate – Behavior is hard to randomize – Behavior is often stable across various malware version 3

  4. Motivation Secure Systems Lab Technical University Vienna • Behavior-based detection received some attention over last couple of years • Despite promising detection results, binary signatures remain the method of choice 4

  5. Motivation Secure Systems Lab Technical University Vienna • Behavior-based detection received some attention over last couple of years • Despite promising detection results, binary signatures remain the method of choice + efficiency - emulation binary signatures behavior - evasion + effectiveness 5

  6. Motivation Secure Systems Lab Technical University Vienna • Behavior-based detection received some attention over last couple of years • Despite promising detection results, binary signatures remain the method of choice + efficiency - emulation binary signatures behavior - evasion + effectiveness 6

  7. Outline Secure Systems Lab Technical University Vienna • Motivation • Detecting Behavior – Motivating example (Agent) • Matching Behavior Graphs • Extracting Behavior Graphs • Evaluation 7

  8. Secure Systems Lab Technical University Vienna Detecting Behavior 8

  9. Detecting Behavior Secure Systems Lab Technical University Vienna • Characteristic malware behavior – Manifest on system (i.e., survive reboot) • (Over-) write system executables, dlls, files • Create registry entries (autorun) • Register as Windows (startup) service – Conceal from being detected • Restart under some stealthy name (e.g., svchost.exe) • Inject into legitimate processes – Replicate • Send eMails ( 'check out this picture I found: pic.jpg.exe' ) • Copy to Samba shares, USB drives, etc. • Scan and exploit services on LAN or WAN 9

  10. Detecting Behavior System Overview Secure Systems Lab Technical University Vienna • Detection based on execution characteristics – Execute malware in full system emulator (Anubis) – Monitor interaction with the operating system – Perform detailed (taint-) analysis – Generate detection graphs • Describe sequence of required system calls leading to security relevant system activity • Include dependencies to related, previous calls (using taint dependencies) • Detect described behavior on end host – Log system call activity of unknown executable – Match against behavior graph 10

  11. Detecting Behavior Developer Perspective Secure Systems Lab Technical University Vienna • Example: Agent (trojan horse) • As part of its system manifestation, it – Reads content from binary image – Decrypts binary content • Proprietary decryption routine • Simple, XOR based algorithm – Stores binary in system file ( C:\Wind...\drivers\ip6fw.sys ) – Later, restarts IPv6 firewall • Turns itself into a system service 11

  12. Detecting Behavior Taint-Trace Perspective Secure Systems Lab Technical University Vienna GetModuleFileNameA Name FileHandle Mode: Open NtCreateFile NtCreateSection SectionHandle Mode: Create NtCreateFile NtMapViewOfSection C:\Win... \ip6fw.sys (read & decrypt buffer) FileHandle NtWriteFile 12

  13. Detecting Behavior System Perspective Secure Systems Lab Technical University Vienna GetModuleFileNameA Name FileHandle Mode: Open NtCreateFile NtCreateSection SectionHandle Mode: Create NtCreateFile NtMapViewOfSection C:\Win... \ip6fw.sys (read & decrypt buffer) FileHandle NtWriteFile 13

  14. Detecting Behavior System Perspective Secure Systems Lab Technical University Vienna GetModuleFileNameA Name FileHandle Mode: Open NtCreateFile NtCreateSection SectionHandle Mode: Create NtCreateFile NtMapViewOfSection C:\Win... \ip6fw.sys (read & decrypt buffer) FileHandle NtWriteFile 14

  15. Detecting Behavior System Perspective Secure Systems Lab Technical University Vienna GetModuleFileNameA Name FileHandle Mode: Open NtCreateFile NtCreateSection SectionHandle Mode: Create NtCreateFile NtMapViewOfSection C:\Win... \ip6fw.sys (read & decrypt buffer) FileHandle NtWriteFile 15

  16. Detecting Behavior System Perspective Secure Systems Lab Technical University Vienna Mode: Open NtCreateFile NtCreateFile Mode: Create FileHandle C:\Win... \ip6fw.sys NtCreateSection FileHandle SectionHandle NtMapViewOfSection NtWriteFile 16

  17. Detecting Behavior System Perspective Secure Systems Lab Technical University Vienna Mode: Open NtCreateFile NtCreateFile Mode: Create FileHandle C:\Win... \ip6fw.sys NtCreateSection FileHandle SectionHandle NtMapViewOfSection NtWriteFile 17

  18. Detecting Behavior Secure Systems Lab Technical University Vienna • Detection based on execution characteristics – Works well as long as we can see all types of dependencies between system calls – Handle dependencies • Insufficient for detection • Behavior graphs break into trivial subgraphs – Data dependencies • Convenient for behavior graph generation • Necessary for behavior detection 18

  19. Secure Systems Lab Technical University Vienna Matching Behavior Graphs 19

  20. Matching Behavior Graphs Secure Systems Lab Technical University Vienna • Maintaining dependencies using taint propagation – Performance overhead: Extended emulation engine – Memory overhead: Shadow memory – Not applicable to production systems / end hosts • Maintaining dependencies without taint propagation – Handle dependencies • Direct value propagation • System provided identifiers – File, section, process, thread handles – Registry keys – Socket identifiers – Must be constant between call invocations 20

  21. Matching Behavior Graphs Secure Systems Lab Technical University Vienna • Maintaining dependencies without taint propagation – Data dependencies • Arbitrary data (& control) dependency between system calls • Might modify values between system calls – Our proposal: Anticipate precise call arguments • Use recorded execution semantics • Extract data propagation/manipulation formulas • Emulate taint dependency between system call A and B – Log outgoing parameters of call A – Use as input to propagation formula – Predicted incoming parameters for system call B – Compare predicted and monitored input parameters – Assume dependency between A and B if prediction holds 21

  22. Matching Behavior Graphs System Perspective Secure Systems Lab Technical University Vienna GetModuleFileNameA Name FileHandle Mode: Open NtCreateFile NtCreateSection SectionHandle Mode: Create NtCreateFile NtMapViewOfSection C:\Win... \ip6fw.sys (read & decrypt buffer) FileHandle NtWriteFile 22

  23. Matching Behavior Graphs System Perspective Secure Systems Lab Technical University Vienna GetModuleFileNameA f 1, data Name FileHandle Mode: Open NtCreateFile f 2, handle NtCreateSection SectionHandle f 3, handle Mode: Create NtCreateFile NtMapViewOfSection C:\Win... f 5, handle \ip6fw.sys (read & decrypt buffer) f 4, data FileHandle NtWriteFile 23

  24. Matching Behavior Graphs System Perspective Secure Systems Lab Technical University Vienna GetModuleFileNameA f 1, data Name FileHandle Mode: Open NtCreateFile f 2, handle NtCreateSection SectionHandle f 3, handle Mode: Create NtCreateFile NtMapViewOfSection NtMapViewOfSection C:\Win... f 5, handle \ip6fw.sys (read & decrypt buffer) (read & decrypt buffer) f 4, data FileHandle f 4, data NtWriteFile NtWriteFile 24

  25. Matching Behavior Graphs System Perspective Secure Systems Lab Technical University Vienna NtMapViewOfSection NtMapViewOfSection( out m_buffer [0...size] , out m_size) NtCreateFile( out c_handle) f 5, handle NtWriteFile 25

  26. Matching Behavior Graphs System Perspective Secure Systems Lab Technical University Vienna NtMapViewOfSection NtMapViewOfSection( out m_buffer [0...size] , out m_size) NtCreateFile( out c_handle) (p_buffer, p_size) = f 4 (m_buffer, m_size) f 5, handle NtWriteFile NtWriteFile( in w_handle, in w_buffer [0...size] , in w_size) 26

  27. Matching Behavior Graphs System Perspective Secure Systems Lab Technical University Vienna NtMapViewOfSection NtMapViewOfSection( out m_buffer [0...size] , out m_size) NtCreateFile( out c_handle ) ( p_buffer , p_size ) = f 4 (m_buffer, m_size) f 5, handle NtWriteFile NtWriteFile( in w_handle , in w_buffer [0...size] , in w_size ) 27

Recommend

Efficient Malware Detection using Model-Checking Tayssir Touili LIPN, CNRS & Univ. Paris 13

Efficient Malware Detection using Model-Checking Tayssir Touili LIPN, CNRS & Univ. Paris 13

Efficient Malware Detection using Model-Checking Tayssir Touili LIPN, CNRS & Univ. Paris 13 Motivation: Malware Detection The number of new malware exceeds 75 million by the end of 2011, and is still increasing. The number of

1.04k views • 52 slides
Security versus Energy Tradeoffs in Host-Based Mobile Malware Detection Jeffrey Bickford *, H.

Security versus Energy Tradeoffs in Host-Based Mobile Malware Detection Jeffrey Bickford *, H.

Security versus Energy Tradeoffs in Host-Based Mobile Malware Detection Jeffrey Bickford *, H. Andrs Lagar-Cavilla #, Alexander Varshavsky #, Vinod Ganapathy *, and Liviu Iftode * * Rutgers University # AT&T Labs Research Smart Phone

709 views • 28 slides
A CUCKOOS EGG IN THE MALWARE NEST ON-THE-FLY SIGNATURE-LESS MALWARE ANALYSIS, DETECTION AND

A CUCKOOS EGG IN THE MALWARE NEST ON-THE-FLY SIGNATURE-LESS MALWARE ANALYSIS, DETECTION AND

A CUCKOOS EGG IN THE MALWARE NEST ON-THE-FLY SIGNATURE-LESS MALWARE ANALYSIS, DETECTION AND CONTAINMENT FOR LARGE NETWORKS CHRISTIAAN SCHADE TWENTE SECURITY LAB UNIVERSITY OF TWENTE THE NETHERLANDS MALWARE WARS In the last

313 views • 14 slides
On Static Malware Detection Tayssir Touili LIPN, CNRS & Univ. Paris 13 Motivation: Malware

On Static Malware Detection Tayssir Touili LIPN, CNRS & Univ. Paris 13 Motivation: Malware

On Static Malware Detection Tayssir Touili LIPN, CNRS & Univ. Paris 13 Motivation: Malware Detection The number of new malware exceeds 75 million by the end of 2011, and is still increasing. The number of malware that produced

1.34k views • 91 slides
MALWARE DETECTION BY EATING A WHOLE EXE Presented by: Edward Raff Jared Sylvester Robert

MALWARE DETECTION BY EATING A WHOLE EXE Presented by: Edward Raff Jared Sylvester Robert

MALWARE DETECTION BY EATING A WHOLE EXE Presented by: Edward Raff Jared Sylvester Robert Brandon 1 November 2017 2 Malware Detection? Dont AVs do that? Single incidents of malware are now causing millions in damages. Potential

347 views • 19 slides
D&D of malware with exotic C&C D&D = Description & Detection C&C = Command

D&D of malware with exotic C&C D&D = Description & Detection C&C = Command

D&D of malware with exotic C&C D&D = Description & Detection C&C = Command & Control Automotive Consumer Energy & Chemicals Paul Rascagneres - @r00tbsd Eric Leblond - @Regiteric D&D of malware with exotic C&C

781 views • 34 slides
Experiences of of La Landing Machine Le Learning onto Market-Scale Mobile Malware Detection

Experiences of of La Landing Machine Le Learning onto Market-Scale Mobile Malware Detection

Experiences of of La Landing Machine Le Learning onto Market-Scale Mobile Malware Detection Liangyi Gong, Zhenhua Li, Feng Qian, Zifan Zhang, Qi Alfred Chen, Zhiyun Qian, Hao Lin, Yunhao Liu Mobile Malware Detection Android App Markets

424 views • 29 slides
Entrapment: Tricking Malware with Transparent, Scalable Malware Analysis Paul Royal

Entrapment: Tricking Malware with Transparent, Scalable Malware Analysis Paul Royal

Entrapment: Tricking Malware with Transparent, Scalable Malware Analysis Paul Royal paul@gtisc.gatech.edu Agenda Modern Malware Obfuscations, Server-side Polymorphism, Collection Volume Malware Analysis Detection

504 views • 27 slides
Analyzing Malware Detection Effectiveness with Multiple Anti- Malware Programs Jose A. Morales

Analyzing Malware Detection Effectiveness with Multiple Anti- Malware Programs Jose A. Morales

Analyzing Malware Detection Effectiveness with Multiple Anti- Malware Programs Jose A. Morales Shouhuai Xu Ravi Sandhu SEI @ CMU CS @ UTSA ICS @ UTSA Roadmap Motivation Experimental Methodology Experimental Results Summary

560 views • 16 slides
NoMoAds: Effective and Efficient Cross-App Mobile Ad-Blocking Anastasia Shuba Athina

NoMoAds: Effective and Efficient Cross-App Mobile Ad-Blocking Anastasia Shuba Athina

NoMoAds: Effective and Efficient Cross-App Mobile Ad-Blocking Anastasia Shuba Athina Markopoulou Zubair Shafiq UC Irvine UC Irvine University of Iowa Motivation Issues with Mobile Ads: 1. Intrusive 2. Overhead 3. Tracking 4. Malware

859 views • 27 slides
How does Malware Use RDTSC? A Study on Operations Executed by Malware with CPU Cycle Measurement

How does Malware Use RDTSC? A Study on Operations Executed by Malware with CPU Cycle Measurement

How does Malware Use RDTSC? A Study on Operations Executed by Malware with CPU Cycle Measurement Yoshihiro Oyama University of Tsukuba 1 Background Many malware programs execute operations for analysis evasion Detection of

346 views • 31 slides
Behavioral Detection and Containment of Proximity Malware in Delay Tolerant Networks Wei Peng,

Behavioral Detection and Containment of Proximity Malware in Delay Tolerant Networks Wei Peng,

Behavioral Detection and Containment of Proximity Malware in Delay Tolerant Networks Wei Peng, Feng Li, Xukai Zou, and Jie Wu 1 / 47 Proximity malware Definition. Proximity malware is a malicious program which propagates

618 views • 47 slides
Apposcopy: Semantics- Based Detection of Android Malware through Static Analysis By Feng et al

Apposcopy: Semantics- Based Detection of Android Malware through Static Analysis By Feng et al

Apposcopy: Semantics- Based Detection of Android Malware through Static Analysis By Feng et al [FSE 14] Presented by Maaz Ahmad The Malware Problem (Feb, 2015) Motive Security Labs estimates 16 million infected mobile devices. [1]

828 views • 30 slides
Netgator: Malware Detection Using Program Interactive Challenges Brian Schulte, Haris

Netgator: Malware Detection Using Program Interactive Challenges Brian Schulte, Haris

Netgator: Malware Detection Using Program Interactive Challenges Brian Schulte, Haris Andrianakis, Kun Sun, and Angelos Stavrou Intro Increase of stealthy malware in enterprises Obfuscation, polymorphic techniques Often uses

766 views • 25 slides
LO-PH Low-Observable Physical Host Instrumentation for Malware Analysis Chad Spensky ,

LO-PH Low-Observable Physical Host Instrumentation for Malware Analysis Chad Spensky ,

LO-PH Low-Observable Physical Host Instrumentation for Malware Analysis Chad Spensky , Hongyi Hu and Kevin Leach cspensky@cs.ucsb.edu hongyihu@alum.mit.edu kjl2y@virginia.edu lophi@mit.edu The Network and

532 views • 24 slides
Outline Intrusion detection systems Malware and the network CSci 5271 Announcements

Outline Intrusion detection systems Malware and the network CSci 5271 Announcements

Outline Intrusion detection systems Malware and the network CSci 5271 Announcements intermission Introduction to Computer Security Middlebox, malware, anonymity combined slides Denial of service and the network Anonymous communications

253 views • 11 slides
Botnets Leonidas Stylianou CS 682 23/04/2020 Lifecycle of a bot Infected host Botnet malware

Botnets Leonidas Stylianou CS 682 23/04/2020 Lifecycle of a bot Infected host Botnet malware

Botnets Leonidas Stylianou CS 682 23/04/2020 Lifecycle of a bot Infected host Botnet malware Botmaster controls becomes a bot and infects a host. the botnet. joins the botnet . Coordination of f bots with C&C server Bots query the

1k views • 58 slides
Impeding Automated Malware Analysis with Environment-sensitive Malware Chengyu Song , Paul Royal

Impeding Automated Malware Analysis with Environment-sensitive Malware Chengyu Song , Paul Royal

Impeding Automated Malware Analysis with Environment-sensitive Malware Chengyu Song , Paul Royal and Wenke Lee College of Computing Georgia Institute of Technology Agenda l Background l Defeating Automated Malware Analysis Host

1.28k views • 28 slides
Intrusion Detection Distributed Host-Based Network-Based ITS335: IT Security Honeypots

Intrusion Detection Distributed Host-Based Network-Based ITS335: IT Security Honeypots

ITS335 Intrusion Detection Intruders Intrusion Detection Host-Based Intrusion Detection Distributed Host-Based Network-Based ITS335: IT Security Honeypots Summary Sirindhorn International Institute of Technology Thammasat University

585 views • 30 slides
Malware Behavioral Detection by Attribute-Automata using Abstraction from Platform and Language

Malware Behavioral Detection by Attribute-Automata using Abstraction from Platform and Language

Malware Behavioral Detection by Attribute-Automata using Abstraction from Platform and Language JACOB Grgoire 1/2 , DEBAR Herv 1 , FILIOL Eric 2 1 Orange Labs / France Tlcom R&D, Security and Trusted Transactions (MAPS/STT). 2

327 views • 21 slides
Alpha Presentation Next Generation Malware Detection, Clustering and Heuristics The Capstone

Alpha Presentation Next Generation Malware Detection, Clustering and Heuristics The Capstone

Alpha Presentation Next Generation Malware Detection, Clustering and Heuristics The Capstone Experience Team Proofpoint George Zhao Yash Patel Graham Thomas Brad Doherty Crystal Lewis Department of Computer Science and Engineering

761 views • 10 slides
Project Plan Next Generation Malware Detection, Clustering and Heuristics The Capstone

Project Plan Next Generation Malware Detection, Clustering and Heuristics The Capstone

Project Plan Next Generation Malware Detection, Clustering and Heuristics The Capstone Experience Team Proofpoint Crystal Lewis Yash Patel George Zhao Graham Thomas Brad Doherty Department of Computer Science and Engineering Michigan

402 views • 11 slides
Machine Virtualization: Efficient Hypervisors, Stealthy Malware Muli Ben-Yehuda Technion &

Machine Virtualization: Efficient Hypervisors, Stealthy Malware Muli Ben-Yehuda Technion &

Machine Virtualization: Efficient Hypervisors, Stealthy Malware Muli Ben-Yehuda Technion & Hypervisor Technologies and Consulting Ltd Muli Ben-Yehuda (Technion & Hypervisor) Efficient Hypervisors, Stealthy Malware Cyberday, 2013 1 /

553 views • 21 slides
Outline Intrusion detection systems CSci 5271 Malware and the network Introduction to Computer

Outline Intrusion detection systems CSci 5271 Malware and the network Introduction to Computer

Outline Intrusion detection systems CSci 5271 Malware and the network Introduction to Computer Security Announcements intermission Day 22: Malware and Denial of Service Stephen McCamant Denial of service and the network University of

684 views • 10 slides

More recommend